Chroma DB Guide — Open Source Vector Database for Local LLM Development

Introduction

Why This Matters

Chroma is the go-to vector database for LLM development when you want zero infrastructure overhead. It runs as an embedded Python library — no Docker, no cloud account, no API key required. You can have a fully functional vector store in your RAG application in under five minutes.

For development, prototyping, and small production deployments (under 1 million vectors), Chroma is the right choice. It integrates natively with LangChain, LlamaIndex, and every major embedding provider. Its persistent storage mode writes to a local SQLite-backed directory, making it simple to share vector indexes across sessions without re-embedding.

Chroma also offers a server mode for production use cases where you need a shared vector store accessible from multiple services or a Docker-based deployment. Understanding when to use in-process vs. server mode — and how to integrate Chroma with your embedding and LLM stack — is what this guide covers.

Installation

pip install chromadb

Chroma requires no additional dependencies for local use. For OpenAI embeddings integration:

pip install chromadb openai

Client Modes

Chroma supports three deployment modes:

import chromadb
 
# 1. In-memory (ephemeral — lost on restart)
client = chromadb.Client()
 
# 2. Persistent local storage (survives restarts)
client = chromadb.PersistentClient(path="./chroma_data")
 
# 3. HTTP server (for multi-service production use)
# First: docker run -p 8000:8000 chromadb/chroma
client = chromadb.HttpClient(host="localhost", port=8000)

Creating Collections

Collections are analogous to tables — they hold vectors, metadata, and optional documents:

import chromadb
 
client = chromadb.PersistentClient(path="./chroma_data")
 
# Create or get an existing collection
collection = client.get_or_create_collection(
    name="company_knowledge_base",
    metadata={
        "hnsw:space": "cosine",       # distance metric: cosine | l2 | ip
        "hnsw:construction_ef": 200,  # index quality (higher = better quality, slower build)
        "hnsw:search_ef": 100,        # query quality (higher = better recall, slower query)
    }
)
 
# List all collections
print(client.list_collections())
 
# Delete a collection
client.delete_collection("old_collection")

Adding Documents with Custom Embeddings

When you provide embeddings explicitly (using any embedding model):

from openai import OpenAI
import chromadb
 
openai_client = OpenAI()
chroma_client = chromadb.PersistentClient(path="./chroma_data")
collection = chroma_client.get_or_create_collection(
    name="docs",
    metadata={"hnsw:space": "cosine"}
)
 
def get_embeddings(texts: list[str]) -> list[list[float]]:
    response = openai_client.embeddings.create(
        model="text-embedding-3-small",
        input=texts
    )
    return [item.embedding for item in response.data]
 
documents = [
    "Returns are accepted within 30 days of purchase.",
    "Enterprise plans include 99.9% SLA.",
    "API rate limits are 1000 req/min on Pro plan.",
]
 
metadatas = [
    {"source": "handbook", "section": "returns", "page": 12},
    {"source": "contracts", "tier": "enterprise"},
    {"source": "api-docs", "plan": "pro"},
]
 
ids = ["doc-1", "doc-2", "doc-3"]
embeddings = get_embeddings(documents)
 
collection.add(
    ids=ids,
    embeddings=embeddings,
    documents=documents,
    metadatas=metadatas
)
 
print(f"Collection size: {collection.count()}")

Auto-Embedding with Built-in Functions

Chroma can handle embedding automatically via embedding functions:

from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction
import chromadb
 
client = chromadb.PersistentClient(path="./chroma_data")
 
embedding_fn = OpenAIEmbeddingFunction(
    api_key="your-openai-key",
    model_name="text-embedding-3-small"
)
 
# Collection automatically embeds documents on add() and query()
collection = client.get_or_create_collection(
    name="auto_embedded",
    embedding_function=embedding_fn
)
 
collection.add(
    documents=["Refund policy: 30-day returns.", "SLA: 99.9% uptime."],
    ids=["d1", "d2"],
    metadatas=[{"source": "handbook"}, {"source": "contracts"}]
)
 
# Query with plain text — no manual embedding needed
results = collection.query(
    query_texts=["what is the return policy?"],
    n_results=2
)
 
print(results["documents"])
print(results["distances"])

Querying with Metadata Filters

# Where clause filters (applied before vector search)
results = collection.query(
    query_texts=["API limits"],
    n_results=3,
    where={"source": "api-docs"},       # exact match
    include=["documents", "metadatas", "distances"]
)
 
# Compound filters
results = collection.query(
    query_texts=["SLA guarantees"],
    n_results=5,
    where={
        "$and": [
            {"source": {"$eq": "contracts"}},
            {"tier": {"$in": ["enterprise", "business"]}}
        ]
    }
)
 
# Full-text document content filter (where_document)
results = collection.query(
    query_texts=["uptime"],
    n_results=3,
    where_document={"$contains": "99.9%"}
)
 
print(results)

Supported operators: $eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $and, $or

CRUD Operations

# Get specific documents by ID
items = collection.get(ids=["doc-1", "doc-2"], include=["documents", "metadatas"])
 
# Update documents (embeddings recalculated if not provided)
collection.update(
    ids=["doc-1"],
    documents=["Returns are accepted within 60 days of purchase."],
    metadatas=[{"source": "handbook", "section": "returns", "updated": True}]
)
 
# Upsert (insert or update)
collection.upsert(
    ids=["doc-new"],
    documents=["New policy: free shipping on enterprise orders."],
    metadatas=[{"source": "handbook", "section": "shipping"}]
)
 
# Delete by ID
collection.delete(ids=["doc-1"])
 
# Delete by filter
collection.delete(where={"source": "outdated-docs"})

LangChain Integration

from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser
 
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
 
# Create from documents
vectorstore = Chroma.from_documents(
    documents=chunks,               # list of LangChain Document objects
    embedding=embeddings,
    collection_name="company_docs",
    persist_directory="./chroma_data"
)
 
# Load existing
vectorstore = Chroma(
    collection_name="company_docs",
    embedding_function=embeddings,
    persist_directory="./chroma_data"
)
 
retriever = vectorstore.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 4, "filter": {"source": "handbook"}}
)
 
# Full RAG chain
llm = ChatOpenAI(model="gpt-4o")
prompt = ChatPromptTemplate.from_template("Context: {context}\n\nQuestion: {question}")
 
rag_chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)
 
answer = rag_chain.invoke("What is the refund policy?")

Common Mistakes / Pitfalls

Using in-memory client for anything you need to persist — switch to PersistentClient from the start
Adding duplicate IDs without checking — Chroma raises an error; use upsert() when IDs may already exist
Not specifying the distance metric on collection creation — default is L2, but cosine is better for normalized text embeddings
Storing millions of vectors in Chroma's local mode — performance degrades significantly beyond ~500K vectors
Calling add() with raw text when an embedding function is not set — you must provide either embeddings or an embedding function

Best Practices

Always use get_or_create_collection() instead of create_collection() to make your code idempotent
Include all the text you need in documents — it is returned in query results, eliminating a separate text lookup
Use PersistentClient with a versioned directory path when iterating on chunking strategies to avoid mixing old and new embeddings
For production serving, use Chroma in HTTP server mode behind your API — this lets multiple workers share a single vector store
Migrate to Pinecone or Qdrant Cloud when your collection exceeds 500K vectors or requires distributed search

Key Takeaways

Chroma runs embedded in Python with no external dependencies — the fastest setup of any vector database
Three modes: in-memory (ephemeral), persistent local (SQLite-backed), and HTTP server (multi-service)
Collections store vectors, documents, and metadata — query by semantic similarity, metadata filter, or document content
The embedding_function parameter enables auto-embedding on add() and query() — no manual embedding code required
Supports OpenAI, Cohere, Hugging Face, and SentenceTransformers as built-in embedding functions
Metadata filtering with where clauses supports $eq, $in, $gt, compound $and/$or operators
Native LangChain and LlamaIndex integration via Chroma vectorstore class — drop-in replacement for other stores
Best suited for up to ~500K vectors; migrate to distributed vector stores for larger production workloads

Introduction#

Why This Matters#

Installation#

Client Modes#

Creating Collections#

Adding Documents with Custom Embeddings#

Auto-Embedding with Built-in Functions#

Querying with Metadata Filters#

CRUD Operations#

LangChain Integration#

Common Mistakes / Pitfalls#

Best Practices#

Key Takeaways#

Related reading