Sentence Transformers — Generate Text Embeddings Locally Without API Calls

Introduction

Why This Matters

Sentence Transformers is the leading open-source library for generating text embeddings locally. Unlike OpenAI's API, it runs entirely on your machine — no API key required, no per-call cost, no data leaving your infrastructure. For teams with data privacy requirements, high embedding volumes, or offline deployment needs, Sentence Transformers is the standard solution.

The library wraps hundreds of pretrained models from Hugging Face, ranging from compact 22M-parameter models that run on CPUs to large 335M-parameter models that rival OpenAI's quality on benchmarks. Models like BAAI/bge-large-en-v1.5 and intfloat/e5-large-v2 consistently score above 60 on MTEB — competitive with text-embedding-3-small.

Understanding which Sentence Transformer model to use, how to batch encode efficiently, how to run inference on GPU, and how to integrate with vector databases like Chroma, Pinecone, and Weaviate is essential for building private, cost-effective AI applications.

Installation

pip install sentence-transformers torch

For GPU acceleration (recommended for production):

pip install sentence-transformers torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121

Basic Usage

from sentence_transformers import SentenceTransformer
import numpy as np
 
# Load a model (downloads automatically on first use)
model = SentenceTransformer("BAAI/bge-large-en-v1.5")
 
# Encode a single sentence
embedding = model.encode("The quick brown fox jumps over the lazy dog.")
print(f"Dimension: {len(embedding)}")  # 1024
print(f"Type: {type(embedding)}")      # numpy.ndarray
 
# Encode a list of sentences (batched)
sentences = [
    "Refund requests must be submitted within 30 days.",
    "Enterprise plans include 99.9% SLA.",
    "API rate limits are 1000 requests per minute.",
]
embeddings = model.encode(sentences, normalize_embeddings=True)
print(f"Shape: {embeddings.shape}")  # (3, 1024)

Choosing the Right Model

The MTEB (Massive Text Embedding Benchmark) is the standard for comparing embedding models:

# Best open-source English model — MTEB 64.23, 560MB
model = SentenceTransformer("BAAI/bge-large-en-v1.5")
 
# Faster, lighter — MTEB 63.55, 140MB (good for CPU)
model = SentenceTransformer("BAAI/bge-base-en-v1.5")
 
# Smallest, fastest — MTEB 51.68, 33MB (edge/mobile)
model = SentenceTransformer("BAAI/bge-small-en-v1.5")
 
# E5 models — Microsoft, strong on asymmetric search
model = SentenceTransformer("intfloat/e5-large-v2")
 
# Multilingual — 50+ languages
model = SentenceTransformer("intfloat/multilingual-e5-large")
 
# All-MiniLM — lightweight, 22MB, great for CPU-only
model = SentenceTransformer("all-MiniLM-L6-v2")

Model selection guide:

High quality, GPU available: BAAI/bge-large-en-v1.5
Fast CPU inference: BAAI/bge-base-en-v1.5
Edge/embedded: all-MiniLM-L6-v2
Multilingual: intfloat/multilingual-e5-large

Query vs Document Embeddings (BGE and E5 Models)

BGE and E5 models are trained with asymmetric objectives — queries and documents should be encoded differently:

from sentence_transformers import SentenceTransformer
 
model = SentenceTransformer("BAAI/bge-large-en-v1.5")
 
# BGE: add "Represent this sentence:" prefix to queries
query = "Represent this sentence: " + "What is the refund policy?"
doc = "Refund requests must be submitted within 30 days of purchase."
 
query_emb = model.encode(query, normalize_embeddings=True)
doc_emb = model.encode(doc, normalize_embeddings=True)
 
similarity = float(query_emb @ doc_emb)
print(f"Similarity: {similarity:.4f}")
 
# E5 models use different prefixes
model_e5 = SentenceTransformer("intfloat/e5-large-v2")
 
# E5 query prefix
query_e5 = "query: " + "What is the refund policy?"
# E5 document prefix
doc_e5 = "passage: " + "Refund requests must be submitted within 30 days of purchase."

Failing to use the correct prefix for BGE/E5 models significantly reduces retrieval quality.

GPU Acceleration

import torch
from sentence_transformers import SentenceTransformer
 
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using: {device}")
 
model = SentenceTransformer("BAAI/bge-large-en-v1.5", device=device)
 
# Larger batch sizes for GPU
embeddings = model.encode(
    sentences,
    batch_size=64,           # increase for GPU (32-128 typical)
    normalize_embeddings=True,
    show_progress_bar=True,
    convert_to_numpy=True
)

Batch Encoding Large Datasets

from sentence_transformers import SentenceTransformer
import numpy as np
 
model = SentenceTransformer("BAAI/bge-large-en-v1.5")
 
def encode_dataset(texts: list[str], batch_size: int = 32) -> np.ndarray:
    """Encode a large dataset efficiently."""
    return model.encode(
        texts,
        batch_size=batch_size,
        normalize_embeddings=True,
        show_progress_bar=True,
        convert_to_numpy=True
    )
 
# Encode 100,000 documents
# On CPU: ~30 min; on GPU (A100): ~2 min
documents = ["Document text " + str(i) for i in range(100_000)]
all_embeddings = encode_dataset(documents, batch_size=128)
print(f"Shape: {all_embeddings.shape}")  # (100000, 1024)
 
# Save to disk
np.save("embeddings.npy", all_embeddings)
# Load later
all_embeddings = np.load("embeddings.npy")

Similarity Search

import numpy as np
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
 
model = SentenceTransformer("BAAI/bge-large-en-v1.5")
 
corpus = [
    "The refund window is 30 days from purchase.",
    "Enterprise plans include dedicated support.",
    "API rate limits: 1000 requests/minute on Pro.",
    "Free plan limited to 100 API calls per day.",
]
 
corpus_embs = model.encode(corpus, normalize_embeddings=True, convert_to_tensor=True)
 
query = "Represent this sentence: " + "how many API calls can I make?"
query_emb = model.encode(query, normalize_embeddings=True, convert_to_tensor=True)
 
# Efficient cosine similarity using sentence_transformers utility
scores = cos_sim(query_emb, corpus_embs)[0]
top_k = scores.topk(3)
 
print("Top results:")
for score, idx in zip(top_k.values, top_k.indices):
    print(f"  {score:.4f} | {corpus[idx]}")

Integration with Chroma

import chromadb
from chromadb import EmbeddingFunction, Embeddings
from sentence_transformers import SentenceTransformer
from typing import List
 
class BGEEmbeddingFunction(EmbeddingFunction):
    def __init__(self, model_name: str = "BAAI/bge-base-en-v1.5"):
        self.model = SentenceTransformer(model_name)
 
    def __call__(self, input: List[str]) -> Embeddings:
        # Add BGE query prefix for queries — for documents, no prefix needed
        return self.model.encode(input, normalize_embeddings=True).tolist()
 
client = chromadb.PersistentClient(path="./chroma_data")
embedding_fn = BGEEmbeddingFunction()
 
collection = client.get_or_create_collection(
    name="local_rag",
    embedding_function=embedding_fn,
    metadata={"hnsw:space": "cosine"}
)
 
collection.add(
    documents=corpus,
    ids=[f"doc-{i}" for i in range(len(corpus))]
)
 
results = collection.query(
    query_texts=["Represent this sentence: what is the API limit?"],
    n_results=2
)
print(results["documents"])

Common Mistakes / Pitfalls

Forgetting the query prefix for BGE and E5 models — retrieval quality drops significantly without it
Using the same batch_size for CPU and GPU — GPU can handle 64-128 while CPU is better at 8-16
Comparing embeddings from different model variants — bge-small and bge-large vectors are not comparable
Not normalizing embeddings — cosine similarity gives wrong results without normalization
Loading the model inside a hot loop — load once globally and reuse the instance

Best Practices

Load models once at application startup and keep in memory — model loading takes 1-5 seconds
Set normalize_embeddings=True in encode() — eliminates manual normalization and enables dot-product similarity
For BGE models, prefix queries with "Represent this sentence: " but encode documents without a prefix
Use convert_to_tensor=True when doing similarity calculations with cos_sim() — stays on GPU
Cache encoded embeddings to disk with np.save() — re-encoding the same corpus costs time, not money

Key Takeaways

Sentence Transformers generates high-quality text embeddings locally — no API key, no per-call cost, no data leaving your infrastructure
BAAI/bge-large-en-v1.5 is the top open-source English model with MTEB 64.23 — competitive with OpenAI's text-embedding-3-small
BGE and E5 models require different prefixes for queries ("Represent this sentence: " / "query: ") vs documents — a common source of quality issues
GPU acceleration with batch sizes of 64-128 reduces encoding time by 10-20x compared to CPU
Encoding 100K documents takes ~2 minutes on a modern GPU — batch processing at scale is practical
normalize_embeddings=True in encode() produces unit vectors — enables fast dot-product similarity instead of cosine
Integration with Chroma, Pinecone, and Weaviate requires a custom EmbeddingFunction wrapper around the SentenceTransformer model
For multilingual applications, intfloat/multilingual-e5-large supports 50+ languages with strong cross-lingual retrieval

Introduction#

Why This Matters#

Installation#

Basic Usage#

Choosing the Right Model#

Query vs Document Embeddings (BGE and E5 Models)#

GPU Acceleration#

Batch Encoding Large Datasets#

Similarity Search#

Integration with Chroma#

Common Mistakes / Pitfalls#

Best Practices#

Key Takeaways#

Related reading