Embeddings Explained — How Semantic Search Works in LLM Applications

Introduction

Why This Matters

Embeddings are the foundational technology behind semantic search, RAG, recommendation systems, and anomaly detection in LLM applications. Without understanding embeddings, you cannot reason about retrieval quality, debug why your RAG system returns irrelevant chunks, or make informed decisions about which embedding model to use.

Unlike keyword search (which matches exact words), semantic search understands meaning. A query for "how do I cancel my subscription?" will find documents about "account termination procedures" even though no keyword overlaps. This is possible because both phrases map to nearby points in a high-dimensional embedding space.

Every major AI application — from GitHub Copilot to Notion AI to enterprise knowledge bases — depends on embeddings. Mastering how they work and how to use them effectively is one of the most transferable skills in the AI engineering toolkit.

What Are Embeddings?

An embedding is a dense vector of floating-point numbers that represents the semantic meaning of a piece of text (or image, audio, or code). Similar meanings produce similar vectors — mathematically close in the embedding space.

from openai import OpenAI
import numpy as np
 
client = OpenAI()
 
def embed(text: str) -> np.ndarray:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return np.array(response.data[0].embedding)
 
# Similar concepts have similar embeddings
vec1 = embed("machine learning")
vec2 = embed("artificial intelligence")
vec3 = embed("pizza recipe")
 
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
 
print(cosine_similarity(vec1, vec2))  # ~0.92 — very similar
print(cosine_similarity(vec1, vec3))  # ~0.45 — dissimilar

The embedding model (text-embedding-3-small) transforms text into a 1536-dimensional vector — a point in 1536-dimensional space. The direction of that point encodes semantic content.

How Embedding Models Work

Modern embedding models are based on the BERT architecture (bidirectional transformers). Unlike GPT-style models that predict the next token, BERT-style models process the entire input bidirectionally to produce a single representation.

Training uses contrastive objectives:

Sentence-BERT: pairs of semantically similar sentences are trained to produce nearby embeddings
Contrastive learning: positive pairs (paraphrase, question-answer) are pulled together; negative pairs are pushed apart
MNRL (Multiple Negatives Ranking Loss): efficient batch training where each example's negatives come from other batch items

The result: a model that maps semantically equivalent texts to nearby vectors regardless of surface-level word choice.

Choosing an Embedding Model

# OpenAI — best quality for English, API-based
from openai import OpenAI
client = OpenAI()
 
# text-embedding-3-small: 1536 dims, cheapest, great quality
# text-embedding-3-large: 3072 dims, best quality, 5x cost
response = client.embeddings.create(
    model="text-embedding-3-small",
    input=["Hello world", "Goodbye world"]
)
 
# Sentence Transformers — free, local, no API calls
from sentence_transformers import SentenceTransformer
 
model = SentenceTransformer("BAAI/bge-large-en-v1.5")  # best open-source English model
embeddings = model.encode(["Hello world", "Goodbye world"], normalize_embeddings=True)
 
# Cohere — strong multilingual support
import cohere
co = cohere.Client("your-key")
response = co.embed(
    texts=["Hello world"],
    model="embed-multilingual-v3.0",
    input_type="search_document"
)

Model comparison:

Model	Dimensions	MTEB Score	Cost
text-embedding-3-large	3072	64.6	$0.13/1M tokens
text-embedding-3-small	1536	62.3	$0.02/1M tokens
BAAI/bge-large-en-v1.5	1024	63.98	Free (local)
embed-multilingual-v3.0	1024	64.0	$0.10/1M tokens

Similarity Metrics

Three distance metrics are used for comparing embeddings:

import numpy as np
 
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    """Most common. Measures angle between vectors. Range: -1 to 1."""
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
 
def dot_product(a: np.ndarray, b: np.ndarray) -> float:
    """Faster than cosine. Use when embeddings are already normalized."""
    return float(np.dot(a, b))
 
def euclidean_distance(a: np.ndarray, b: np.ndarray) -> float:
    """Less common for text. Measures absolute distance."""
    return float(np.linalg.norm(a - b))
 
# For normalized embeddings (unit vectors), dot product == cosine similarity
# Most modern embedding models return normalized vectors
a_normalized = embed("query") / np.linalg.norm(embed("query"))

Rule of thumb: Use cosine similarity for text embeddings. Use dot product when vectors are pre-normalized (faster). Use euclidean distance rarely for text — mostly for image embeddings.

Building Semantic Search from Scratch

import numpy as np
from openai import OpenAI
 
client = OpenAI()
 
class SemanticSearchEngine:
    def __init__(self):
        self.documents = []
        self.embeddings = []
 
    def embed_batch(self, texts: list[str]) -> list[np.ndarray]:
        response = client.embeddings.create(
            model="text-embedding-3-small",
            input=texts
        )
        return [np.array(item.embedding) for item in response.data]
 
    def index(self, documents: list[str]):
        self.documents = documents
        self.embeddings = self.embed_batch(documents)
        # Normalize for faster cosine similarity via dot product
        self.embeddings = [e / np.linalg.norm(e) for e in self.embeddings]
 
    def search(self, query: str, top_k: int = 5) -> list[dict]:
        query_emb = self.embed_batch([query])[0]
        query_emb = query_emb / np.linalg.norm(query_emb)
 
        scores = [float(np.dot(query_emb, doc_emb)) for doc_emb in self.embeddings]
        top_indices = np.argsort(scores)[::-1][:top_k]
 
        return [
            {"document": self.documents[i], "score": scores[i]}
            for i in top_indices
        ]
 
# Usage
engine = SemanticSearchEngine()
engine.index([
    "The refund window is 30 days from purchase date.",
    "Enterprise plans include dedicated support and 99.9% SLA.",
    "API rate limits: 1000 requests per minute on Pro tier.",
    "To cancel your subscription, go to Account Settings.",
])
 
results = engine.search("how do I stop my subscription?")
for r in results:
    print(f"{r['score']:.4f} | {r['document']}")

Chunking Strategy Affects Embedding Quality

The unit of text you embed significantly affects retrieval quality:

from langchain.text_splitter import RecursiveCharacterTextSplitter
 
# Too large: embeddings average out too much detail
large_splitter = RecursiveCharacterTextSplitter(chunk_size=2000, chunk_overlap=200)
 
# Too small: embeddings lack context for accurate semantic matching
small_splitter = RecursiveCharacterTextSplitter(chunk_size=100, chunk_overlap=20)
 
# Good default: 512-1024 tokens with sentence-aware splitting
good_splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    separators=["\n\n", "\n", ". ", " ", ""]
)

Handling Long Documents

Most embedding models have a 512 or 8192 token limit. For longer documents:

from openai import OpenAI
import tiktoken
 
client = OpenAI()
tokenizer = tiktoken.get_encoding("cl100k_base")
 
def safe_embed(text: str, max_tokens: int = 8000) -> list[float]:
    tokens = tokenizer.encode(text)
    if len(tokens) > max_tokens:
        # Truncate to max tokens
        text = tokenizer.decode(tokens[:max_tokens])
 
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding

For documents longer than the model limit, chunk them first, embed each chunk, and store separately.

Common Mistakes / Pitfalls

Using the same embedding model for indexing and a different one for querying — vectors are only comparable within the same model
Embedding entire 10-page documents as one unit — quality degrades; the embedding averages out all the information
Not normalizing embeddings before cosine similarity comparison — raw dot product gives wrong scores
Mixing input_type="search_document" and input_type="search_query" incorrectly in Cohere — query and document embeddings use different projections
Re-embedding the same chunks on every application restart — persist embeddings to disk or a vector store

Best Practices

Always use the same embedding model for both indexing and querying — never mix models
Normalize embeddings before storage — saves compute on every similarity calculation
Start with text-embedding-3-small for cost efficiency; upgrade to text-embedding-3-large only if recall quality is insufficient
Benchmark on your domain-specific data using BEIR or MTEB-style evaluation before choosing a model
Use batched embedding calls (input=["text1", "text2", ...]) — far more efficient than one call per text

Key Takeaways

Embeddings are dense float vectors that encode semantic meaning — similar texts produce geometrically close vectors
Modern embedding models are BERT-style transformers trained with contrastive objectives to learn semantic similarity
Cosine similarity is the standard metric for text embedding comparison; dot product is equivalent for normalized vectors
OpenAI text-embedding-3-small (1536 dims) and BAAI/bge-large-en-v1.5 (1024 dims, free) are the most practical models in 2025
Chunk size critically affects embedding quality — 512-1024 tokens with sentence-boundary awareness is a reliable default
Always use the exact same embedding model for indexing and querying — cross-model comparison produces meaningless scores
Batched embedding API calls are 10-100x more efficient than single-text calls at scale
For documents exceeding the model token limit, chunk first and embed each chunk separately — never truncate silently

Introduction#

Why This Matters#

What Are Embeddings?#

How Embedding Models Work#

Choosing an Embedding Model#

Similarity Metrics#

Building Semantic Search from Scratch#

Chunking Strategy Affects Embedding Quality#

Handling Long Documents#

Common Mistakes / Pitfalls#

Best Practices#

Key Takeaways#

Related reading