AI-Powered Search — Building Semantic Search That Actually Works in Production

Introduction

Keyword search misses synonyms, context, and intent. Semantic search — powered by embeddings and vector databases — retrieves documents by meaning rather than exact token matches. But production semantic search has sharp edges: slow indexing, stale embeddings, poor recall on short queries, and high infrastructure cost. This post covers the architecture of a production-grade semantic search system that handles all of these.

Why This Matters

A search system that returns irrelevant results loses users immediately. Semantic search consistently outperforms BM25 keyword search by 15-40% on recall metrics for natural language queries. For enterprise products where users phrase queries in their own words, the gap is even larger. The investment in an embedding-based search pipeline pays back quickly in reduced support tickets and higher retention.

Embedding Pipeline

Build an indexing pipeline that generates and stores embeddings for all documents:

import Anthropic from '@anthropic-ai/sdk';
 
const client = new Anthropic();
 
interface Document {
  id: string;
  title: string;
  content: string;
  metadata: Record<string, unknown>;
}
 
interface IndexedDocument extends Document {
  embedding: number[];
  indexedAt: string;
}
 
// Use voyage-3 via the Anthropic embeddings endpoint or any embedding model
async function embedText(text: string): Promise<number[]> {
  // Placeholder: replace with your embedding provider (Voyage, OpenAI, Cohere)
  // For Anthropic-integrated workflows, use Voyage via the API
  const normalized = text.slice(0, 8192); // respect token limits
  return Array.from({ length: 1536 }, (_, i) => Math.sin(i + normalized.length));
}
 
async function indexDocuments(docs: Document[]): Promise<IndexedDocument[]> {
  const BATCH_SIZE = 20;
  const indexed: IndexedDocument[] = [];
 
  for (let i = 0; i < docs.length; i += BATCH_SIZE) {
    const batch = docs.slice(i, i + BATCH_SIZE);
 
    const embeddings = await Promise.all(
      batch.map((doc) => embedText(`${doc.title}\n\n${doc.content}`))
    );
 
    for (let j = 0; j < batch.length; j++) {
      indexed.push({
        ...batch[j],
        embedding: embeddings[j],
        indexedAt: new Date().toISOString(),
      });
    }
 
    // Rate limit: avoid overwhelming the embedding API
    if (i + BATCH_SIZE < docs.length) {
      await new Promise((r) => setTimeout(r, 100));
    }
  }
 
  return indexed;
}

Hybrid Search: Semantic + Keyword

Pure semantic search fails on exact lookups (product codes, names). Combine with BM25:

interface SearchResult {
  id: string;
  title: string;
  content: string;
  semanticScore: number;
  keywordScore: number;
  hybridScore: number;
}
 
class HybridSearchEngine {
  private documents: IndexedDocument[] = [];
 
  index(docs: IndexedDocument[]): void {
    this.documents = docs;
  }
 
  async search(query: string, topK = 10): Promise<SearchResult[]> {
    const queryEmbedding = await embedText(query);
 
    const results = this.documents.map((doc) => {
      const semanticScore = this.cosineSimilarity(queryEmbedding, doc.embedding);
      const keywordScore = this.bm25Score(query, doc.content);
      // RRF (Reciprocal Rank Fusion) with alpha=0.7 for semantic weight
      const hybridScore = 0.7 * semanticScore + 0.3 * keywordScore;
 
      return {
        id: doc.id,
        title: doc.title,
        content: doc.content,
        semanticScore,
        keywordScore,
        hybridScore,
      };
    });
 
    return results
      .sort((a, b) => b.hybridScore - a.hybridScore)
      .slice(0, topK);
  }
 
  private cosineSimilarity(a: number[], b: number[]): number {
    let dot = 0, magA = 0, magB = 0;
    for (let i = 0; i < a.length; i++) {
      dot += a[i] * b[i];
      magA += a[i] ** 2;
      magB += b[i] ** 2;
    }
    return dot / (Math.sqrt(magA) * Math.sqrt(magB));
  }
 
  private bm25Score(query: string, text: string): number {
    const queryTerms = query.toLowerCase().split(/\s+/);
    const textTerms = text.toLowerCase().split(/\s+/);
    const termFreq = new Map<string, number>();
    for (const t of textTerms) termFreq.set(t, (termFreq.get(t) ?? 0) + 1);
 
    const k1 = 1.5, b = 0.75;
    const avgLen = 200;
    const len = textTerms.length;
 
    return queryTerms.reduce((score, term) => {
      const tf = termFreq.get(term) ?? 0;
      const idf = Math.log(1 + (1000 - 1 + 0.5) / (1 + 1)); // simplified IDF
      const tfScore = (tf * (k1 + 1)) / (tf + k1 * (1 - b + b * len / avgLen));
      return score + idf * tfScore;
    }, 0);
  }
}

Query Rewriting with LLM

Improve recall by rewriting vague user queries before embedding:

async function rewriteQuery(rawQuery: string): Promise<string[]> {
  const response = await client.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 256,
    system: 'Rewrite the user query into 3 alternative phrasings that capture the same intent. Return a JSON array of strings.',
    messages: [{ role: 'user', content: rawQuery }],
  });
 
  const text = response.content[0].type === 'text' ? response.content[0].text.trim() : '[]';
  try {
    const variants = JSON.parse(text.replace(/^```json\n?/, '').replace(/\n?```$/, ''));
    return Array.isArray(variants) ? [rawQuery, ...variants].slice(0, 4) : [rawQuery];
  } catch {
    return [rawQuery];
  }
}
 
async function searchWithQueryExpansion(query: string, engine: HybridSearchEngine, topK = 10) {
  const queries = await rewriteQuery(query);
 
  const allResults = await Promise.all(queries.map((q) => engine.search(q, topK)));
 
  // Merge and deduplicate by ID, keeping highest score
  const merged = new Map<string, SearchResult>();
  for (const results of allResults) {
    for (const result of results) {
      const existing = merged.get(result.id);
      if (!existing || result.hybridScore > existing.hybridScore) {
        merged.set(result.id, result);
      }
    }
  }
 
  return Array.from(merged.values())
    .sort((a, b) => b.hybridScore - a.hybridScore)
    .slice(0, topK);
}

Re-ranking with LLM

Use an LLM to re-rank the top candidates for higher precision:

interface RerankCandidate {
  id: string;
  title: string;
  content: string;
  initialScore: number;
}
 
async function rerankResults(query: string, candidates: RerankCandidate[], topK = 5) {
  const candidateList = candidates
    .slice(0, 10) // only re-rank top 10 to limit cost
    .map((c, i) => `[${i}] ${c.title}: ${c.content.slice(0, 200)}`)
    .join('\n');
 
  const response = await client.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 128,
    system: 'Re-rank the following search results by relevance to the query. Return a JSON array of indices in order of relevance, most relevant first.',
    messages: [{
      role: 'user',
      content: `Query: "${query}"\n\nResults:\n${candidateList}`,
    }],
  });
 
  const text = response.content[0].type === 'text' ? response.content[0].text.trim() : '[]';
  try {
    const indices: number[] = JSON.parse(text.replace(/```json\n?/, '').replace(/\n?```/, ''));
    return indices
      .filter((i) => i >= 0 && i < candidates.length)
      .map((i) => candidates[i])
      .slice(0, topK);
  } catch {
    return candidates.slice(0, topK);
  }
}

Search Result Caching

Cache expensive embedding computations and search results:

class SearchCache {
  private queryCache = new Map<string, { results: SearchResult[]; cachedAt: number }>();
  private embeddingCache = new Map<string, number[]>();
  private readonly TTL_MS = 5 * 60 * 1000; // 5 minutes
 
  getCachedQuery(query: string): SearchResult[] | null {
    const cached = this.queryCache.get(query);
    if (!cached) return null;
    if (Date.now() - cached.cachedAt > this.TTL_MS) {
      this.queryCache.delete(query);
      return null;
    }
    return cached.results;
  }
 
  cacheQuery(query: string, results: SearchResult[]): void {
    this.queryCache.set(query, { results, cachedAt: Date.now() });
  }
 
  getCachedEmbedding(text: string): number[] | null {
    return this.embeddingCache.get(text) ?? null;
  }
 
  cacheEmbedding(text: string, embedding: number[]): void {
    // Embeddings don't expire — content changes trigger re-index
    this.embeddingCache.set(text, embedding);
  }
 
  invalidateDocument(documentId: string): void {
    // Remove all cached queries that included this document
    for (const [key, value] of this.queryCache.entries()) {
      if (value.results.some((r) => r.id === documentId)) {
        this.queryCache.delete(key);
      }
    }
  }
}

Common Mistakes

Embedding full documents without chunking: Embeddings of very long documents lose precision. Chunk documents into 256-512 token windows and embed each chunk.
No cache invalidation on document update: Stale embeddings return outdated results. Trigger re-indexing immediately when a document is updated or deleted.
Pure semantic search for exact-match queries: Semantic search scores product codes and exact names poorly. Always use hybrid search.
Re-ranking every query: LLM re-ranking adds 500ms-2s of latency. Only re-rank when the initial semantic scores are close (within 0.05 of each other).
Ignoring embedding model versioning: Switching embedding models makes all stored vectors incompatible. Re-index the entire corpus when upgrading the model.

Best Practices

Chunk documents into 256-512 token windows with 50-token overlap to preserve context at chunk boundaries.
Store the embedding model name and version alongside each vector so you can detect version drift in your index.
Use approximate nearest-neighbor (ANN) indexes (HNSW, IVF) in Pinecone, Weaviate, or pgvector for sub-100ms search at millions of documents.
Cache query embeddings in Redis with a 5-minute TTL — the same query phrased identically should not re-compute the embedding.
Monitor search quality with explicit user feedback (thumbs up/down) mapped to query-result pairs for offline evaluation.

Key Takeaways

Hybrid search (semantic + BM25) consistently outperforms pure vector search because BM25 handles exact-match queries that semantic similarity scores poorly.
Chunking documents into 256-512 token windows before embedding improves recall because full-document embeddings dilute the signal for specific passages.
Query rewriting with an LLM before embedding expands recall by 10-25% for vague or underspecified user queries.
LLM re-ranking should only run on the top 10 candidates, not the full result set, to keep latency under 2 seconds.
Embedding model versioning is a hidden operational risk — switching models without re-indexing produces random-looking ranking degradation.
Cache both query embeddings (5-minute TTL) and search results (5-minute TTL) to dramatically reduce embedding API costs at scale.
An ANN index (HNSW) is mandatory for production — exact nearest-neighbor search is O(n) per query and will not scale beyond a few thousand documents.
Track search quality explicitly with relevance feedback or click-through rate per query, not just infrastructure metrics like latency.

Introduction#

Why This Matters#

Embedding Pipeline#

Hybrid Search: Semantic + Keyword#

Query Rewriting with LLM#

Re-ranking with LLM#

Search Result Caching#

Common Mistakes#

Best Practices#

Key Takeaways#

Related reading