Build a RAG Application with LangChain and OpenAI — 2026 Guide

Introduction#

Why This Matters#

LLMs trained before a cutoff date cannot answer questions about your proprietary data, recent events, or internal documents. Retrieval-Augmented Generation (RAG) solves this by dynamically fetching relevant context at query time and injecting it into the prompt.

RAG is now the dominant architecture for enterprise AI assistants, customer support bots, and internal knowledge bases. Companies using well-tuned RAG pipelines report 60-80% reductions in hallucination rates compared to vanilla LLM responses. Getting the pipeline right — chunking strategy, retrieval method, and reranking — makes the difference between a demo and a production system.

This guide builds a complete RAG system from scratch: document loading, embedding, storage, retrieval, reranking, and a FastAPI endpoint ready for production deployment.

How RAG Works#

User Query → Embed query → Search vector DB → Retrieve top-K chunks
    → Inject into prompt → LLM generates grounded answer

The key insight: the LLM sees only the retrieved chunks plus the user question. It cannot hallucinate facts not present in the retrieved context (when the prompt is written correctly).

Step 1: Setup and Document Loading#

pip install langchain langchain-openai langchain-community chromadb pypdf tiktoken

import os
os.environ["OPENAI_API_KEY"] = "your-key-here"
 
from langchain_community.document_loaders import PyPDFLoader, DirectoryLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
 
# Load all PDFs from a directory
loader = DirectoryLoader('./docs', glob="**/*.pdf", loader_cls=PyPDFLoader)
docs = loader.load()
 
# Recursive splitter respects paragraph and sentence boundaries
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    separators=["\n\n", "\n", ".", "!", "?", " "],
)
chunks = splitter.split_documents(docs)
print(f"Created {len(chunks)} chunks from {len(docs)} documents")

Chunking rules: 500-1000 tokens per chunk works for most use cases. Use 15-20% overlap to prevent losing context at boundaries. Smaller chunks give more precise retrieval but less context per chunk.

Step 2: Embeddings and Vector Store#

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
 
embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
 
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db",
    collection_name="my_docs"
)

Step 3: MMR Retriever#

# Maximum Marginal Relevance — diverse, non-redundant results
retriever = vectorstore.as_retriever(
    search_type="mmr",
    search_kwargs={
        "k": 6,
        "fetch_k": 20,
        "lambda_mult": 0.7,  # 1.0 = pure similarity, 0.0 = pure diversity
    }
)

Step 4: RAG Chain#

from langchain_openai import ChatOpenAI
from langchain.prompts import PromptTemplate
from langchain.chains import RetrievalQA
 
llm = ChatOpenAI(model="gpt-4o", temperature=0)
 
prompt_template = """Use the following context to answer the question.
If the answer is not in the context, say you don't have enough information. Do not fabricate.
 
Context:
{context}
 
Question: {question}
 
Answer:"""
 
prompt = PromptTemplate(template=prompt_template, input_variables=["context", "question"])
 
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=retriever,
    chain_type_kwargs={"prompt": prompt},
    return_source_documents=True,
)
 
result = qa_chain.invoke({"query": "What is the refund policy?"})
print(result["result"])
for doc in result["source_documents"]:
    print(f"  Source: {doc.metadata.get('source')}, page {doc.metadata.get('page')}")

Step 5: Reranking (Production Essential)#

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder
 
model = HuggingFaceCrossEncoder(model_name="BAAI/bge-reranker-base")
compressor = CrossEncoderReranker(model=model, top_n=3)
 
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=retriever
)

Reranking corrects the gap between vector similarity and true relevance. A chunk that is semantically close to the query may not actually answer it — the cross-encoder reranker scores true relevance, not just embedding proximity.

Step 6: FastAPI Endpoint#

from fastapi import FastAPI
from pydantic import BaseModel
 
app = FastAPI()
 
class QueryRequest(BaseModel):
    question: str
 
@app.post("/ask")
async def ask(req: QueryRequest):
    result = qa_chain.invoke({"query": req.question})
    return {
        "answer": result["result"],
        "sources": [
            {"file": doc.metadata.get("source"), "page": doc.metadata.get("page")}
            for doc in result["source_documents"]
        ]
    }

Common Mistakes / Pitfalls#

• Chunk size too large — retrieves broad context but loses precision; use 500-1000 tokens
• No chunk overlap — context at chunk boundaries gets lost; always use 15-20% overlap
• Using cosine similarity alone without reranking — retrieved chunks are similar but not always relevant
• Skipping source attribution — users need to trust and verify RAG answers; always return sources
• Not evaluating the pipeline — deploy RAGAS or a similar eval framework before going live

Best Practices#

• Use temperature=0 for factual RAG to reduce creative fabrication
• Add metadata filters to the retriever to scope queries to the right document set
• Log every retrieved chunk and LLM response for debugging and quality monitoring
• Evaluate with RAGAS metrics: faithfulness, answer relevancy, context precision, context recall
• Implement incremental indexing so new documents are embedded without re-processing the entire corpus

Key Takeaways#

• RAG lets LLMs answer questions about data beyond their training cutoff without fine-tuning
• Chunk overlap of 15-20% is critical to prevent context loss at document boundaries
• Maximum Marginal Relevance (MMR) retrieval returns diverse results, avoiding redundant chunks
• Reranking with a cross-encoder model is a production necessity that significantly improves answer quality
• The "stuff" chain type works for short context; use "map-reduce" or "refine" for long documents
• RAGAS provides four key metrics: faithfulness, answer relevancy, context precision, and context recall
• Setting temperature=0 on the LLM dramatically reduces hallucination in RAG systems
• Always return source documents with answers so users can verify claims independently