LlamaIndex Complete Guide — Build Production RAG Apps in 2025

Introduction

Why This Matters

LlamaIndex is the most focused and developer-friendly framework for building Retrieval-Augmented Generation systems. While LangChain provides a broad orchestration platform, LlamaIndex goes deep on the retrieval layer — offering purpose-built abstractions for ingesting documents, building indexes, and querying them with sophisticated strategies.

RAG is now the dominant architecture for enterprise AI applications: it combines the language fluency of LLMs with accurate, up-to-date information from your own data. LlamaIndex makes building high-quality RAG systems dramatically faster, with sensible defaults that outperform naive implementations.

In 2024 and 2025, LlamaIndex added workflows, agents, and multi-modal support, evolving from a pure retrieval library into a complete AI application framework while retaining its RAG-first philosophy.

Installation and Setup

pip install llama-index llama-index-llms-openai llama-index-embeddings-openai

Configure globally to avoid passing model objects everywhere:

import os
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding
 
os.environ["OPENAI_API_KEY"] = "your-key"
 
Settings.llm = OpenAI(model="gpt-4o", temperature=0.1)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")
Settings.chunk_size = 1024
Settings.chunk_overlap = 128

Loading Documents

LlamaIndex provides 100+ data connectors called Readers:

from llama_index.core import SimpleDirectoryReader
from llama_index.readers.web import SimpleWebPageReader
 
# Load from a directory (PDFs, txt, markdown, docx)
documents = SimpleDirectoryReader("./data").load_data()
 
# Load from URLs
web_reader = SimpleWebPageReader(html_to_text=True)
web_docs = web_reader.load_data(["https://docs.example.com/api"])
 
# Load from LlamaHub (100+ connectors: Notion, Slack, GitHub, S3...)
# pip install llama-index-readers-notion
from llama_index.readers.notion import NotionPageReader
notion_docs = NotionPageReader(integration_token="...").load_data(page_ids=["abc123"])

Building Your First Vector Index

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
 
documents = SimpleDirectoryReader("./knowledge_base").load_data()
 
# Automatically chunks, embeds, and stores documents
index = VectorStoreIndex.from_documents(documents, show_progress=True)
 
# Persist to disk
index.storage_context.persist("./storage")

Load from persistent storage:

from llama_index.core import StorageContext, load_index_from_storage
 
storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)

Query Engines

Query engines are the primary interface for asking questions against an index:

from llama_index.core import VectorStoreIndex
 
index = VectorStoreIndex.from_documents(documents)
 
# Basic query engine
query_engine = index.as_query_engine(
    similarity_top_k=5,         # retrieve top 5 chunks
    response_mode="compact",    # compact | refine | tree_summarize
)
 
response = query_engine.query("What is the refund policy for annual plans?")
print(response.response)
 
# See the source nodes used for the answer
for node in response.source_nodes:
    print(f"Score: {node.score:.3f} | {node.text[:200]}")

Advanced Indexing Strategies

LlamaIndex supports multiple index types beyond vector search:

from llama_index.core import (
    VectorStoreIndex,
    SummaryIndex,
    KeywordTableIndex,
)
from llama_index.core.indices.knowledge_graph import KnowledgeGraphIndex
 
# Summary index: good for summarization tasks
summary_index = SummaryIndex.from_documents(documents)
summary_engine = summary_index.as_query_engine(response_mode="tree_summarize")
 
# Keyword index: fast, no embedding cost
kw_index = KeywordTableIndex.from_documents(documents)
 
# Knowledge graph: extract entities and relationships
kg_index = KnowledgeGraphIndex.from_documents(
    documents,
    max_triplets_per_chunk=5,
    include_embeddings=True,
)

Multi-Index Routing

The RouterQueryEngine intelligently selects which index to query based on the question:

from llama_index.core.query_engine import RouterQueryEngine
from llama_index.core.selectors import LLMSingleSelector
from llama_index.core.tools import QueryEngineTool
 
finance_engine = finance_index.as_query_engine()
hr_engine = hr_index.as_query_engine()
product_engine = product_index.as_query_engine()
 
router_engine = RouterQueryEngine(
    selector=LLMSingleSelector.from_defaults(),
    query_engine_tools=[
        QueryEngineTool.from_defaults(
            query_engine=finance_engine,
            name="finance",
            description="Financial reports, revenue, expenses, budgets"
        ),
        QueryEngineTool.from_defaults(
            query_engine=hr_engine,
            name="hr",
            description="HR policies, benefits, PTO, onboarding"
        ),
        QueryEngineTool.from_defaults(
            query_engine=product_engine,
            name="product",
            description="Product roadmap, features, release notes"
        ),
    ]
)
 
response = router_engine.query("How many vacation days do new employees get?")
# Automatically routes to hr_engine

Sub-Question Query Engine

Decompose complex questions into sub-questions for better accuracy:

from llama_index.core.query_engine import SubQuestionQueryEngine
from llama_index.core.tools import QueryEngineTool
 
tools = [
    QueryEngineTool.from_defaults(
        query_engine=index.as_query_engine(),
        name="company_docs",
        description="Company policies, procedures, and documentation"
    )
]
 
sub_question_engine = SubQuestionQueryEngine.from_defaults(
    query_engine_tools=tools,
    verbose=True
)
 
# Breaks this into: "What was revenue in 2023?" and "What was revenue in 2024?"
response = sub_question_engine.query(
    "Compare our revenue growth between 2023 and 2024 and explain what drove the change."
)

Persistent Vector Stores

For production, use a persistent vector store instead of in-memory storage:

import chromadb
from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.chroma import ChromaVectorStore
 
# Chroma (local)
chroma_client = chromadb.PersistentClient(path="./chroma_db")
collection = chroma_client.get_or_create_collection("company_docs")
vector_store = ChromaVectorStore(chroma_collection=collection)
 
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)
 
# Pinecone (cloud)
# pip install llama-index-vector-stores-pinecone
from llama_index.vector_stores.pinecone import PineconeVectorStore
import pinecone
 
pc = pinecone.Pinecone(api_key="your-key")
pinecone_index = pc.Index("your-index-name")
vector_store = PineconeVectorStore(pinecone_index=pinecone_index)

Evaluating RAG Quality

LlamaIndex integrates with evaluation frameworks to measure retrieval and answer quality:

from llama_index.core.evaluation import (
    FaithfulnessEvaluator,
    RelevancyEvaluator,
    AnswerRelevancyEvaluator,
)
 
faithfulness_eval = FaithfulnessEvaluator()
relevancy_eval = RelevancyEvaluator()
 
query = "What is the cancellation policy?"
response = query_engine.query(query)
 
# Is the answer faithful to the retrieved context?
faith_result = faithfulness_eval.evaluate_response(response=response)
print(f"Faithful: {faith_result.passing} | Score: {faith_result.score}")
 
# Is the retrieved context relevant to the query?
rel_result = relevancy_eval.evaluate_response(query=query, response=response)
print(f"Relevant: {rel_result.passing} | Score: {rel_result.score}")

Common Mistakes / Pitfalls

Using default chunk size (1024) for all document types — code needs smaller chunks, legal docs benefit from larger ones
Not persisting indexes — re-embedding all documents on every restart is slow and costly
Ignoring response_mode — tree_summarize is better for long-document summarization, compact for short Q&A
Skipping evaluation — retrieval quality varies significantly with chunking and embedding choices
Not filtering by metadata — use metadata filters to restrict retrieval to relevant document subsets

Best Practices

Always evaluate retrieval quality with a ground-truth Q&A set before going to production
Use metadata during ingestion (doc.metadata = {"source": "handbook", "year": 2025}) for filtered retrieval
Enable async with query_engine.aquery() in high-throughput production environments
Hybrid search (vector + BM25 keyword) consistently outperforms pure vector search on domain-specific content
For large document collections, use IngestionPipeline with a document store to deduplicate and batch efficiently

Key Takeaways

LlamaIndex is the leading Python framework for building RAG systems, with purpose-built abstractions for indexing and retrieval
The VectorStoreIndex is the default starting point — it automatically chunks, embeds, and stores documents
RouterQueryEngine can intelligently route queries across multiple specialized indexes without custom logic
SubQuestionQueryEngine decomposes complex multi-part questions into sub-queries for higher accuracy
LlamaIndex supports 100+ data connectors (LlamaHub) and integrates with Pinecone, Chroma, Weaviate, and all major vector stores
Response modes (compact, refine, tree_summarize) significantly affect answer quality depending on use case
Built-in evaluators for faithfulness, relevancy, and answer quality make it straightforward to measure RAG performance
For production systems, always persist indexes and use metadata filtering to improve both speed and retrieval precision

Introduction#

Why This Matters#

Installation and Setup#

Loading Documents#

Building Your First Vector Index#

Query Engines#

Advanced Indexing Strategies#

Multi-Index Routing#

Sub-Question Query Engine#

Persistent Vector Stores#

Evaluating RAG Quality#

Common Mistakes / Pitfalls#

Best Practices#

Key Takeaways#

Related reading