RAG fixes two problems simultaneously: a language model's training data goes stale, and a model under-specified on a fact will confidently generate a plausible wrong answer. Retrieval gives it a source to draw from instead. It introduces one new way to fail: bad retrieval with false confidence attached.
The pipeline is seven steps and conceptually simple. Getting retrieval right is not. Most production mistakes aren't in the LLM call. They're in the three steps before it.
- —RAG fixes stale training data and hallucination simultaneously by giving the model an actual source to draw from instead of generating plausible-sounding answers from memory.
- —Bad retrieval is worse than no RAG: it adds latency and cost while producing ungrounded answers, now with a citation attached to give them unearned authority.
- —Most production RAG failures happen in the three steps before the LLM call: chunking strategy, embedding quality, and retrieval precision, not in the generation step itself.
- —Hybrid retrieval (keyword plus semantic search) with a reranking step consistently outperforms pure vector search for production use cases with mixed query types.
RAG (Retrieval-Augmented Generation) pairs a language model with an external retrieval system. Instead of answering purely from memorised training data, the model first retrieves relevant documents from a knowledge source, then generates its answer using those retrieved documents as grounding context. The answer cites specific sources. You can check it.
Staleness: model training has a cutoff. RAG works with information from after that cutoff, or with private data the model never saw.
Hallucination: without retrieval, a model under-specified on a fact generates a plausible-sounding wrong answer. With retrieval, it has an actual source.
False confidence: a RAG system with poor retrieval is worse than no RAG. It adds latency and cost while producing an ungrounded answer, now with a citation attached to give it unearned authority.
Fine-tuning changes model behaviour: tone, format, task specialisation. RAG changes what facts the model has access to. If you need the model to know something specific, use RAG. If you need it to behave differently, fine-tune. RAG is also cheaper and faster to update: swap or add documents rather than retraining. This matters most when the underlying information changes frequently.
Seven steps, two phases. Steps 1–4 run at ingestion time (once per document, or when documents update). Steps 5–7 run at query time for every user request. The quality of the query-time steps depends entirely on how well the ingestion steps were done.
Collect and clean source documents — PDFs, markdown, web pages, database records. Quality here determines quality throughout. Cleaning means removing headers and footers, stripping embedded-image text that won't extract, normalising encoding.
Split documents into smaller passages. Retrieval works at the chunk level. Chunk too small: context gets severed. Chunk too large: relevance gets diluted and tokens get wasted. Chunk size is a real trade-off — the right answer depends on your document type and query patterns.
Convert each chunk into a vector representation capturing semantic meaning. The embedding model choice affects retrieval quality directly. Domain-specific embedding models often outperform general-purpose ones on specialized content.
Store vectors in a vector database for fast similarity search. Choice of database affects latency, scale, and operational complexity — but it's a secondary concern until you've got the first three steps right.
At query time, embed the user's query and compare against the index to surface the most relevant chunks. This is where hybrid search pays off: vector search finds conceptually related content, keyword search catches exact terms the embedding might blur.
A secondary model reorders the retrieved candidates by relevance before the final set is sent to the LLM. Common pattern: retrieve top 20, rerank down to 3–5. First-pass vector similarity is a rough filter, not a precise one.
The query plus the final selected chunks are sent to the LLM, which generates the answer grounded in that context. This is the step most people optimise first. It's usually the wrong place to start.
Fixed-size chunking — split every N characters — is a fast start and a low ceiling. The chunk boundary is arbitrary, so it frequently falls mid-sentence, severing context and making the extracted chunk ambiguous without what came before or after.
Semantic chunking computes embeddings sentence-by-sentence and starts a new chunk when semantic similarity between adjacent sentences drops below a threshold. Boundaries correspond to where meaning actually shifts. Retrieving a semantically coherent chunk against a semantically similar query produces meaningfully better results — especially on documents that shift topic mid-section.
Dense vector search and sparse keyword search (BM25) are complementary, not competing. Vector search finds conceptually related content even without matching words — useful for paraphrase, synonyms, and domain inference. Keyword search catches exact terms, proper names, codes, and identifiers that embeddings can blur by collapsing similar-sounding but distinct things.
Hybrid retrieval combined with reranking is the default for production systems in 2026. It's not exotic — it's the sensible baseline. The common reranking pattern: retrieve a broad candidate pool (~20), rerank down to 3–5 strong candidates, and send only those to the LLM. Reranking larger pools (100+) rarely pays off — useful signal concentrates at the head of the distribution.
Raw user queries are often poorly shaped for retrieval. Two techniques address this:
Generate several reformulations of the same question to widen the retrieval net. A user asking "how does this work with large files" might retrieve more with "performance characteristics on large datasets" or "scalability with file size" added as parallel queries.
Have the model generate a hypothetical answer first, then embed and retrieve using that rather than the raw query. The hypothetical answer contains domain language the user's original question likely lacks. It's counterintuitive — you generate before you retrieve — but it works well for knowledge-intensive queries where the user's vocabulary doesn't match the document vocabulary.
Know these exist. Don't reach for them by default.
The model iteratively decides to run multiple retrieval steps, reformulating its own queries and reasoning across rounds before answering. Strong for multi-step questions where a single retrieve-then-generate pass misses intermediate context.
Builds a knowledge graph over source data and retrieves via graph traversal rather than similarity search. The right tool when questions are relationship-heavy: 'how do these three entities connect' — where the answer isn't in any single chunk and vector similarity won't find it.
A query classifier routes each incoming question to the cheapest pipeline that can handle it. Simple factual questions go to fast vector RAG. Complex multi-step questions go to agentic RAG. Relationship questions go to GraphRAG. Emerging as the sensible default for production systems in 2026 — most real-world queries are simple and don't need the expensive path.
The most common production mistake is not under-engineering RAG — it's over-engineering it. Start with hybrid retrieval plus a reranker. Measure retrieval quality before adding anything else. Only add query transformation, agentic loops, or graph structures once metrics prove the simpler approach genuinely falls short for a specific, real class of queries — not because it seems more sophisticated.
The right architecture depends on scale. One setup I use routinely; one for production applications.
One markdown file per topic — not one giant document. Retrieval works at the chunk level, and a single well-scoped file chunks more predictably than a sprawling one.
Use headings as natural chunk boundaries. A model retrieving 'just the pricing section' should find it by heading alone.
No exotic formatting — strip tables-as-images, embedded screenshots of text, heavy nested formatting. These are extraction failures identical to the Gate 3 problem in AEO.
Plain files on disk (e.g. Obsidian 30-library/) are sufficient at this scale. Claude's context window plus good file organization functions as your retrieval layer.
Update files in place rather than creating 'v2' copies. Retrieval should never have to guess which version is current.
Define the question set first. What will this system actually be asked? Build against real query patterns, not hypothetical ones.
Choose a vector database on latency, pricing, and indexing behavior for your actual data volume — not on whichever is most discussed. Chroma and pgvector for smaller deployments; Pinecone, Weaviate, Qdrant, or Milvus for larger scale.
Build hybrid retrieval + reranking before anything fancier. This alone puts you ahead of most production deployments.
Instrument evaluation from the start — don't bolt it on after launch.
Only add complexity (agentic loops, graph retrieval, query transformation) once evaluation data shows the simple pipeline is genuinely insufficient for a real class of queries.
Bad chunking
Context lost at boundaries, or chunks too large to be precise. The most common retrieval quality problem — usually fixed by moving from fixed-size to semantic chunking.
Irrelevant embeddings
Retrieval surfaces topically adjacent content that doesn't actually answer the question. Often a sign of a generic embedding model on domain-specific content, or missing query transformation.
Outdated index
The knowledge base goes stale even though the system is technically "using RAG." Retrieval is only as current as the last index update.
Ambiguous queries
Vague questions retrieve vague or scattered results. Query expansion and HyDE exist specifically for this — but the first fix is often pushing back on the query design.
False confidence
A model will generate a fluent answer from irrelevant retrieved context if not explicitly instructed to say when context doesn't address the question. Faithfulness evaluation catches this. Visual inspection usually doesn't.
"It looks like it's working" is not evaluation. Systematic evaluation from day one is becoming standard — a majority of new RAG deployments now build it in from the start. Three things to measure:
Are the retrieved chunks actually relevant to the query? Measure precision and recall against a labeled test set. This is the most important metric — if retrieval is bad, generation can't save it.
Does the generated answer reflect what's in the retrieved chunks, or did the model drift from them? A model that ignores its own retrieved context is producing ungrounded output — retrieval failed to constrain it.
Does the final answer address what was actually asked? Distinct from faithfulness — an answer can be faithful to the retrieved context while still not answering the question if retrieval surfaced the wrong chunks.
RAGAS is a commonly used open framework for scoring these dimensions systematically.
Core RAG mechanism per Lewis et al. (2020), the original RAG paper. Implementation patterns — hybrid retrieval, reranking ratios, adaptive routing, evaluation-first deployment — drawn from practitioner sources reporting on production trends (2026). Treat specific tool and vendor comparisons as directional rather than definitive; verify against current documentation before committing to a specific vector database or framework.
Lewis et al. · NeurIPS 2020
The original RAG paper — establishes the retriever-generator architecture and shows it outperforms parametric models on open-domain QA while remaining updateable without retraining.
Karpukhin et al. · EMNLP 2020
Introduces the bi-encoder retrieval approach that powers most production RAG systems — shows dense retrieval substantially outperforms BM25 on knowledge tasks.
Daniel Jurafsky & James Martin · Stanford University (draft)
The standard NLP textbook, with up-to-date chapters on dense retrieval, vector semantics, and language model architecture — freely available at web.stanford.edu/~jurafsky/slp3.