Introduction
Retrieval-augmented generation sounds simple when described in one sentence - retrieve relevant documents, feed them to a model, generate a grounded answer - and that simplicity is exactly what makes naive RAG implementations disappoint so often in practice. The gap between a working RAG demo and a production RAG system that reliably answers real user questions comes down to a handful of specific, well-understood problems at each stage of the pipeline: how documents get chunked and indexed, how a user's query gets matched against that index, and how the initial retrieved set gets refined before it ever reaches the generation step. Each of these stages has its own well-documented failure modes and well-documented fixes, and understanding them as a connected pipeline, rather than a single black-box "retrieval" step, is what separates a RAG system that works reliably from one that merely worked in the demo.
This article works through that pipeline end to end - indexing decisions that shape everything downstream, retrieval techniques that address the specific mismatch between how users phrase questions and how documents are actually written, and re-ranking as the precision-adding step that closes the gap between "roughly relevant" and "actually the right answer." The goal is to leave you able to diagnose which specific stage is responsible when a RAG system underperforms, rather than treating "the retrieval isn't good enough" as an undifferentiated, unfixable complaint.
Context: Why Naive RAG Underperforms in Predictable Ways
The most fundamental problem in RAG retrieval, and the one that explains a large share of disappointing results, is the query-document gap: a user's query and the document that actually answers it are written in structurally different kinds of text, even when embedded in the same vector space. A question like "how do I fix a leaking kitchen faucet" and a repair guide's actual answer - a procedural, declarative passage about washers, O-rings, and shutoff valves - share the underlying topic but not the linguistic form, and dense embedding models, trained to capture semantic similarity, don't automatically compensate for this structural mismatch between a question and a statement that happens to answer it. This gap is invisible in a demo built around queries phrased similarly to the document content, and it becomes the dominant source of retrieval failure the moment real users phrase their questions in their own natural, conversational way.
The second fundamental problem is a granularity mismatch between what's good for embedding and what's good for generation. A chunk small enough to embed precisely - capturing one specific, focused idea - is often too small to give a generation model enough surrounding context to produce a complete, well-grounded answer; a chunk large enough to provide rich context for generation often dilutes the specific signal an embedding needs to match accurately against a focused query. This tension is not a minor implementation detail - it's a structural property of using the same chunk for both retrieval matching and generation context, and it's exactly the gap that techniques like parent-document and sentence-window retrieval, covered below, exist specifically to resolve.
The third problem is that a single retrieval pass, however well-tuned, tends to leave real precision on the table at exactly the position that matters most: the very top of the results, which is what a generation step actually sees given a typical context budget. A retrieval step tuned for broad recall - casting a wide enough net that the genuinely correct document is very likely somewhere in the results - is a different, and looser, standard than a retrieval step tuned for precision at the top of a small final list. Re-ranking exists specifically to bridge this gap: taking a broader, cheaper-to-produce candidate set and applying a more computationally expensive, more discerning model to sharpen precisely which few candidates actually make it into the generation prompt.
Deep Technical Explanation: Indexing, Retrieval Transformation, and Re-Ranking
Indexing strategy needs to address the granularity mismatch directly, and two established patterns do this well. Parent-document retrieval indexes small, precise child chunks for embedding and matching, while retrieving and passing the larger parent document (or a larger surrounding section) to the generation step once a child chunk matches - getting precise retrieval matching and rich generation context from the same underlying document without forcing one chunk size to serve both purposes. Sentence-window retrieval applies a similar idea at finer granularity: indexing individual sentences for maximally precise matching, then expanding to a window of surrounding sentences when passing context to the generation model, so a single, precisely-matched sentence doesn't reach the model stripped of the context needed to interpret it correctly. Both patterns share the same underlying principle: decouple what's optimal for matching from what's optimal for generation, rather than compromising both by using one chunk size for everything.
Hypothetical Document Embeddings (HyDE), introduced by Gao et al. in "Precise Zero-Shot Dense Retrieval without Relevance Labels," directly addresses the query-document gap at the retrieval stage. HyDE enhances document retrieval by leveraging an LLM to generate a hypothetical answer to a query, capitalizing on the model's ability to produce context-rich text that, once embedded, serves as a more effective basis for retrieval than the original query embedding alone. The mechanism is straightforward: rather than embedding the user's raw question, an LLM is asked to generate a plausible-sounding answer to that question - even though this generated answer isn't itself grounded in any retrieved document and may contain errors - and that hypothetical answer's embedding is used for the actual similarity search, because the resulting embedding identifies a neighborhood in the document embedding space from which similar, real documents can be retrieved based on vector similarity. The technique works precisely because it reframes retrieval as document-to-document matching rather than question-to-document matching, comparing like with like in exactly the way that closes the structural gap described above.
Multi-query retrieval is a complementary technique addressing a related but distinct problem: any single phrasing of a query, however well-transformed, might not be the phrasing that best matches the specific document that actually contains the answer. The multi-query technique enhances retrieval by expanding a single user query into multiple similar queries with the assistance of an LLM, running retrieval against each variant and combining the results - a pattern directly analogous to the hybrid-search fusion techniques covered elsewhere in AI search, but applied to multiple phrasings of the same underlying intent rather than to multiple distinct retrieval methods. Both HyDE and multi-query retrieval add a real cost - an extra LLM call before retrieval even begins - which needs to be weighed against the retrieval quality improvement for a given application's specific query patterns and latency budget, rather than applied indiscriminately to every query regardless of whether the original query would have retrieved well on its own.
Re-ranking, as the final stage before generation, applies a cross-encoder or similar model that directly scores the interaction between the query and each candidate document, rather than comparing independently pre-computed embeddings the way the initial retrieval pass does. This produces meaningfully better relevance judgments at the cost of being too computationally expensive to run against an entire corpus, which is exactly why it's applied only to the bounded candidate set - commonly the top 50 to 100 results - that survives the earlier, cheaper retrieval stage, sharpening precision specifically at the position that matters most for what a generation model will actually see.
Implementation: Building HyDE and Parent-Document Retrieval
The Python example below implements a HyDE retriever, generating a hypothetical answer before embedding and retrieval, followed by the actual document search against an existing index.
# hyde_retriever.py
from openai import OpenAI
client = OpenAI()
def generate_hypothetical_document(query: str) -> str:
# The hypothetical document doesn't need to be factually correct -
# it only needs to resemble, in form and vocabulary, the kind of
# document that would actually answer the query.
response = client.chat.completions.create(
model="gpt-5-mini",
messages=[
{"role": "developer", "content": (
"Write a short, plausible passage that would answer the "
"following question, as if it came from a reference document. "
"Do not mention that this is hypothetical."
)},
{"role": "user", "content": query},
],
)
return response.choices[0].message.content
def embed(text: str) -> list[float]:
response = client.embeddings.create(input=text, model="text-embedding-3-small")
return response.data[0].embedding
def hyde_retrieve(query: str, vector_store, top_k: int = 10):
hypothetical_doc = generate_hypothetical_document(query)
hyde_vector = embed(hypothetical_doc)
# The hypothetical document's embedding, not the raw query's, drives
# the actual similarity search - comparing document-to-document.
return vector_store.search(vector=hyde_vector, limit=top_k)
Parent-document retrieval, addressing the granularity mismatch at indexing time, requires tracking the relationship between small indexed chunks and their larger parent context. The TypeScript example below shows this pattern: indexing fine-grained chunks for precise matching while storing a reference back to the parent document, so retrieval returns richer context than the matched chunk alone.
// parentDocumentRetrieval.ts
interface ParentDocument {
id: string;
fullText: string;
}
interface ChildChunk {
id: string;
parentId: string;
text: string;
vector: number[];
}
async function indexWithParentReferences(
parentDoc: ParentDocument,
embedFn: (text: string) => Promise<number[]>,
vectorStore: { upsert: (chunk: ChildChunk) => Promise<void> },
childChunkSize = 200 // small, precise chunks for embedding accuracy
) {
const words = parentDoc.fullText.split(" ");
for (let i = 0; i < words.length; i += childChunkSize) {
const chunkText = words.slice(i, i + childChunkSize).join(" ");
const vector = await embedFn(chunkText);
await vectorStore.upsert({
id: `${parentDoc.id}_chunk_${i}`,
parentId: parentDoc.id, // the link back to richer generation context
text: chunkText,
vector,
});
}
}
async function retrieveWithParentContext(
query: string,
embedFn: (text: string) => Promise<number[]>,
vectorStore: { search: (vector: number[], limit: number) => Promise<ChildChunk[]> },
parentStore: Map<string, ParentDocument>,
topK = 5
): Promise<string[]> {
const queryVector = await embedFn(query);
const matchedChunks = await vectorStore.search(queryVector, topK);
// Retrieval matched on precise, small chunks, but generation receives
// the full parent document - richer context than the matched chunk alone.
const uniqueParentIds = [...new Set(matchedChunks.map((c) => c.parentId))];
return uniqueParentIds.map((id) => parentStore.get(id)!.fullText);
}
Trade-offs and Pitfalls
The most common mistake is applying query transformation techniques like HyDE or multi-query retrieval indiscriminately to every query, without validating whether the added LLM call and latency actually improve results for the application's specific query patterns. HyDE tends to help most when queries are short, conversational, and structurally distant from how source documents are written; for queries that already closely resemble the document content they're meant to match - precise technical lookups using the same terminology as the source material - the added generation step is pure overhead with little corresponding retrieval benefit. Validating each technique against a representative evaluation set, rather than assuming any published technique automatically transfers to a specific corpus and query distribution, is the only way to know whether the added cost is actually justified.
A second pitfall is choosing indexing granularity without validating it against real retrieval quality, defaulting to whatever chunk size a tutorial happened to use rather than testing what actually works for a specific corpus's structure. A corpus of long, cross-referencing legal clauses needs different indexing granularity than a corpus of short, self-contained support articles, and parent-document or sentence-window retrieval's benefit is highly dependent on how well the parent-child relationship actually captures the corpus's real structure - a poorly chosen parent boundary (splitting a document at an arbitrary point rather than a natural section break) can still leave a retrieved parent document incomplete or confusing.
A third pitfall is treating re-ranking as a substitute for adequate retrieval depth rather than a complement to it. A cross-encoder reranker can only reorder the candidates it's given - if the genuinely correct document didn't survive the initial retrieval pass due to too narrow a candidate pool, no amount of reranking sophistication downstream can recover it. This is the same candidate-pool-sizing concern that applies to any multi-stage retrieval pipeline, and it's worth validating explicitly: measuring what fraction of known-correct documents actually appear in the candidate pool passed to reranking, not just how well the reranker orders whatever pool it receives.
Best Practices for RAG Architecture
Decouple indexing granularity from generation context deliberately, using parent-document or sentence-window retrieval rather than forcing a single chunk size to serve both matching precision and generation richness. This single architectural decision resolves a structural tension that no amount of chunk-size tuning alone can fully escape, since any single chunk size is a compromise between two genuinely competing needs.
Validate query transformation techniques like HyDE and multi-query retrieval against your own evaluation data before adopting them as a default, since their benefit is highly dependent on how closely your actual queries already resemble your document content. A technique documented as improving retrieval on one benchmark corpus doesn't automatically transfer its benefit to a different corpus with different structural characteristics, and the added latency and cost of an extra LLM call before retrieval needs to be weighed against a measured, not assumed, improvement.
Size retrieval depth deliberately at every stage, validating that the initial retrieval pass's candidate pool actually captures the correct documents often enough before investing further effort in reranking sophistication. A reranker's ceiling is set entirely by the candidate pool it receives, and no amount of downstream refinement compensates for a first-pass retrieval depth that's too narrow to reliably include the right answer in the first place.
Analogies and Mental Models
The query-document gap that HyDE addresses is well captured by the difference between describing a movie you're trying to recall and reading an actual plot summary. If you describe a half-remembered movie in your own vague, conversational words - "the one where the guy loses his memory and has to figure out who he is" - a search matching your description directly against official plot summaries (which are written in a completely different, more formal register) can miss even though the movie you're thinking of is right there in the database. HyDE's trick is generating what a plausible plot summary of your description might look like first, then searching with that summary - comparing summary-to-summary instead of vague description-to-summary, which is a fundamentally easier and more reliable match.
Parent-document retrieval is well captured by how a library's card catalog historically worked: a small, precise index card captures just enough distilled information - title, subject, call number - to be quickly searchable, while the actual full book, richer and more complete than any card could be, is what a patron ultimately receives once the right card has been found. Nobody expects the index card itself to contain the answer to a research question; its job is precise, efficient matching, while the full book it points to is what actually provides the rich context needed. Parent-document retrieval applies exactly this division of labor to RAG: small chunks for precise matching, larger parent documents for rich generation context.
The 80/20 of RAG Architecture
A small number of architectural decisions account for most of a RAG system's real-world retrieval quality. Decoupling indexing granularity from generation context - through parent-document or sentence-window retrieval - is the single highest-leverage decision, because it resolves the structural tension between precise matching and rich context that no single chunk-size choice can fully escape on its own. Validating whether query transformation techniques like HyDE genuinely improve retrieval for your specific corpus and query patterns, rather than adopting them by default, is the second highest-leverage practice, since their benefit varies considerably depending on how much of a query-document gap actually exists for a given application.
The third disproportionately valuable practice is validating retrieval depth at the initial pass before investing further in reranking sophistication, since a reranker's precision ceiling is entirely bounded by whether the correct documents survived into its candidate pool in the first place. Everything beyond these three - sophisticated multi-hop retrieval for questions requiring several retrieval steps, fine-tuned domain-specific embedding models, elaborate ensemble reranking - adds genuine value for mature, high-stakes RAG systems, but is refinement layered on top of a foundation these three decisions already establish. Teams building their first serious RAG system get disproportionately more value from getting indexing granularity, validated query transformation, and adequate retrieval depth right than from any more advanced technique explored first.
Key Takeaways
- Decouple indexing granularity from generation context using parent-document or sentence-window retrieval, rather than forcing one chunk size to serve both precise matching and rich generation context.
- Consider HyDE when queries are phrased conversationally and structurally distant from how your source documents are written, but validate the improvement against your own evaluation data before adopting it as a default.
- Use multi-query retrieval to hedge against any single query phrasing failing to match the document that actually contains the answer.
- Validate that your initial retrieval pass's candidate pool actually contains the correct documents often enough before investing further effort in reranking - a reranker cannot recover documents that never made it into its candidate set.
- Treat every RAG optimization technique as a hypothesis to validate against your specific corpus and query distribution, not a universally-beneficial default to apply without measurement.
Conclusion
RAG architecture, done well, is the discipline of recognizing and addressing a small number of specific, well-understood gaps at each stage of the pipeline: the structural mismatch between how questions and documents are phrased, the tension between what's optimal for precise matching versus rich generation context, and the precision ceiling that only a dedicated reranking stage can push past. Indexing strategies like parent-document retrieval, query transformation techniques like HyDE and multi-query retrieval, and cross-encoder reranking each address one of these gaps specifically, and understanding them this way - as targeted fixes for named problems, not generic "make retrieval better" techniques - is what lets an engineer diagnose which stage is actually responsible when a RAG system underperforms.
None of these techniques is a substitute for validating their effect against real evaluation data specific to your own corpus and query patterns; each one's benefit depends on how much of the specific gap it addresses actually exists for your application. Teams that build RAG systems by understanding the pipeline this way - as a sequence of named, addressable problems rather than one opaque retrieval step - end up with systems that are genuinely debuggable when they underperform, rather than systems where "the retrieval isn't good enough" remains a vague complaint nobody can act on.
References
- Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE) - Gao et al., 2022: https://arxiv.org/abs/2212.10496
- ARAGOG: Advanced RAG Output Grading - comparative study of RAG techniques including HyDE and multi-query: https://arxiv.org/abs/2404.01037
- LlamaIndex - Parent Document Retriever documentation: https://docs.llamaindex.ai/en/stable/examples/retrievers/auto_merging_retriever/
- LangChain - MultiQueryRetriever documentation: https://python.langchain.com/docs/how_to/MultiQueryRetriever/
- Passage Re-ranking with BERT - Nogueira & Cho, 2019: https://arxiv.org/abs/1901.04085
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks - Lewis et al., 2020: https://arxiv.org/abs/2005.11401