Introduction
Retrieval-Augmented Generation (RAG) has become the default architecture for grounding large language models in domain-specific or up-to-date knowledge without retraining them. At the heart of every RAG pipeline sits a retrieval component, and the quality of that retrieval step determines, more than almost any other factor, whether the final generated answer is accurate, relevant, and trustworthy. Engineers building these systems quickly discover that "just embed everything and do a vector search" is a reasonable starting point but rarely the final architecture. The retrieval layer is where most of the real engineering decisions live, and those decisions compound as the system scales.
This article walks through the landscape of retrieval strategies used in RAG, starting from classic lexical search, moving through dense semantic embeddings, and ending with hybrid and late-interaction architectures that combine the strengths of both. Rather than treating semantic embeddings as a silver bullet, the goal here is to give engineers a mental model for when pure semantic search is sufficient, when it fails, and how to design systems that gracefully blend lexical precision with semantic recall. The examples use Python and TypeScript, reflecting the two ecosystems most RAG systems are built in today.
Context: Why Retrieval Strategy Is the Hard Part of RAG
When RAG first appeared in the literature - notably in the 2020 paper by Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" - the retrieval step was often treated as a solved problem: encode documents and queries into the same vector space, run nearest-neighbor search, and pass the top-k results to the generator. In practice, production systems reveal that this assumption breaks down in several common scenarios. Dense embeddings excel at capturing semantic similarity - "cardiac arrest" and "heart stopped" will be close in vector space - but they often underperform on queries that depend on exact terms: product SKUs, error codes, legal citations, or proper nouns that didn't appear frequently in the embedding model's training data.
This is not a new problem in information retrieval; it is a restatement of a tension that has existed since the earliest search engines. Lexical methods like TF-IDF and BM25 (Robertson and Walker's probabilistic relevance framework, later refined as BM25 in the 1990s and still the default scoring function in Elasticsearch and OpenSearch) are extremely good at exact and near-exact term matching. They are fast, interpretable, and require no training. What they lack is any understanding of meaning: a query for "automobile" will not match a document that only says "car" unless synonym expansion is configured manually. Semantic embeddings solve exactly this gap, which is why the two approaches are complementary rather than competing.
The practical consequence for engineering teams is that the choice of retrieval strategy is not a one-time architectural decision but an ongoing tuning problem tied to the nature of the corpus and the query distribution. A RAG system over internal engineering documentation, full of acronyms and ticket numbers, behaves very differently from a RAG system over customer support transcripts written in natural conversational language. Teams that treat retrieval as "pick an embedding model and a vector database" tend to hit a quality ceiling within the first few months of production use, usually surfaced as user complaints that "the bot didn't find the obvious answer" even though the document existed in the index.
Deep Technical Explanation: How Semantic Embeddings Actually Work
A semantic embedding model maps a piece of text - a word, sentence, paragraph, or document chunk - into a fixed-length dense vector, typically between 384 and 3072 dimensions depending on the model. The defining property of a good embedding space is that semantic similarity in natural language corresponds to geometric proximity in vector space, usually measured with cosine similarity or dot product. Modern embedding models are almost universally built on transformer encoders fine-tuned with contrastive learning objectives, where the model is trained to pull semantically similar pairs closer together and push dissimilar pairs apart. Sentence-BERT (Reimers and Gurevych, 2019) was one of the foundational papers that made this practical by showing how to derive fixed-size sentence embeddings from BERT efficiently, rather than running expensive cross-encoder comparisons for every pair.
In production RAG systems, the embedding model is typically used twice: once offline, to encode every chunk of the knowledge base into vectors stored in a vector index, and once online, to encode the incoming user query into the same space so that an approximate nearest-neighbor search can retrieve the closest document vectors. The approximate part matters at scale - exact nearest-neighbor search over millions of vectors is computationally expensive, so production vector databases (Pinecone, Weaviate, Qdrant, Milvus, or pgvector on top of PostgreSQL) use approximate algorithms such as HNSW (Hierarchical Navigable Small World graphs) or IVF (Inverted File Index) to trade a small amount of recall for large gains in query latency. Understanding that this is an approximation, not an exact computation, is important when debugging retrieval quality issues, since the "nearest" result returned by the index is not guaranteed to be the mathematically closest vector in the dataset.
The Spectrum of Alternatives: From Lexical to Hybrid to Advanced Semantic
It helps to think of retrieval strategies as points along a spectrum rather than a binary choice, since each step up in complexity trades implementation simplicity for better handling of specific failure modes.
Lexical search (TF-IDF, BM25) sits at the simple end of the spectrum. BM25 scores documents based on term frequency, inverse document frequency, and document length normalization, and it remains the default ranking algorithm in Elasticsearch, OpenSearch, and Apache Solr. It requires no training data, no GPU, and is fully explainable - you can always point to which terms drove a match. Its weakness is the vocabulary mismatch problem: if the query and the document use different words for the same concept, BM25 will not connect them.
Dense semantic embeddings are the next step up. A single embedding model (OpenAI's text-embedding-3 family, Cohere's embed models, or open-source options like BAAI's BGE models or sentence-transformers' all-MiniLM and mpnet variants) encodes the entire corpus once and supports approximate nearest-neighbor retrieval at query time. This is the approach most RAG tutorials demonstrate because it is conceptually simple and works impressively well on natural-language queries. Its main weaknesses are the exact-match problem described earlier and the fact that embedding quality is highly domain-dependent - a general-purpose embedding model trained mostly on web text can underperform badly on specialized legal, medical, or code-heavy corpora unless fine-tuned or paired with domain-adapted models.
Hybrid search combines lexical and semantic scores, typically using a fusion algorithm such as Reciprocal Rank Fusion (RRF), which merges two independently ranked result lists by summing a rank-based score for each document rather than trying to normalize two incompatible scoring scales directly. This is the approach increasingly treated as the production default: Elasticsearch, OpenSearch, Weaviate, and Qdrant all now ship native hybrid search support precisely because practitioners found that neither method alone was sufficient across diverse query types. Hybrid search captures BM25's precision on exact terms while retaining the embedding model's ability to surface semantically relevant but lexically dissimilar documents.
Late-interaction and cross-encoder reranking models represent the more complex end of the spectrum. ColBERT (Khattab and Zaharia, 2020) introduced late interaction, where instead of compressing a document into a single vector, it keeps token-level embeddings and computes a more fine-grained similarity at query time - a middle ground between the speed of single-vector embeddings and the accuracy of full cross-encoders. Cross-encoder rerankers (such as those built on top of MS MARCO fine-tuned BERT models, or commercial rerankers like Cohere Rerank) go further: they take the query and a candidate document together as a single input and produce a highly accurate relevance score, at the cost of being too slow to run over an entire corpus. The common production pattern is to use a cheap method (BM25, dense embeddings, or hybrid fusion) to retrieve a few hundred candidates, then apply a cross-encoder reranker to reorder just that shortlist before passing the top handful to the LLM.
Implementation: Building a Hybrid Retrieval Pipeline
The cleanest way to understand these trade-offs is to see them in code. The example below shows a hybrid retrieval function in Python that combines BM25 scoring with dense vector search and fuses the results using Reciprocal Rank Fusion, a technique simple enough to implement without a specialized library but effective enough that it's used internally by several commercial hybrid search products.
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
import numpy as np
class HybridRetriever:
def __init__(self, documents: list[str], embedding_model: str = "BAAI/bge-small-en-v1.5"):
self.documents = documents
self.tokenized_corpus = [doc.lower().split() for doc in documents]
self.bm25 = BM25Okapi(self.tokenized_corpus)
self.encoder = SentenceTransformer(embedding_model)
self.doc_embeddings = self.encoder.encode(
documents, normalize_embeddings=True, show_progress_bar=False
)
def _dense_search(self, query: str, top_k: int) -> list[tuple[int, float]]:
query_vec = self.encoder.encode([query], normalize_embeddings=True)[0]
scores = self.doc_embeddings @ query_vec
top_idx = np.argsort(scores)[::-1][:top_k]
return [(int(i), float(scores[i])) for i in top_idx]
def _lexical_search(self, query: str, top_k: int) -> list[tuple[int, float]]:
scores = self.bm25.get_scores(query.lower().split())
top_idx = np.argsort(scores)[::-1][:top_k]
return [(int(i), float(scores[i])) for i in top_idx]
def search(self, query: str, top_k: int = 10, rrf_k: int = 60) -> list[tuple[int, float]]:
dense_results = self._dense_search(query, top_k=50)
lexical_results = self._lexical_search(query, top_k=50)
# Reciprocal Rank Fusion: combine rank-based scores rather than raw scores,
# since BM25 and cosine similarity are not on comparable scales.
fused_scores: dict[int, float] = {}
for rank, (doc_id, _) in enumerate(dense_results):
fused_scores[doc_id] = fused_scores.get(doc_id, 0.0) + 1.0 / (rrf_k + rank + 1)
for rank, (doc_id, _) in enumerate(lexical_results):
fused_scores[doc_id] = fused_scores.get(doc_id, 0.0) + 1.0 / (rrf_k + rank + 1)
ranked = sorted(fused_scores.items(), key=lambda x: x[1], reverse=True)
return ranked[:top_k]
This implementation is intentionally minimal - in production, the BM25 index and the vector index would typically live in separate, purpose-built systems (OpenSearch for lexical, Qdrant or pgvector for dense) rather than in-process, and the fusion step would often be followed by a reranking pass. The TypeScript example below shows how a typical RAG orchestration layer wires retrieval into the generation step, including the reranking stage that narrows the fused candidate set before it reaches the LLM.
interface RetrievedChunk {
id: string;
text: string;
score: number;
}
async function retrieveAndRerank(
query: string,
hybridTopK: number = 30,
finalTopK: number = 6
): Promise<RetrievedChunk[]> {
// Step 1: cheap, high-recall hybrid retrieval (BM25 + dense, fused via RRF)
const candidates = await hybridSearchService.search(query, { topK: hybridTopK });
// Step 2: expensive, high-precision reranking over the small candidate set
const reranked = await rerankerClient.rerank({
query,
documents: candidates.map((c) => c.text),
topN: finalTopK,
});
return reranked.map((r) => ({
id: candidates[r.index].id,
text: candidates[r.index].text,
score: r.relevanceScore,
}));
}
async function buildContextForGeneration(query: string): Promise<string> {
const chunks = await retrieveAndRerank(query);
return chunks
.map((chunk, i) => `[Source ${i + 1}]\n${chunk.text}`)
.join("\n\n");
}
Separating the retrieval and reranking steps this way keeps each stage doing what it's best at: the hybrid retriever optimizes for recall over a large corpus cheaply, and the reranker spends its computational budget only on the handful of candidates that matter.
Trade-offs and Pitfalls
No retrieval strategy is free of downsides, and engineers who skip this analysis tend to discover the costs in production rather than during design. Dense embeddings carry a meaningful operational cost: every document update requires re-embedding, embedding models drift in quality between versions (switching embedding models typically requires re-indexing the entire corpus, since vectors from different models are not comparable), and GPU or API costs for embedding generation scale linearly with corpus size and update frequency. Hybrid search avoids embedding-only blind spots but introduces its own complexity - tuning the fusion weights between lexical and semantic scores, or the RRF constant, is an empirical process that depends on the query distribution and often needs to be revisited as usage patterns change.
A subtler pitfall is chunking strategy, which is frequently underestimated relative to the choice of embedding model itself. Chunks that are too small lose context and produce embeddings that don't capture the full meaning of a passage; chunks that are too large dilute the embedding with unrelated content and reduce retrieval precision. Teams often find that improving the chunking strategy - using semantic or structure-aware chunking rather than fixed character counts - yields larger quality gains than swapping to a more expensive embedding model. Similarly, over-reliance on cross-encoder reranking without first narrowing the candidate set with a cheap retriever creates unacceptable latency at scale, since cross-encoders scale poorly with the number of documents scored; the two-stage retrieve-then-rerank pattern exists specifically to avoid this trap.
Best Practices for Production RAG Retrieval
Based on the trade-offs above, a few practical guidelines consistently hold up across different domains and corpus types. First, default to hybrid search rather than pure semantic search unless the corpus and query distribution are both known to be purely natural-language with no exact-match requirements - most real-world corpora contain identifiers, names, or jargon that benefit from lexical matching. Second, treat chunking as a first-class design decision, not an afterthought; experiment with chunk size and overlap on a representative evaluation set before investing further effort in embedding model selection. Third, build a retrieval evaluation harness early, using a set of representative query-document relevance pairs and standard information retrieval metrics such as Recall@k, Mean Reciprocal Rank (MRR), or normalized Discounted Cumulative Gain (nDCG@k), so that changes to the retrieval pipeline can be measured objectively rather than judged anecdotally.
Fourth, when latency budgets allow it, add a reranking stage after hybrid retrieval, since the quality improvement from a cross-encoder reranker over a well-fused hybrid candidate set is often larger than any single change to the first-stage retriever. Fifth, monitor retrieval quality in production separately from generation quality - a wrong or irrelevant answer from the LLM is often a retrieval failure in disguise, and conflating the two makes debugging much harder. Logging which chunks were retrieved for each query, independent of what the LLM ultimately generated, is one of the highest-leverage observability investments a RAG team can make.
Key Takeaways
- Start with hybrid search (BM25 or similar lexical scoring fused with dense embeddings), not pure semantic search, for most real-world corpora.
- Invest in chunking strategy before swapping embedding models - it frequently has a larger impact on retrieval quality.
- Add a cross-encoder reranking stage over a narrowed candidate set when latency budgets allow it; don't rerank the full corpus.
- Build an evaluation harness using Recall@k, MRR, or nDCG@k before making retrieval architecture changes, so improvements are measurable.
- Log and monitor retrieval outputs independently of generation outputs to isolate whether failures originate in retrieval or in the LLM.
Conclusion
Semantic embeddings are a powerful default for RAG retrieval, but treating them as the entire solution misses the lesson that decades of information retrieval research already taught: lexical and semantic matching solve different problems, and the best production systems use both. The spectrum from BM25 to dense embeddings to hybrid fusion to late-interaction reranking is not a ladder where higher is always better - it's a set of tools with different cost, latency, and accuracy profiles that should be matched to the actual corpus and query patterns a system needs to handle.
Engineering teams that succeed with RAG in production tend to share a common pattern: they start simple, instrument retrieval quality from day one, and add complexity - hybrid fusion, reranking, domain-specific fine-tuning - only when evaluation data shows it's needed rather than because a more sophisticated architecture is available. That discipline, more than the choice of any single embedding model or vector database, is what separates RAG systems that work reliably from those that merely demo well.
References
- Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS. https://arxiv.org/abs/2005.11401
- Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP. https://arxiv.org/abs/1908.10084
- Khattab, O., & Zaharia, M. (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR. https://arxiv.org/abs/2004.12832
- Robertson, S., & Zaragoza, H. (2009). The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval.
- Malkov, Y. A., & Yashunin, D. A. (2018). Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE TPAMI. https://arxiv.org/abs/1603.09320
- Elasticsearch Documentation - BM25 similarity and hybrid retrieval. https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-similarity.html
- OpenSearch Documentation - Hybrid search. https://opensearch.org/docs/latest/search-plugins/hybrid-search/
- pgvector - Open-source vector similarity search for PostgreSQL. https://github.com/pgvector/pgvector
- Qdrant Documentation - Hybrid queries and fusion. https://qdrant.tech/documentation/concepts/hybrid-queries/
- Cohere Rerank Documentation. https://docs.cohere.com/docs/rerank
- Sentence-Transformers Library Documentation. https://www.sbert.net/
- Cormack, G. V., Clarke, C. L. A., & Buettcher, S. (2009). Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods. SIGIR.