Retrieving the Right Passages

Retrieval is the bottleneck

Here is the uncomfortable truth of RAG: if the right passage isn't in what you retrieve, no amount of clever prompting will save the answer. The generator can only ground itself in what it was handed. So retrieval quality is the single biggest lever on the whole system — invest here before anything else.

Semantic search from the last guide is the workhorse: embed the query, ask the vector store for its nearest neighbours. But pure meaning-matching has blind spots, and fixing them is what this guide is about.

Dense vs sparse

There are two great families of retriever, and knowing their trade-off is core RAG literacy. Dense retrieval is the embedding approach: it matches meaning, so it finds paraphrases and synonyms — but it can fumble exact strings it never learned, like a part number `RX-7G` or a rare surname.

Sparse retrieval is classic keyword search (think BM25): it matches words and rewards exact overlap. It nails that part number and any literal term — but it misses a question phrased with entirely different vocabulary from the document. Dense is strong where sparse is weak, and vice versa. That complementarity is the whole motivation for combining them.

Hybrid retrieval

Hybrid retrieval runs both a dense and a sparse search and merges their results. A common merge is reciprocal rank fusion: each candidate scores by where it ranks in each list, so a chunk that both methods like floats to the top, while a one-method fluke is discounted. In practice hybrid is a reliable default — it recovers exact-match cases dense search drops without losing the semantic ones.

Reciprocal rank fusion: each candidate scores by where it ranks in each list, so a passage near the top of either retriever rises in the merge.

Reranking

First-stage retrieval is built for speed, so it's a little coarse — it scores the query and each chunk separately and compares the vectors. Reranking adds a precise second pass: retrieve a generous set (say the top 30), then run a slower cross-encoder that reads the query and each candidate together and scores true relevance. Keep the best handful. You get the recall of fast retrieval with the precision of careful reading.

1. Retrieve broadly — pull 20–50 candidates with hybrid search, favouring recall.
2. Rerank — score each candidate against the query with a cross-encoder; sort by that score.
3. Cut to the top k — pass only the 3–8 best chunks to the generator, so the prompt stays focused.

Measure before you tune. Pull a few real questions, look at exactly which chunks come back, and ask: *was the answer in there?* If retrieval keeps missing, no downstream trick helps — adjust chunking, embeddings, or add reranking until the right passage reliably shows up.