I keep getting pulled into the same fire drill. A team ships a RAG assistant. It demos beautifully. Three weeks into production, someone notices it keeps citing a deprecated doc, or missing a statute, or answering with confidence while leaving out the part that actually mattered.
The team’s first instinct is to add a reranker. When that underwhelms, they tune chunk sizes. When that underwhelms, they reach for a bigger context window and stuff more in. None of it moves the needle, because none of it touches the actual problem.
Your RAG pipeline is a search problem. You just refused to solve it as one.
A reranker cannot save retrieval that was never good at search
Here is the uncomfortable mechanic. A reranker only reorders the candidates retrieval already handed it. Reranking is bounded by first-stage recall: a reranker can’t promote a document that never appears in the candidate pool. If the answer-bearing passage never makes the shortlist, your expensive cross-encoder is polishing the wrong documents.
Atlan made this plain in a recent piece titled, bluntly, why reranking can’t fix bad retrieval. They walked through a real enterprise case: a semantic search returned 10 assets and 7 glossary terms with near-identical relevance scores, none in the required certified-production scope, so the assistant fell back to keyword SQL. The scope data already existed. Retrieval just never used it. The fix was pushing scope constraints into retrieval, not swapping the reranker.
That is the whole lesson in one anecdote. The reranker was never the bottleneck. Search was.
And it gets worse when you overcorrect. Teams assume that if recall is the problem, they should just retrieve more. Adding distracting documents can measurably cut answer accuracy, and deeper pools don’t automatically help. Past a certain point, piling on more candidates can pull recall below what retrieval alone would return. More is not better. Right is better.
The two failure modes you are not measuring
Production retrieval breaks in exactly two ways, and most teams can’t tell you which one is hitting them because they never instrumented it.
The first is a recall problem. The answer exists in your corpus, but retrieval never surfaces the answer-bearing passage. The second is a ranking problem. Retrieval returns candidates, but the best evidence is buried below irrelevant or loosely related passages.
These demand opposite fixes. A recall failure means your query never reached the gold document, so no reorder on earth helps. A ranking failure means the right doc is there, just at position 23, drowning under near-misses the model latched onto first. Reranking helps the second case and is useless for the first.
So before you buy a reranker, buy a number. Measure recall at k and context recall on a labeled set. The research community has drawn a usable line here: below recall@50 of 0.85, fix retrieval and chunking instead. If you are under that threshold, a reranker is a distraction.
This is not exotic. This is information retrieval, a field that has been rigorous for decades. We are relearning it the expensive way because “RAG” made a lot of engineers forget that the R stands for retrieval, and retrieval is search.
Fix the fundamentals in order
Here is the order that actually works, from the forward-deployed seat.
Start with chunking, because it defines the units you can even retrieve. Chunking defines the units of knowledge your system can retrieve. Bad chunking guarantees failure even with strong embeddings, hybrid search, and expensive rerankers. A high-leverage move costs you almost nothing: especially chunk size and overlap, metadata filtering, and reranking are where the fastest wins live, and heading-aware chunks with the full section path stored as metadata punch far above their weight.
Then go hybrid. Short, underspecified, weakly lexicalized queries are what real users type, and frequently failed to surface the correct knowledge-base articles, even when using strong hybrid retrievers. These retrieval failures propagated downstream, constraining the generator’s output and resulting in incomplete or low-quality responses despite capable LLMs. Lexical and semantic each catch what the other misses. Give both a vote.
Then make metadata do work. Scope, recency, access control, entity type: push these into retrieval as filters, not as afterthoughts you hope the reranker infers.
Only now do you add a reranker, and only as what it actually is. A reranker earns its place by sharpening the last step before generation, but it can only reorder what retrieval hands it. Treat it as a precision layer on top of a strong retrieval layer, not a substitute for one.
The Cost of Doing Nothing here is not an embarrassing demo. It is an agent that sounds authoritative and is quietly, systematically incomplete. That is the failure mode that burns trust fastest and shows up last, long after the launch Slack thread went quiet.
Stop stacking band-aids on bad search. Instrument recall, fix chunking and metadata, go hybrid, then rerank. The teams who win the agentic era will be the ones who remembered that retrieval was always a search problem, and finally solved it.