Reranking
Also known as: re-ranking, search reranking, cross-encoder reranking, reranker
Reranking is a second pass in a search system: a fast first step finds a broad set of possibly relevant results, then a more careful AI model reads each one alongside the query and reorders them by how well they actually answer it.
In AI chatbots that answer from documents, reranking helps put the best passages in front of the model. While reranking adds a bit of delay and cost, it often improves answer quality more than switching to a bigger language model.
Reranking, explained at three levels
At home
Imagine asking a librarian for books about fixing a leaky roof. First they quickly grab twenty books with promising titles. Then they flip through each one with your question in mind and put the three most useful on top. Reranking is that second, careful step. A fast search finds candidates, then a slower AI model reads each one and puts the best first.
At work
Reranking is a second stage in a search system: a fast first search gathers a broad set of possible results, then an AI model scores each one against the query and reorders them by true relevance. It sharpens company search tools and AI assistants that answer from documents, often improving answer quality more cheaply than switching to a bigger language model.
At home
The first search compares your question and each document separately, using stored embeddings or keywords, which is fast but rough, like judging books by their covers. A reranker instead reads your question and one document together, so it can notice whether the document actually answers what you asked, not just whether it uses similar words. That is far slower per document, so it only checks the top few dozen results.
At work
A typical pipeline retrieves the top 50 to 100 candidates with vector or hybrid search, scores each with a cross-encoder reranker, and passes the top three to ten to the language model. Options include Cohere Rerank, open models such as BAAI's bge-reranker and Jina's rerankers, and using an LLM itself to rank. Rerankers can also drop results below a relevance threshold, so the model isn't fed misleading context.
At home
In a chatbot, whatever passages reach the model become its evidence, and it will answer from them confidently. Without reranking, the top passage may just share vocabulary with your question, like a refund policy for a different product. While the model's answer sounds right, it rests on the wrong source. In my opinion, that explains a lot of confidently wrong chatbot answers.
At work
Cross-encoders are expensive because they process every query and document pair from scratch, so latency and cost grow with the number of candidates. Most also cap passage length at a few hundred to a few thousand tokens, and quality varies by language and domain, so French or legal text may need a multilingual or fine-tuned model. I'd measure recall and nDCG on your own queries before and after adding a reranker, and tune the candidate count.
Real-world examples
At home
Thomas searched his company's help centre and got the right article first, even though his wording didn't match its title. The search tool used a reranker to judge which result truly answered his question.
At work
Aisha builds a policy assistant for a Halifax hospital network's HR team. Adding a reranker between search and the language model noticeably reduced answers pulled from the wrong policy.
How it compares
| feature | Reranking | Embeddings |
|---|---|---|
| Role | Orders a short list carefully | Finds candidates quickly |
| Reads query and document together | Yes | No, compared separately |
| Speed | Slower per item | Very fast at scale |
| Accuracy | Higher | Good but rougher |
Embeddings cast the net. Rerankers pick the best catch.
For analysts
If an AI assistant keeps citing the wrong document, I'd test adding a reranker before upgrading the language model. It's usually cheaper and targets the real problem.
History & origin
Search engines have long used multi-stage ranking, with cheap methods narrowing millions of documents and costlier methods ordering the top results. In 2019, Rodrigo Nogueira and Kyunghyun Cho showed that a BERT model reading a query and passage together, called a cross-encoder, sharply improved ranking on Microsoft's MS MARCO benchmark. Toronto-based Cohere launched its Rerank API in 2023, and open-source rerankers from groups such as BAAI and Jina AI followed. As retrieval-augmented generation spread in 2023 and 2024, reranking became a standard step in RAG pipelines. Some teams now use language models themselves as rerankers. In my view, reranking is the most cost-effective fix for chatbots that 'almost' find the right answer.