Do I need a reranker to build RAG?

No. Start with embeddings alone and measure. Add a reranker when you can see the failure it fixes — the right chunk retrieved but ranked below your cutoff. Adding one before you have that evidence is premature.

Can one model do both?

No, they are different architectures. Some families publish an embedding model and a reranker trained together, which pair well, but you load and run them separately.

How much does a reranker slow things down?

It adds one forward pass per candidate, so latency scales with your shortlist size. Rerankers are small compared with generation models, and the pass usually costs less than generating the answer — but rerank a hundred candidates rather than fifty and you feel it.

What about hybrid search?

Combining vector search with keyword search improves recall, particularly for exact terms, names and codes that embeddings handle poorly. It is complementary to reranking, not an alternative: hybrid improves what reaches the shortlist, reranking improves the order within it.

Retrieval

Embedding Model vs Reranker

An embedding model finds candidates fast across millions of documents. A reranker reads the query and each candidate together and scores them properly. They are stages, not alternatives.

The short answer

These are not competing choices — they are two stages of the same pipeline. An embedding model encodes the query and every document separately, so document vectors can be computed once and searched in milliseconds across millions of items. A reranker reads the query and a candidate together, which is far more accurate and far too slow to run over a whole corpus. The standard pattern is to retrieve fifty to a hundred candidates with embeddings, then rerank them down to the handful you send the model.

Embedding model

Fast, precomputable, scales to millions. Finds candidates.

Reranker

Slow, accurate, per-query. Puts the right candidate first.

Embedding model vs Reranker at a glance

Factor	Embedding model	Reranker
Architecture	Bi-encoder — query and document encoded independently	Cross-encoder — query and document encoded together
Precomputable	Yes, index documents once	No, every pair is scored at query time
Scale per query	Millions of documents via vector search	Tens to low hundreds of candidates
Output	A vector, compared by cosine similarity	A relevance score for one query-document pair
Storage	One vector per chunk, persisted in an index	Nothing stored
Where it runs	Indexing time and query time	Query time only, after retrieval

The structural difference, and why it decides everything

A bi-encoder turns a query into a vector and a document into a vector, independently, and compares them. Because the document never sees the query, every document vector can be computed once, at indexing time, and stored. Searching is then a nearest-neighbour problem over vectors, which is extremely fast even across millions of items.

That independence is also the weakness. The document's representation has to be a single point that works for every possible query — it cannot emphasise the part relevant to yours, because it was computed before your query existed.

A cross-encoder reads both together in one pass, so attention runs across the query and the document at the same time and the score reflects their actual interaction. That is substantially more accurate, and it means nothing can be precomputed: every query requires a fresh forward pass per candidate.

Two stages, and why the first one has a different job

Because reranking costs a forward pass per candidate, you cannot rerank a corpus. You can rerank a shortlist. So retrieval's job is not to get the answer first — it is to get the answer somewhere in the top fifty.

That reframes what you optimise. Retrieval is judged on recall at the shortlist size: did the right chunk make the cut at all? Reranking is judged on precision at the top: is it now first? Tuning your embedding model for precision at rank one is optimising the wrong stage.

A typical configuration retrieves fifty to a hundred candidates and reranks to three or five. The shortlist size is your latency dial — more candidates means better recall and a longer rerank pass.

What a reranker is worth in practice

The failure it fixes is specific and common: the right chunk is retrieved but ranked sixth, so it falls outside the top five you pass to the model, and the model answers from worse context or says it does not know. The retrieval was fine; the ordering was not.

It also compensates for imperfect chunking. Embeddings are sensitive to chunk boundaries and to chunks that mix topics. A cross-encoder looking at the query and the chunk together is more tolerant of both.

The cost is latency and a second model to run. Rerankers are typically much smaller than generation models, so on a machine already running an LLM locally the added memory is modest — but it is another forward pass in the request path, and if your retrieval quality is already good it may not earn that.

Choose Embedding model when

You are building retrieval and need the first stage — everyone needs this
The corpus is large and queries must be fast
You want to index once and search many times

Choose Reranker when

Retrieval finds the right chunk but ranks it too low
You pass only a few chunks to the model and ordering matters
Your chunks are uneven or mix topics
Answer quality matters more than a few hundred milliseconds

Work it out for your own setup

Model directoryLocal models, including embedding models, with sizes and licences.VRAM calculatorBudget memory for a retrieval model alongside your LLM.Model recommenderWhat your hardware can run comfortably.

Common questions

Do I need a reranker to build RAG?: No. Start with embeddings alone and measure. Add a reranker when you can see the failure it fixes — the right chunk retrieved but ranked below your cutoff. Adding one before you have that evidence is premature.
Can one model do both?: No, they are different architectures. Some families publish an embedding model and a reranker trained together, which pair well, but you load and run them separately.
How much does a reranker slow things down?: It adds one forward pass per candidate, so latency scales with your shortlist size. Rerankers are small compared with generation models, and the pass usually costs less than generating the answer — but rerank a hundred candidates rather than fifty and you feel it.
What about hybrid search?: Combining vector search with keyword search improves recall, particularly for exact terms, names and codes that embeddings handle poorly. It is complementary to reranking, not an alternative: hybrid improves what reaches the shortlist, reranking improves the order within it.

Guides that go deeper

Longer reads on the concepts behind this comparison.

Related comparisons

Fine-tuning vs RAGCustomisation LoRA vs Full fine-tuningCustomisation Local LLM vs Cloud APILocal vs cloud