Best Local Embedding Models for RAG in 2026: Sizes, Dimensions & Licences

If you are building a local RAG system in 2026, the embedding model matters more than the LLM you plug it into. A weak retriever hands your model the wrong chunks, and no amount of parameters downstream will fix that. The good news is that the best open embedding models are small — most run comfortably in under 1 GB, alongside your main model, on hardware you already own.

This guide covers the open embedding and reranking models worth using right now, what each one actually costs in storage and memory, and how to pick dimensions without quietly blowing up your vector database. All specifications come from official model cards and configuration files, checked on September 12, 2026.

Quick answer

• Best all-round local embedder: Qwen3-Embedding-0.6B — 595M parameters, 1024 dimensions, 32K context, Apache 2.0.
• Best tiny/on-device: EmbeddingGemma-300m — 308M parameters, 768 dimensions shrinkable to 128, runs in under 200 MB quantized.
• Best multilingual: BAAI/bge-m3 — 1024 dimensions, 8192 context, MIT licence, and by far the most-downloaded option.
• Best lightweight English: nomic-embed-text-v1.5 — 137M parameters, 768 dimensions, 8192 context, 274 MB via Ollama.
• Add a reranker. It is usually a bigger accuracy win than upgrading your embedder.
• Watch the licence. Jina’s v5 and v3 models are CC BY-NC 4.0 — non-commercial only.

The models worth running locally

Every model below can be pulled through Ollama or loaded with sentence-transformers. Parameter counts come from each model’s safetensors index on Hugging Face; dimensions and context come from the published configs.

A few things stand out. The Qwen3 embedding family is unusually generous on context — 32K on the 0.6B model, where most embedders stop at 512 or 8192 tokens. That matters less than it sounds, because you should be chunking well below those limits anyway, but it does mean you can embed a whole document for summary-level retrieval without truncation errors.

EmbeddingGemma is the interesting one for constrained hardware. Google describes the 308M parameters as roughly 100M model parameters plus 200M embedding parameters, trained across 100+ languages, and small enough to run in under 200 MB of RAM once quantized. Its 2048-token context is the shortest here, but for typical 200–500 token chunks that is irrelevant.

The part most guides skip: what dimensions cost you

Embedding dimension is the single biggest driver of your vector database size, and it is the decision people most often get wrong. The arithmetic is simple and worth doing before you index a million documents. The database adds its own overhead on top of the raw vectors — graph, ID map and headroom — which is worked through in our guide to the best vector database for local RAG.

Storage per vector is dimensions × bytes per value. At float32 that is 4 bytes per dimension; float16 is 2; int8 quantization is 1. The table below is straight multiplication for one million chunks — no benchmarks, just arithmetic you can verify yourself.

Read the top and bottom rows together. Moving from a 4096-dimension embedder at float32 to a 256-dimension Matryoshka-truncated vector cuts storage by 16×. On a 10M-chunk corpus that is the difference between 164 GB and 10 GB — the difference between needing a server and running on a laptop.

That is what Matryoshka Representation Learning buys you. Models trained with MRL — EmbeddingGemma and nomic-embed-text-v1.5 among them — pack the most important information into the leading dimensions, so you can simply slice the vector short and keep most of the retrieval quality. With nomic, that is literally two lines:

matryoshka_dim = 512
embeddings = embeddings[:, :matryoshka_dim]

A model without MRL does not support this. Truncating a non-MRL embedding discards information more or less arbitrarily and retrieval quality degrades badly. Check the model card before you slice.

Rerankers: the cheapest accuracy you will ever buy

A common mistake is to spend weeks swapping embedding models chasing a few points of retrieval quality, while skipping the reranker entirely. The standard two-stage pattern is more effective: use a fast embedder to pull the top 50–100 candidates, then a cross-encoder reranker to reorder and keep the top 5.

Rerankers score the query and document together rather than comparing pre-computed vectors, which is far more accurate — and far too slow to run across a whole corpus, which is exactly why the two stages exist.

If you are on limited VRAM, note that the reranker only needs to be resident while it scores a handful of candidates. You can load it on demand rather than keeping it alongside your generation model — a useful trick if you are already close to the limits described in our guide to how much RAM and VRAM local LLMs need.

Licensing: read this before you ship

This is the trap in the current lineup. Jina’s jina-embeddings-v5-text-small and jina-reranker-v3 are both published under CC BY-NC 4.0 — the NC means non-commercial. They are excellent models and completely fine for research, prototypes and personal projects, but you cannot ship them in a commercial product without a separate licence from Jina.

By contrast, the Qwen3 embedding and reranker family, nomic-embed-text-v1.5, mxbai-rerank-large-v2 and the BGE models are Apache 2.0 or MIT — genuinely permissive. EmbeddingGemma uses Google’s Gemma licence, which permits commercial use but carries its own use restrictions worth reading.

Added 10 October 2026: Google has since released EmbeddingGemma 2, and it is Apache 2.0. It is a different shape of model: 740M parameters in total, combining a 270M text model with vision and audio encoders, mapping text, images, video and audio into one 768-dimensional space with an 8,192-token context. The table above describes the original 300M text-only model.

Getting started

The fastest path is Ollama, which serves embeddings over an HTTP endpoint. If you have not set it up, our comparison of local LLM runners covers Ollama, LM Studio, Jan, llama.cpp and vLLM. To put these models to work answering questions over your own files, see building a private RAG chatbot.

# Pull an embedding model
ollama pull qwen3-embedding
ollama pull embeddinggemma
ollama pull nomic-embed-text

# Generate an embedding
curl http://localhost:11434/api/embed -d '{
  "model": "qwen3-embedding",
  "input": "Local RAG keeps your documents on your own machine."
}'

For Python work, sentence-transformers remains the most direct route:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")
vectors = model.encode(["first chunk", "second chunk"])
print(vectors.shape)   # (2, 1024)

Choosing: a short decision guide

• General English or multilingual RAG, commercial project: start with Qwen3-Embedding-0.6B or bge-m3. Both are permissively licensed and strong.
• Laptop, phone or edge device: EmbeddingGemma-300m, truncated to 256 dimensions.
• Millions of chunks and storage pressure: pick an MRL model and truncate, or quantize to int8. Revisit the dimension table above before committing.
• Quality is the priority and you have VRAM to spare: Qwen3-Embedding-4B plus Qwen3-Reranker-4B. Budget for the 2560-dimension storage cost.
• Code retrieval: look at nomic-embed-code rather than a general-purpose text embedder. Our guide to running Qwen coding models in VS Code and Cursor covers the generation half of that stack.

Whatever you pick, size the rest of the system before you download anything. The VRAM calculator will tell you how much memory your generation model and its KV cache need, and the local models database lists current open models with their memory requirements. If you are still choosing hardware, the local hardware matrix gives estimated throughput by device, and the model recommender matches models to the memory you actually have.

Frequently asked questions

Do I need a GPU to run embedding models?

No. Embedding models in this size range run acceptably on CPU, and the small ones — nomic-embed-text at 274 MB, EmbeddingGemma at 622 MB — are designed for it. Indexing a large corpus is much faster on a GPU, but querying is cheap either way.

Can I mix an embedding model from one family with an LLM from another?

Yes. The embedder and the generation model are independent. Retrieval hands the LLM plain text, so any embedder can feed any model. The only hard rule is that you must use the same embedding model for indexing and for querying — vectors from different models are not comparable. In code that distinction shows up as embed_documents() versus embed_query(), which is worth understanding before you index anything.

What happens if I change embedding models later?

You have to re-embed your entire corpus. Vectors from a different model live in a different space, so mixing them silently destroys retrieval quality. Factor that cost in before choosing, especially at millions of chunks.

Is a bigger embedding model always better?

No, and the storage table above shows why. Qwen3-Embedding-8B produces 4096-dimension vectors that cost 16.4 GB per million chunks at float32. Unless retrieval quality is measurably limiting your system, a 0.6B model at 1024 dimensions plus a reranker is usually the better engineering trade.

What chunk size should I use?

There is no single correct answer, and it depends on your documents far more than your model. A common starting point is 200–500 tokens with modest overlap, then tuning against your own evaluation set. Note that every model here supports at least 2048 tokens, so context length is rarely the binding constraint.

Are these speed numbers benchmarks?

There are no speed numbers in this guide. The storage figures are arithmetic — dimensions multiplied by bytes per value — which you can reproduce exactly. Throughput depends heavily on your hardware, batch size and runner; see the hardware matrix for estimates.

Sources

• Qwen3-Embedding-0.6B model card, and the 4B and 8B variants on Hugging Face
• google/embeddinggemma-300m and the EmbeddingGemma announcement
• BAAI/bge-m3
• nomic-embed-text-v1.5
• jina-reranker-v3 and jina-embeddings-v5-text-small (CC BY-NC 4.0)
• mxbai-rerank-large-v2
• Ollama embedding model library

EmbeddingGemma 2 note added 10 October 2026. Updated September 12, 2026. Parameter counts, dimensions, context lengths and licences were read from official model cards and configuration files on that date. Model licences in particular do change — verify before shipping commercially.