Quantization Guide for Local LLMs: Q4_K_M vs Q8_0 vs FP8 (DeepSeek R1 Examples)

When you download a model for Ollama or LM Studio, you’ll see labels like Q4_K_M, Q8_0 or FP8. These are quantization levels: ways of compressing a model’s weights so it takes less memory and runs faster, at some cost to accuracy. This guide uses the DeepSeek R1 distills as examples, but the same rules apply to Qwen, Gemma, gpt-oss and other models. It explains what each format means, how much memory it needs, and which one to pick.

Quick Answer

• Q4_K_M: the default choice for most people. It’s about a third of the size of FP16, with a small quality loss that’s usually hard to notice in everyday use.
• Q5_K_M / Q6_K: use these if you have spare memory and want a bit more quality.
• Q8_0: close to full quality at about half the size of FP16. Good if memory isn’t tight.
• Below 4-bit (Q3, Q2, IQ2): only when you must squeeze a big model in. Quality drops noticeably, especially on smaller models.
• FP8 / FP4 (NVFP4, MXFP4): GPU-oriented formats used by runtimes such as vLLM, and by some models that ship natively in them (gpt-oss uses MXFP4).

Memory Needed by Format (Weights Only)

Add memory for the KV cache (context) and runtime overhead on top of these figures. Our VRAM Calculator does the full calculation for your context length.

How Much Quality Do You Lose?

Quantization quality is usually measured with perplexity (how “surprised” a model is by real text) and task benchmarks. Published results from the llama.cpp project and the community show a consistent pattern:

• Q8_0 is nearly indistinguishable from full precision in most tests.
• Q6_K and Q5_K_M show very small losses.
• Q4_K_M shows a small but measurable loss. It’s widely considered the best size-to-quality trade-off.
• Below 4 bits, losses grow quickly. Smaller models suffer more than large ones.
• Larger models tolerate quantization better. A 32B model at Q4 generally outperforms a 14B model at Q8 of similar size.

Reasoning models like DeepSeek R1 produce long chains of thought, so small errors can compound. If you notice odd reasoning at low bit-widths, step up a level. We don’t publish perplexity numbers of our own. For exact figures, see the llama.cpp perplexity discussions and the model publishers’ quantization notes.

Newer Quantization Techniques

• i-quants (IQ formats) and importance matrices: llama.cpp can use calibration data to keep the most important weights more precise, which improves quality at low bit-widths.
• Dynamic quantization: some publishers keep sensitive layers at higher precision, and this has made very low-bit versions of huge models (like the full 671B R1) usable.
• FP8 and FP4 on GPUs: NVIDIA’s Ada and Blackwell GPUs accelerate FP8, and Blackwell adds native FP4. Formats like NVFP4 and MXFP4 let GPU runtimes run large models efficiently.
• KV cache quantization: storing the context cache at 8-bit can save a lot of memory for long conversations, with little quality impact.

Which Should You Choose?

1. Pick the largest model that fits your memory at Q4_K_M, with room for context.
2. If you have leftover memory, move up to Q5_K_M, Q6_K or Q8_0 of the same model.
3. Only drop below 4-bit to run a much bigger model you couldn’t otherwise fit.

For how memory, bandwidth and speed relate, see How Much RAM and VRAM Do You Need? To run the models, follow How to Run DeepSeek R1 Locally. You can also compare models in the Local LLM Directory.

FAQ

Is Q4_K_M good enough?

For most chat, writing and coding use, yes. It’s the most popular choice for local models.

Does quantization make models faster?

Usually, yes. Smaller weights mean less data to read per token, and memory bandwidth is the main speed limit on consumer hardware.

Sources

llama.cpp project (quantization types and perplexity tooling), DeepSeek model cards, OpenAI gpt-oss-20b model card (MXFP4).

Updated September 11, 2026: removed perplexity and speed figures that weren’t from our own testing, and added current formats (FP4, i-quants, KV cache quantization).