Q4 vs Q8 Quantization

Q8_0 is close enough to the original weights that the difference rarely shows in normal use. Q4_K_M is about 42% smaller, and it is the level most people run because it is the largest that fits. The honest framing is not which is better in isolation — Q8_0 is — but which fits. A larger model at Q4 generally beats a smaller model at Q8 at the same memory budget, and a model resident in VRAM at Q4 always beats one spilling into system RAM at Q8.

Q4_K_M vs Q8_0 at a glance

Weight size at each level, for common model sizes

Weights only — the KV cache and framework overhead sit on top. Bits per weight are measured from real GGUF files, not taken from the format names.

Neither name means what it says

A Q4_K_M file is not four bits per weight and a Q8_0 file is not eight. Both are mixed precision: the quantizer keeps attention projections and embedding tables at higher bit depth, because error there propagates through the whole network, and pushes the bulk of the feed-forward weights down.

We measured this against real GGUF files rather than taking the labels at face value — file size in bytes times eight, divided by the exact parameter count from the model's safetensors index. Q4_K_M lands at about 4.90 bits per weight and Q8_0 at about 8.51.

That extra weight above the nominal figure is the point. It is why Q4_K_M holds up as well as it does, and why comparisons against a strict 4-bit format elsewhere in the ecosystem are not comparing like with like.

The decision is almost never quality in isolation

At the same memory budget you usually have a choice between a bigger model at Q4 and a smaller one at Q8. For most tasks the bigger model wins, and it is not close — the capability difference between parameter counts is larger than the quality difference between quantization levels.

The second and sharper form of the decision is residency. If Q8_0 does not fit in your VRAM and Q4_K_M does, Q4_K_M is the right answer regardless of quality, because every offloaded layer runs at system-memory bandwidth on every token. A slightly degraded model that responds is better than a pristine one that crawls.

This is why Q4_K_M became the default recommendation. It is not the best quantization; it is the best one that fits on the hardware most people own.

When Q8 is the right call

When it fits with room to spare. On a 24GB card running an 8B model, Q8_0 needs about 8.5GB against Q4_K_M's 4.9GB, and both leave ample room for context. There is no reason to take the quality loss.

When you are evaluating. If you are measuring a model's capability, or comparing two models, quantization artefacts are noise in your results. Use the highest precision you can hold.

And when the task is precision-sensitive — structured output that must parse, code that must compile, long chains of reasoning where small errors compound. These are the cases where users most often report a difference between quantization levels, and they are worth testing on your own workload rather than trusting a general claim.

Work it out for your own setup

Common questions

How much worse is Q4 really?

Small on most benchmarks and task-dependent in practice. It shows up most in precise work — structured output, code, long reasoning — and least in open-ended writing. Rather than trusting a single number, run both on a prompt that represents your actual use if you have the memory to try.

What about Q5 and Q6?

They sit between, at about 5.71 and 6.57 bits per weight. Q6_K is a good compromise when Q8_0 overflows and you have room above Q4_K_M. The ladder exists precisely so you can find the highest rung that fits.

Is Q4_K_M better than Q4_K_S?

It keeps more tensors at higher precision, so it is slightly larger — about 4.90 bits per weight against 4.66 — and slightly better. If both fit, take the M.

Does quantization affect speed?

Yes, and in the direction that helps. Generation is bound by reading weights from memory, so fewer bytes per weight means faster tokens. A Q4 model generates faster than the same model at Q8, on top of being smaller.