How big is a 70B model at Q4_K_M?

About 42.9GB. Q4_K_M averages 4.90 bits per weight in real GGUF files, so 70 billion parameters x 4.90 / 8 = roughly 42.9GB, before the KV cache and runtime overhead. In practice you need around 47GB of memory to run it with a short context.

Why is Q2_K bigger than 2 bits per weight?

Because GGUF quantization is mixed precision. The name refers to the dominant block format, but attention and embedding tensors are kept at higher precision to protect quality. Measured across four real models, Q2_K files land near 3.15 bits per weight, not 2.0.

Which quantization should I use?

Q4_K_M is the default recommendation and the usual size-to-quality sweet spot. Step up to Q5_K_M or Q6_K if the model still fits comfortably, and only drop to Q3 or Q2 when nothing else will load.

Is the file size the same as the memory it needs?

No. The weights are the largest part, but you also need the KV cache, which grows with context length, plus framework overhead. A short context adds roughly 8% plus about 1GB. Long contexts cost far more, which is what the VRAM calculator is for.

Where do these bits-per-weight numbers come from?

They are derived, not quoted: GGUF file size in bytes x 8, divided by the exact parameter count from each model's safetensors index on Hugging Face. The figures are averaged across Llama 3.1 8B, Qwen2.5 14B, Qwen2.5 32B and Gemma 2 9B, measured on 12 September 2026. Checked afterwards against two models not used to build the table, Mistral 7B v0.3 and Phi-3.5-mini, predictions landed within 6% worst-case and within 1-3% for Q4_K_M, Q5_K_M, Q6_K and Q8_0.

GGUF Quantization Size Calculator

Pick a model size and see exactly how large it becomes at every GGUF quantization level — and whether it fits on your machine. The bits-per-weight figures here are measured from real GGUF files, not taken from the format names, which understate the true size.

Model

Check against a device

Parameters70B

0.5B30B60B90B120B

Apple MacBook Air 13-inch (M4, 2025) has about 10.7GB usable for a model. No quantization of a 70B model fits in that budget — try a smaller model.

A 70B model at every GGUF quantization

Quant	Bits/weight	File size	In memory	Fits?
Q2_KSmallest usable k-quant. Noticeable quality loss; use only when nothing else fits.	3.15	27.6 GB	~30.8 GB	Too big
Q3_K_SSmall 3-bit variant. Measurable degradation on reasoning tasks.	3.63	31.8 GB	~35.3 GB	Too big
Q3_K_MMedium 3-bit. A reasonable last resort to fit a larger model into limited memory.	4	35.0 GB	~38.8 GB	Too big
Q3_K_LLarge 3-bit. Close to Q4_K_S in size, so Q4_K_S is usually the better pick.	4.31	37.7 GB	~41.7 GB	Too big
IQ4_XSImportance-matrix 4-bit. Smaller than Q4_K_S at similar quality, but slower on some backends.	4.41	38.6 GB	~42.7 GB	Too big
Q4_K_SSmall 4-bit. A little tighter than Q4_K_M when you need the headroom.	4.66	40.8 GB	~45.0 GB	Too big
Q4_K_MThe default recommendation. Best size-to-quality trade-off for most local use.	4.9	42.9 GB	~47.3 GB	Too big
Q5_K_S5-bit small. Marginal quality gain over Q4_K_M for roughly 14% more size.	5.57	48.7 GB	~53.6 GB	Too big
Q5_K_M5-bit medium. Worth it if the model still fits comfortably.	5.71	50.0 GB	~55.0 GB	Too big
Q6_KVery close to Q8_0 quality at three quarters of the size.	6.57	57.5 GB	~63.1 GB	Too big
Q8_0Effectively indistinguishable from the original weights for most tasks.	8.51	74.5 GB	~81.4 GB	Too big
BF16 / FP16Unquantized. Mainly for fine-tuning or serving on GPU clusters.	16	140.0 GB	~152.2 GB	Too big

File size = parameters × bits per weight ÷ 8. In memory adds roughly 8% plus 1GB for the KV cache at a short context and framework overhead — a long context costs considerably more, which the VRAM calculator works out precisely. Bits per weight are measured from real GGUF files, not nominal names.

Where these numbers come from

Bits per weight derived empirically: GGUF file size in bytes x 8 / exact parameter count from the model's safetensors index on Hugging Face.

bartowski/Meta-Llama-3.1-8B-Instruct-GGUF (8.03B params)
bartowski/Qwen2.5-14B-Instruct-GGUF (14.77B params)
bartowski/Qwen2.5-32B-Instruct-GGUF (32.76B params)
bartowski/gemma-2-9b-it-GGUF (9.24B params)

Values are the mean across those four models. Spread is small for the high-bit formats (Q8_0 varied only 8.50-8.51) and wider for Q2_K, where quantizers keep more tensors at higher precision than the nominal 2 bits.

How accurate is it? Validated against two holdout models not used to build the table (Mistral 7B Instruct v0.3 and Phi-3.5-mini): predicted sizes came within 6% worst-case, and within 1-3% for the commonly used formats (Q4_K_M, Q5_K_M, Q6_K, Q8_0). Q2_K varies most between quantizers, so treat it as the least precise row.

Measured on 2026-09-12. Quantizers change their tensor-precision rules over time, so re-derive these if the numbers start to drift from what you download.

Common questions

How big is a 70B model at Q4_K_M?: About 42.9GB. Q4_K_M averages 4.90 bits per weight in real GGUF files, so 70 billion parameters x 4.90 / 8 = roughly 42.9GB, before the KV cache and runtime overhead. In practice you need around 47GB of memory to run it with a short context.
Why is Q2_K bigger than 2 bits per weight?: Because GGUF quantization is mixed precision. The name refers to the dominant block format, but attention and embedding tensors are kept at higher precision to protect quality. Measured across four real models, Q2_K files land near 3.15 bits per weight, not 2.0.
Which quantization should I use?: Q4_K_M is the default recommendation and the usual size-to-quality sweet spot. Step up to Q5_K_M or Q6_K if the model still fits comfortably, and only drop to Q3 or Q2 when nothing else will load.
Is the file size the same as the memory it needs?: No. The weights are the largest part, but you also need the KV cache, which grows with context length, plus framework overhead. A short context adds roughly 8% plus about 1GB. Long contexts cost far more, which is what the VRAM calculator is for.
Where do these bits-per-weight numbers come from?: They are derived, not quoted: GGUF file size in bytes x 8, divided by the exact parameter count from each model's safetensors index on Hugging Face. The figures are averaged across Llama 3.1 8B, Qwen2.5 14B, Qwen2.5 32B and Gemma 2 9B, measured on 12 September 2026. Checked afterwards against two models not used to build the table, Mistral 7B v0.3 and Phi-3.5-mini, predictions landed within 6% worst-case and within 1-3% for Q4_K_M, Q5_K_M, Q6_K and Q8_0.

Guides behind this calculator

What each quantization level actually costs you in quality, and how to size memory around it.