Best LLMs for 32GB VRAM

Showing the best quantization each model can use at this capacity — higher precision first, dropping to Q4_K_M only where it has to.

“Needs” is the recommended memory for that quantization, which leaves room for a working context. A 32GB card is treated as ~31GB usable, because the display, framework and CUDA or Metal context reserve roughly 1GB before your model loads.

We profile 5 machines at roughly this capacity. Their memory bandwidth ranges from 89.6 to 1792 GB/s — and since generation speed is bandwidth-bound, two machines with the same 32GB can differ by an order of magnitude in tokens per second. Capacity decides what fits; bandwidth decides how fast.

Image generation on 32GB

11 open image models run at 1024×1024 on a card this size — the best being FLUX.2 [klein] 4B at BF16 / FP16, peaking near 18.8GB. Diffusion pipelines load a large text encoder alongside the image model, so offloading it matters more here than it does for language models.

32GB VRAM: common questions

What is the best LLM for 32GB of VRAM?

Qwen3.6 35B-A3B (MoE) (36B (3B active)) is the largest model with comfortable headroom, at Q4_K_M, needing about 24.9GB against the ~31GB usable on a 32GB card. 23 of the 30 models in our directory fit at 32GB in some form.

Why is only about 31GB of a 32GB card usable?

The display output, the inference framework and the CUDA or Metal context all reserve memory before your model loads. Roughly 1GB is a realistic reservation, so a 32GB card gives you about 31GB to work with. Planning to the full 32GB is the most common reason a model that "should" fit fails to load.

Should I run a bigger model at Q4 or a smaller one at Q8?

Usually the bigger model at Q4_K_M. Above about 4 bits per weight, parameter count matters more than precision, so a larger model quantized harder generally beats a smaller model kept at full precision. Below Q4 that stops being true and quality falls away quickly.

How fast will these run on 32GB?

VRAM capacity decides what fits; memory bandwidth decides how fast it runs, and that varies by device. Among the 5 machines we profile at this capacity, bandwidth ranges from 89.6 to 1792 GB/s — which can mean an order-of-magnitude difference in tokens per second for the same model. The hardware matrix gives an estimate per device.

Can I generate images with 32GB?

Yes — 11 open image models run at 1024×1024 on a card this size, the best being FLUX.2 [klein] 4B at BF16 / FP16. Diffusion models load a large text encoder alongside the image model, so offloading that encoder to CPU matters more than it does for LLMs.

How memory requirements are calculated, and which cards give you this much.

• How Much RAM and VRAM Do You Need to Run Local LLMs?Model weights, KV cache and OS overhead, tier by tier.Read the guide
• Quantization Guide: Q4_K_M vs Q8_0 vs FP8What each quant level costs you in memory and quality.Read the guide
• Best Budget GPUs for Local LLMs16GB cards, used RTX 3090s and current street prices.Read the guide