Best LLMs for 24GB VRAM

Showing the best quantization each model can use at this capacity — higher precision first, dropping to Q4_K_M only where it has to.

“Needs” is the recommended memory for that quantization, which leaves room for a working context. A 24GB card is treated as ~23GB usable, because the display, framework and CUDA or Metal context reserve roughly 1GB before your model loads.

We profile 4 machines at roughly this capacity. Their memory bandwidth ranges from 150 to 1008 GB/s — and since generation speed is bandwidth-bound, two machines with the same 24GB can differ by an order of magnitude in tokens per second. Capacity decides what fits; bandwidth decides how fast.

Image generation on 24GB

11 open image models run at 1024×1024 on a card this size — the best being FLUX.1 [dev] at FP8, peaking near 19.8GB. Diffusion pipelines load a large text encoder alongside the image model, so offloading it matters more here than it does for language models.

24GB VRAM: common questions

What is the best LLM for 24GB of VRAM?

Gemma 4 31B (31.3B) is the largest model with comfortable headroom, at Q4_K_M, needing about 22.1GB against the ~23GB usable on a 24GB card. Cohere Command R 35B (35.0B) is larger and will still load, but only as a tight fit with a short context. 22 of the 30 models in our directory fit at 24GB in some form.

Why is only about 23GB of a 24GB card usable?

The display output, the inference framework and the CUDA or Metal context all reserve memory before your model loads. Roughly 1GB is a realistic reservation, so a 24GB card gives you about 23GB to work with. Planning to the full 24GB is the most common reason a model that "should" fit fails to load.

Should I run a bigger model at Q4 or a smaller one at Q8?

Usually the bigger model at Q4_K_M. Above about 4 bits per weight, parameter count matters more than precision, so a larger model quantized harder generally beats a smaller model kept at full precision. Below Q4 that stops being true and quality falls away quickly.

How fast will these run on 24GB?

VRAM capacity decides what fits; memory bandwidth decides how fast it runs, and that varies by device. Among the 4 machines we profile at this capacity, bandwidth ranges from 150 to 1008 GB/s — which can mean an order-of-magnitude difference in tokens per second for the same model. The hardware matrix gives an estimate per device.

Can I generate images with 24GB?

Yes — 11 open image models run at 1024×1024 on a card this size, the best being FLUX.1 [dev] at FP8. Diffusion models load a large text encoder alongside the image model, so offloading that encoder to CPU matters more than it does for LLMs.

How memory requirements are calculated, and which cards give you this much.

• How Much RAM and VRAM Do You Need to Run Local LLMs?Model weights, KV cache and OS overhead, tier by tier.Read the guide
• Quantization Guide: Q4_K_M vs Q8_0 vs FP8What each quant level costs you in memory and quality.Read the guide
• Best Budget GPUs for Local LLMs16GB cards, used RTX 3090s and current street prices.Read the guide