Best LLMs for 128GB VRAM

Showing the best quantization each model can use at this capacity — higher precision first, dropping to Q4_K_M only where it has to.

“Needs” is the recommended memory for that quantization, which leaves room for a working context. A 128GB card is treated as ~127GB usable, because the display, framework and CUDA or Metal context reserve roughly 1GB before your model loads.

We profile 4 machines at roughly this capacity. Their memory bandwidth ranges from 800 to 1200 GB/s — and since generation speed is bandwidth-bound, two machines with the same 128GB can differ by an order of magnitude in tokens per second. Capacity decides what fits; bandwidth decides how fast.

Image generation on 128GB

11 open image models run at 1024×1024 on a card this size — the best being FLUX.2 [dev] at BF16 / FP16, peaking near 115.6GB. Diffusion pipelines load a large text encoder alongside the image model, so offloading it matters more here than it does for language models.

128GB VRAM: common questions

What is the best LLM for 128GB of VRAM?

OpenAI gpt-oss-120b (116.8B (5.1B active)) is the largest model with comfortable headroom, at Q4_K_M, needing about 68.3GB against the ~127GB usable on a 128GB card. 28 of the 30 models in our directory fit at 128GB in some form.

Why is only about 127GB of a 128GB card usable?

The display output, the inference framework and the CUDA or Metal context all reserve memory before your model loads. Roughly 1GB is a realistic reservation, so a 128GB card gives you about 127GB to work with. Planning to the full 128GB is the most common reason a model that "should" fit fails to load.

Should I run a bigger model at Q4 or a smaller one at Q8?

Usually the bigger model at Q4_K_M. Above about 4 bits per weight, parameter count matters more than precision, so a larger model quantized harder generally beats a smaller model kept at full precision. Below Q4 that stops being true and quality falls away quickly.

How fast will these run on 128GB?

VRAM capacity decides what fits; memory bandwidth decides how fast it runs, and that varies by device. Among the 4 machines we profile at this capacity, bandwidth ranges from 800 to 1200 GB/s — which can mean an order-of-magnitude difference in tokens per second for the same model. The hardware matrix gives an estimate per device.

Can I generate images with 128GB?

Yes — 11 open image models run at 1024×1024 on a card this size, the best being FLUX.2 [dev] at BF16 / FP16. Diffusion models load a large text encoder alongside the image model, so offloading that encoder to CPU matters more than it does for LLMs.

How memory requirements are calculated, and which cards give you this much.

• How Much RAM and VRAM Do You Need to Run Local LLMs?Model weights, KV cache and OS overhead, tier by tier.Read the guide
• Quantization Guide: Q4_K_M vs Q8_0 vs FP8What each quant level costs you in memory and quality.Read the guide
• Best Budget GPUs for Local LLMs16GB cards, used RTX 3090s and current street prices.Read the guide