VRAM vs RAM for Local LLMs

VRAM is the memory attached to your GPU, and it is roughly an order of magnitude faster than system RAM. For local models the practical rule is that whatever sits in VRAM runs fast and whatever spills into system RAM runs slowly — and because every token has to pass through every layer, the slow part sets the pace. Apple Silicon and other unified-memory designs remove the distinction entirely: there is one pool, and the GPU addresses about 70% of it.

VRAM vs System RAM at a glance

Why the slow part sets the pace

Generating one token means reading every weight in the model once. Not some of them — all of them. So the time to produce a token is roughly the model's size divided by the bandwidth of the memory holding it.

That is why the split matters so much. If 90% of a model sits in VRAM at 900GB/s and 10% sits in system RAM at 60GB/s, the 10% does not cost 10% more time. Those layers take fifteen times longer per byte than the rest, and they run on every single token.

The practical consequence: a model that fits entirely in VRAM at a lower quantization is almost always a better experience than a higher-quality quantization that spills. Dropping from Q5_K_M to Q4_K_M to stay resident is usually the right trade.

What system RAM is actually for

It is the fallback that makes running a too-large model possible at all, and that has real value. A 70B model at Q4_K_M needs roughly 43GB of weights. On a 24GB card with 64GB of system RAM it will run — slowly, at a pace better suited to a batch job than to conversation, but it will run. Without offload it simply would not load.

System RAM also holds everything else: the operating system, the browser, the model file while it is being read, and the framework's own overhead. Planning to your total figure is the most common reason a model that looks like it should fit does not, which is why our calculators reserve about 1GB before anything else.

What system RAM cannot do is make a slow setup fast. Adding another 64GB to a machine with an 8GB card raises the ceiling on model size and does nothing for speed.

Unified memory removes the question

On Apple Silicon there is one pool of memory, and the CPU and GPU both address it directly. There is no copy across a bus, and no split to manage. The same is true of AMD's Strix Halo designs and NVIDIA's DGX Spark.

The limit becomes how much of that pool the GPU is allowed to claim — roughly 70% by default on macOS. A 32GB Mac behaves like a machine with about 22GB of graphics memory; a 128GB Mac Studio like one with about 90GB, which is why those machines run models that no consumer GPU can hold.

The trade is bandwidth. Unified memory on a high-end Apple chip is fast by system-RAM standards and slower than a discrete GPU's dedicated memory. You get capacity at the cost of peak speed, which for large models is usually the right side of the trade — a model that fits slowly beats a model that does not fit at all.

Work it out for your own setup

Common questions

How much RAM do I need if I have a 24GB GPU?

Enough for the operating system and the model file as it loads, so 32GB is comfortable and 16GB workable. Extra system RAM beyond that only helps if you intend to run models larger than the card can hold, in which case you want enough to cover the overflow.

Can I add VRAM to my graphics card?

No. It is soldered. The only way to more VRAM is a different card, or a second card — which some runners can split a model across, though that brings its own complications.

Does a 32GB Mac equal a 32GB graphics card?

For capacity, roughly 22GB of it is available to the GPU, so no. For bandwidth, an Apple chip's unified memory is well short of a high-end discrete GPU. It holds more than a similarly priced card and generates more slowly.

Why does my model load but run so slowly?

Almost always because part of it is in system RAM. Check how many layers your runner offloaded to the GPU — if it is not all of them, drop a quantization level so the whole model fits, and compare.