How much memory do you need to run an AI model locally? It’s the most common question from people setting up tools like Ollama or LM Studio. The short answer: 16GB is the practical minimum, 32GB is comfortable, and 64GB or more opens up the big models. The long answer depends on three things: how big the model is, how heavily it’s compressed (quantized), and how much conversation history (context) you want it to remember. This guide shows the math so you can work it out for any model, then applies it to the open models people are actually running in September 2026.
Quick Answer: Memory Needed by Model Size
| Model size (examples) | Approx. memory at 4-bit (Q4_K_M), 8k context | Practical hardware |
|---|---|---|
| 3–4B (Gemma 4 E4B, Phi-4-mini) | ~3–4GB | Almost any 8GB+ machine |
| 8–12B (Qwen3 8B, Gemma 4 12B) | ~6–9GB | 16GB Mac or PC, or an 8–12GB GPU |
| 20–35B (gpt-oss-20b, Qwen3.8-27B, Gemma 4 31B, Qwen3.6-35B-A3B) | ~14–23GB | 24–32GB Mac, or a 16–24GB GPU |
| 70B-class dense models | ~42–48GB | 64GB+ Mac, AMD Strix Halo, or 2× 24GB GPUs |
| ~120B MoE (gpt-oss-120b, Mistral Small 4) | ~65–75GB | 96–128GB unified memory machines |
| Frontier MoE (DeepSeek V4-Flash ~291B, Qwen3.8 2.4T) | ~170GB to well over 1TB | Mac Studio M5 Ultra (up to 512GB) or multi-machine clusters |
These are estimates. Real usage varies by runtime, quantization format and context length.
The Formula: Weights + KV Cache + Overhead
Total memory ≈ model weights + KV cache + runtime overhead
1. Model weights
Each parameter takes 2 bytes at full 16-bit precision (FP16/BF16). Quantization compresses the weights:
- FP16/BF16: ~2.0 bytes per parameter. An 8B model is ~16GB.
- 8-bit (Q8_0): ~1.06 bytes per parameter. An 8B model is ~8.5GB.
- 4-bit (Q4_K_M): ~0.6 bytes per parameter. An 8B model is ~5GB, and a 27B model is ~17GB.
Some newer models ship natively in low precision. OpenAI’s gpt-oss models, for example, use MXFP4, which is why gpt-oss-20b fits in about 16GB of memory.
2. The KV cache
The KV cache stores the model’s working memory of your conversation, so it grows with context length. At a few thousand tokens it’s small. At 32k–128k tokens it can add several gigabytes, or more, depending on the model’s architecture. Many runners let you quantize the KV cache (for example to 8-bit) to save memory. Our VRAM & KV Cache Calculator works out weights and KV cache for a given model and context length.
3. Overhead and your operating system
Runtimes need extra working memory, and your OS and apps need their share too. On a Mac, unified memory is shared between the CPU and GPU. By default macOS lets the GPU use only part of it: roughly two-thirds to three-quarters, depending on total RAM. Our Mac VRAM Optimizer explains the limit and how it can be raised.
Why 8GB Machines Struggle
An 8GB laptop can run small 3–4B models, but a 4-bit 8B model (~5GB) plus the OS usually exceeds physical memory. When that happens the system swaps to the SSD, which is dramatically slower than RAM, and speed collapses. For local AI in 2026, 16GB is the realistic floor.
Capacity vs Speed: Why Memory Bandwidth Matters
Capacity decides whether a model fits. Memory bandwidth mostly decides how fast it generates text, because each new token requires reading the active model weights from memory. A useful rule of thumb:
Estimated tokens/sec ≈ (memory bandwidth × ~0.6–0.75) ÷ size of the active weights in GB
Where that rule comes from, and how it compares with real measured figures, is covered in tokens per second explained.
| Hardware (official bandwidth) | Est. speed, 8B model at Q4 (~5GB) | Est. speed, 27B dense at Q4 (~17GB) |
|---|---|---|
| Mac mini M6, 24GB or 32GB (170GB/s) | ~20–25 tok/s | ~6–8 tok/s (needs 32GB) |
| AMD Ryzen AI Max+ 395 “Strix Halo” (256GB/s) | ~30–38 tok/s | ~9–11 tok/s |
| Mac mini M5 Pro (307GB/s) | ~37–46 tok/s | ~11–14 tok/s |
| RTX 5060 Ti 16GB (448GB/s) | ~54–67 tok/s | Doesn’t fit fully in 16GB at Q4 with context |
| Mac Studio M5 Max (614GB/s) | ~74–92 tok/s | ~22–27 tok/s |
| RTX 4090 24GB (1,008GB/s) | ~120–150 tok/s | ~36–44 tok/s |
| RTX 5090 32GB (1,792GB/s) | ~215–270 tok/s | ~63–79 tok/s |
These are theoretical estimates from specs, not measurements. Real results depend on the runtime, drivers and settings.
Mixture-of-Experts (MoE) models change the speed math. Models like Qwen3.6-35B-A3B, Gemma 4 26B-A4B and gpt-oss only activate a few billion parameters per token. They still need enough memory to hold all the weights, but they generate text much faster than a dense model of the same total size. That’s why large-memory, moderate-bandwidth machines like Strix Halo and Macs are good fits for them.
Recommendations by Budget
- 16GB machine: 4–12B models such as Qwen3 8B, Gemma 4 12B or gpt-oss-20b (tight). Great for chat, summarising and light coding help.
- 24–32GB: The sweet spot in 2026. You can run 27–35B models like Qwen3.8-27B, Gemma 4 31B and Qwen3.6-35B-A3B at 4-bit, which are strong for coding and reasoning.
- 64–128GB unified memory: 70B-class models and ~120B MoE models such as gpt-oss-120b.
- 256GB+: Frontier MoE models like DeepSeek V4-Flash, at low-bit quantization.
Not sure what fits your computer? Try the Local Model Recommender, or compare devices in the Local Hardware Matrix. To try a model without installing anything at all, a single-file llamafile is the quickest start — see our llamafile guide.
A Note on 2026 Hardware Prices
A global memory shortage has pushed GPU prices far above launch prices in 2026. In September 2026, price trackers list the RTX 5060 Ti 16GB at roughly $680–730, well above its $429 launch price, and the RTX 5090 at several times its $1,999 MSRP. Check current prices before assuming a GPU is the cheapest route. Unified-memory machines such as Macs and Strix Halo mini PCs can be better value for large models.
Frequently Asked Questions
Can I run a local LLM with 8GB of RAM?
Only small models of about 3–4B parameters. Most 7–8B models need 16GB to run smoothly.
Is VRAM or system RAM more important?
For speed, the model should sit entirely in fast memory: GPU VRAM or Apple/AMD unified memory. Spilling into regular system RAM or onto the SSD slows generation dramatically. Engines written for mixture-of-experts models narrow that gap: Strata keeps the busiest experts on a 12GB card and computes the rest on the CPU from system RAM.
How much memory does a 70B model need?
About 42–48GB at 4-bit with a moderate context, so a 64GB+ unified-memory machine or two 24GB GPUs.
Sources
Specifications are from Apple, AMD and NVIDIA. Model sizes are from the official model cards on Hugging Face (Qwen, Google Gemma 4, OpenAI gpt-oss, DeepSeek). 2026 GPU pricing is from TechSpot.
Updated September 11, 2026: rewritten with current open models, 2026 hardware and clearly labelled estimates in place of benchmark figures.
Work out what your hardware can actually run
Free calculators and directories, sized from real model weights — before you download gigabytes or order a card:
