AI GPU comparison
The reference table behind our GPU advice. Two numbers decide local AI performance, and neither is the one on the box: VRAM capacity sets which models you can hold at all, memory bandwidth sets how fast the ones that fit will generate. Every card is listed on both — 26 cards, 5 of them at 24GB or more.
Looking for a straight recommendation rather than the data? Best GPU for AI in 2026 names a card at each budget and explains the reasoning.
How to read the speed column
The tokens-per-second figures are calculated from memory bandwidth, not measured on a bench. We do not have a hardware lab and will not pretend otherwise. They are reliable for comparing cards against each other and for setting expectations; they are not benchmark results, and your real throughput will vary with quantization, context length and driver.
Ranked by capacity, then speed
| GPU | VRAM | GB/s | Est. tok/s | Largest model it holds |
|---|---|---|---|---|
| 2x NVIDIA RTX 3090 | 48GB | 936 | ~124.2 | Qwen 2.5 72B Instruct · Q4_K_M |
| NVIDIA GeForce RTX 5090 | 32GB | 1792 | ~237.7 | Qwen3.6 35B-A3B (MoE) · Q4_K_M |
| NVIDIA RTX 4090 | 24GB | 1008 | ~133.7 | Qwen3.6 35B-A3B (MoE) · Q4_K_M |
| AMD Radeon RX 7900 XTX | 24GB | 960 | ~127.3 | Qwen3.6 35B-A3B (MoE) · Q4_K_M |
| NVIDIA RTX 3090 | 24GB | 936 | ~124.2 | Qwen3.6 35B-A3B (MoE) · Q4_K_M |
| NVIDIA GeForce RTX 5080 | 16GB | 960 | ~127.3 | Mistral Small 3 24B · Q4_K_M |
| NVIDIA GeForce RTX 5070 Ti | 16GB | 896 | ~118.9 | Mistral Small 3 24B · Q4_K_M |
| NVIDIA RTX 4080 Super | 16GB | 736 | ~97.6 | Mistral Small 3 24B · Q4_K_M |
| NVIDIA RTX 4070 Ti Super | 16GB | 672 | ~89.1 | Mistral Small 3 24B · Q4_K_M |
| AMD Radeon RX 9070 XT | 16GB | 640 | ~84.9 | Mistral Small 3 24B · Q4_K_M |
| AMD Radeon RX 7800 XT | 16GB | 624 | ~82.8 | Mistral Small 3 24B · Q4_K_M |
| NVIDIA RTX 4090Laptop | 16GB | 576 | ~76.4 | Mistral Small 3 24B · Q4_K_M |
| Intel Arc A770 | 16GB | 560 | ~74.3 | Mistral Small 3 24B · Q4_K_M |
| NVIDIA GeForce RTX 5060 Ti | 16GB | 448 | ~59.4 | Mistral Small 3 24B · Q4_K_M |
| AMD Radeon RX 9060 XT | 16GB | 320 | ~42.4 | Mistral Small 3 24B · Q4_K_M |
| NVIDIA RTX 4060 Ti | 16GB | 288 | ~38.2 | Mistral Small 3 24B · Q4_K_M |
| AMD Radeon RX 7600 XT | 16GB | 288 | ~38.2 | Mistral Small 3 24B · Q4_K_M |
| NVIDIA RTX 4070 Super | 12GB | 504 | ~66.9 | DeepSeek R1 Distill Qwen 14B · Q4_K_M |
| Intel Arc B580 | 12GB | 456 | ~60.5 | DeepSeek R1 Distill Qwen 14B · Q4_K_M |
| NVIDIA RTX 4080 | 12GB | 432 | ~57.3 | DeepSeek R1 Distill Qwen 14B · Q4_K_M |
| NVIDIA RTX 3060 | 12GB | 360 | ~47.8 | DeepSeek R1 Distill Qwen 14B · Q4_K_M |
| NVIDIA RTX 4060 | 8GB | 272 | ~36.1 | Google Gemma 2 9B · Q4_K_M |
| NVIDIA RTX 4070Laptop | 8GB | 256 | ~34 | Google Gemma 2 9B · Q4_K_M |
| NVIDIA RTX 4060Laptop | 8GB | 256 | ~34 | Google Gemma 2 9B · Q4_K_M |
| NVIDIA RTX 4050Laptop | 6GB | 192 | ~25.5 | Llama 3.1 8B Instruct · Q4_K_M |
| NVIDIA GTX 1650 | 4GB | 128 | ~3.4 | Meta Llama 3.2 3B Instruct · Q4_K_M |
Every card name links to its full profile, with the complete model compatibility list. The largest here is the 2x NVIDIA RTX 3090 at 48GB.
Buy capacity before speed
The most expensive mistake in this category is buying a fast card that is too small. A model which does not fit in VRAM does not simply run slower — the layers that spill over execute against system RAM at a fraction of GPU bandwidth, and the card's speed stops mattering. Two cards of the same capacity can differ by an order of magnitude in throughput, but only for the models they both hold.
Decide the largest model you want to run first, check what it needs in the VRAM calculator, and buy the capacity that clears it with room for context. Then, and only then, compare bandwidth at that capacity. If you would rather browse by what fits, the VRAM tier pages list every model at 8GB through 128GB.
Common questions
- How do I compare GPUs for AI?
- On two numbers, in this order. VRAM capacity is a hard limit — a model that does not fit either runs partly on the CPU at a fraction of the speed, or does not run at all. Memory bandwidth then decides how fast the models that do fit will generate. Everything else on a spec sheet matters far less for local inference. Our buying guide makes a specific recommendation at each budget; this page is the underlying data.
- How much VRAM do I need for AI?
- For 7B and 8B models at 4-bit, 8GB will load them and 12GB gives real headroom. 16GB comfortably covers models up to about 14B. 24GB is the point where 30B-class models become practical, and it is the most common recommendation for serious local work. Above that you are buying the ability to run 70B models rather than better performance on smaller ones.
- Is more VRAM better than a faster GPU?
- Up to the point where your models fit, yes. A slower card that holds the whole model beats a faster one that has to offload layers to system RAM, because those layers then run at system memory bandwidth rather than GPU bandwidth. Once the model fits entirely, bandwidth takes over as the thing that matters.
- Do I need an NVIDIA GPU for AI?
- No, though it remains the least troublesome path because CUDA has the widest support. AMD cards work through ROCm and Intel Arc through its own runtime, and both are listed here. Apple Silicon takes a different approach again, sharing unified memory between CPU and GPU rather than having a dedicated pool.
- Is a laptop GPU good enough for AI?
- It works, with two caveats. Mobile parts carry less memory than their desktop namesakes — a mobile RTX 4090 has 16GB against the desktop card's 24GB — and they run at lower sustained power, so throughput is lower than the same name suggests. Check the actual VRAM figure rather than the model number.
Guides behind these numbers
Street prices, why 16GB is the wall, and how memory requirements are calculated.
- Best GPU for AI in 2026: What Each VRAM Tier Actually RunsThe buying recommendation at each tier, and why capacity beats speed.Read the guide
- Two 16GB Graphics Cards Can Be 3.3x Apart for Local AISame capacity, very different speed: the bandwidth table.Read the guide
- Best Budget GPUs for Local LLMs16GB cards, used RTX 3090s and current street prices.Read the guide