Tokens per Second Explained: What Sets Local LLM Speed (and How to Estimate It)

Tokens per second is the number everyone quotes for local LLM speed, and it’s often misunderstood. It isn’t one number: a model reads your prompt at one speed and writes its answer at another, and the two are limited by different parts of the hardware. This guide explains what tokens per second measures, what actually sets it, how to estimate it for your own machine, and how to measure it. It also checks our own estimate against public measurements, including where the estimate goes wrong.

Quick answer

• Two speeds: prompt processing (reading your input, also called prefill) and token generation (writing the reply). Generation is the one you feel as typing speed.
• Generation is set mainly by memory bandwidth. Every new token means reading the model’s active weights from memory, so faster memory means more tokens per second.
• Prompt processing is set mainly by compute. It runs in large parallel batches, so it’s much faster than generation.
• Quick estimate: generation tokens/sec ≈ bandwidth (GB/s) × 0.65 ÷ active model size (GB). It’s a ceiling, and it overshoots on the fastest hardware.

What a “token” is, and why there are two speeds

A token is a chunk of text, often a word or part of a word. When you send a prompt, the model processes every token in it. That work can be done many tokens at a time, so it’s usually fast. Then it generates the reply one token at a time, and each token needs a full pass through the model’s active weights. That sequential step is why replies stream out word by word, and why generation speed is what people mean by “tokens per second”.

Because generation reads the weights over and over, the limit is usually how fast those weights move out of memory, not how fast the GPU can do maths. That’s why a card’s memory bandwidth predicts generation speed so well. It’s also why having more VRAM doesn’t make the same model run faster: capacity decides what fits, bandwidth decides how fast. Twelve 16GB cards show that gap directly, from 288 GB/s to 960 GB/s on identical capacity. VRAM vs RAM covers the difference.

The estimate we use, and a check against real measurements

Our hardware matrix estimates generation speed with one rule:

tokens/sec ≈ memory bandwidth (GB/s) × 0.65 ÷ active weights (GB)

The 0.65 factor reflects that real decoding doesn’t reach the memory’s theoretical bandwidth. To see how good the rule is, compare it with public measurements. The llama.cpp project collects Apple Silicon benchmarks in a GitHub discussion, all on the same test: LLaMA 7B at Q4_0, a 3.56 GiB (3.82 GB) model. There, pp512 is prompt processing of 512 tokens, and tg128 is generating 128 tokens.

Measured values: llama.cpp discussion #4167. The “our estimate” column applies the formula above to the same model; it is not a measurement.

Three things stand out:

• The rule tracks well through the middle of the range. From the M2 Pro to the M4 Max, the estimate lands within about 12% of the measurement. At the low end it errs the other way: it underestimates the base M1 by about a fifth.
• It overshoots at the top. On the M2 Ultra the estimate is 136 t/s against a measured 94. Once memory is fast enough, compute and software become the limit instead. Treat the formula as a ceiling for very high-bandwidth hardware. The llmfit tool uses a lower factor, 0.55, which lands closer on the fastest chips and further off in the middle.
• Prompt processing is an order of magnitude faster than generation, and it scales with GPU compute, not just bandwidth. That’s why a long prompt can still take noticeable time even on a fast card.

What else changes tokens per second

• Model size and quantization. A smaller file means fewer gigabytes to read per token. Q4_K_M generates faster than Q8_0 of the same model because it’s a little over half the size (about 4.9 bits per weight against 8.5).
• Mixture-of-experts models. Only the active experts are read per token. Qwen3-Coder 30B-A3B has 30.5B parameters but 3.3B active, so it generates like a small model, while still needing memory for all 30.5B. Our roundup of the best local LLMs for coding lists which coding models use this design.
• Offloading. If part of the model spills into system RAM, those layers run at system-memory bandwidth, and the slowest part sets the pace for every token.
• Context length. A longer conversation means a bigger KV cache for each new token to read, so generation slows as the context fills. What that cache costs in memory is worked out in LLM context windows explained.

Estimates for real machines

The hardware matrix applies the rule to an 8B model (Qwen3 8B at Q4_K_M). A few examples:

Estimates from the bandwidth formula, not measurements. Given the M2 Ultra result above, the figures for the highest-bandwidth cards are best read as upper bounds.

For more devices, see the hardware matrix and the AI GPU comparison, which ranks cards by VRAM and bandwidth. On a Mac, the Mac VRAM optimizer shows how much unified memory the GPU can actually use.

How to measure your own

• Ollama: run a model with --verbose and it prints timing statistics after each reply. Those include the prompt evaluation rate and the generation (“eval”) rate in tokens per second.

  ollama run qwen3:8b --verbose

• llama.cpp: the llama-bench tool reports the same pp and tg figures used in the table above, so your results are directly comparable with the public ones.

Measure with the model and context length you actually use. A number from a short test prompt will look better than a long working session.

FAQ

Does more VRAM make a model run faster?

No. More VRAM lets a bigger model fit, or keeps a model from spilling into slower system RAM. For a model that already fits, generation speed follows memory bandwidth. That is why the DGX Spark and Strix Halo, which share a 256-bit memory bus, generate text at similar speeds (see DGX Spark vs Strix Halo).

Why is the first reply slower than the rest?

The model may still be loading into memory. And before the first word appears, the whole prompt, including any system prompt and attached documents, has to be processed.

Why is my Mac slower than an RTX 3090 on the same model?

Mostly bandwidth. An RTX 3090’s memory runs at 936 GB/s; many Macs have well under half that. Macs make up for it in capacity: unified memory lets them load models far larger than any single consumer GPU holds.

What counts as a good tokens-per-second figure?

It depends on the job. For chat, speed stops mattering much once text appears faster than you read it. For coding agents and batch jobs that generate a lot of text, higher generation speed saves real time. Also note that a long prompt makes prompt-processing speed matter too.

Sources

• llama.cpp: Performance of llama.cpp on Apple Silicon M-series (discussion #4167)
• llama.cpp: llama-bench
• BFWR estimate method: the hardware matrix (bandwidth × 0.65 ÷ active weights)

Updated September 15, 2026. All BFWR speed figures are estimates from memory bandwidth, not lab measurements; the measured figures are cited from llama.cpp.