LLM Context Windows Explained: What They Cost You in VRAM

Every model page advertises a context window — 8K, 128K, a million tokens. It reads like a feature you either have or don’t. Locally it is not a feature at all: it is memory you pay for, every token of it, and on a long context the bill can exceed the model itself. This guide explains what the context window is, the arithmetic that decides what it costs you, and the three levers that actually reduce it.

Quick answer

• The context window is everything the model can see at once — system prompt, conversation, retrieved documents and the reply it is generating.
• It costs memory through the KV cache, which grows linearly with the number of tokens you allow.
• A typical 8B model at 8K context needs about 1 GB of KV cache. At 256K, the same model needs about 32 GB — more than the weights.
• Three levers: use less context, quantise the KV cache, or pick a model with fewer KV heads.

What actually fills the window

People think of the context window as “how long a conversation can be”. It is really a budget shared by four things:

1. The system prompt, which you usually never see.
2. The conversation so far, every earlier turn, re-read on each new token.
3. Anything retrieved or attached — documents, code files, a web page.
4. The reply being generated, which grows as it writes.

When the budget runs out, something is dropped — usually the oldest turns. That is why a long chat starts “forgetting” the beginning, and why a RAG setup with too small a window quietly ignores the documents you gave it. If that is the symptom you are chasing, see building a private RAG chatbot, where this is the single most common failure.

Why context costs memory: the KV cache

Generating each new token requires the model to attend to every previous token. Recomputing all of that every time would be hopelessly slow, so the intermediate keys and values are kept in memory — the KV cache. It is the price of not redoing the work.

Its size is straightforward arithmetic, and this is exactly what our VRAM calculator computes:

elements = 2 × kv_heads × head_dim × layers × context_tokens
bytes    = elements × bytes_per_element

The 2 is because you store both a key and a value. bytes_per_element is 2 for FP16, 1 for Q8, 0.5 for Q4. Nothing in that formula depends on how large the model’s weights are — it depends on its shape.

A worked example you can check

Take a typical 8B-class model: 32 layers, 8 key-value heads, head dimension 128, FP16 cache, 8,192 tokens of context.

elements = 2 × 8 × 128 × 32 × 8,192 = 536,870,912
bytes    = 536,870,912 × 2           = 1,073,741,824
         = exactly 1 GiB

One gigabyte, for 8K of context. The useful part is what happens next, because the relationship is linear:

Computed with the formula above for a 32-layer, 8-KV-head, 128-head-dim model. GB here means GiB (1024³ bytes), which is how the calculator divides and how GPU VRAM is labelled — so the 8,192-token row is the 1 GiB worked out above, exactly. Your model’s shape will differ; use the calculator for a specific one.

This is the point of the article. That 8B model’s weights are about 4.6 GiB at Q4_K_M — Llama 3.1 8B is 4.58 GiB in the Ollama registry. Give it the 256K context its card advertises and the KV cache alone is 32 GiB, almost exactly seven times the weights. The advertised context window is not free, and on consumer hardware it is usually the binding constraint rather than the model size.

Why some models are far cheaper than others

Two architectural details move this number a lot, and neither is on the marketing page.

Grouped-query attention (the kv_heads term). Older models gave every attention head its own key-value pair. Modern ones share: a model with 32 attention heads but only 8 KV heads stores a quarter as much cache. When two models of the same size have very different memory behaviour at long context, this is usually why.

Sliding-window attention. Some models only let most layers attend to a fixed recent window, with a few full-attention layers. Only the full-attention layers pay for the whole context; the windowed ones are capped at their window size. Our calculator models this explicitly — it counts full-attention and sliding-window layers separately rather than assuming every layer caches everything, which is why its figures for hybrid-attention models come out lower than a naive calculation.

The three levers

• Use less context. The cheapest fix and the most overlooked. If your documents fit in 16K, do not allocate 128K — allocation costs memory whether or not you fill it. In LangChain this is the num_ctx setting, covered in using Ollama with LangChain.
• Quantise the KV cache. Q8 halves it; Q4 quarters it. This is separate from quantising the weights, and it is the biggest single lever at long context — 16 GB becomes 4 GB. Quality impact is usually modest, but test it on your own workload.
• Choose a model with fewer KV heads. Worth checking before you download, because it is fixed once you have.

If none of that is enough, the constraint is the hardware: how to increase VRAM covers what genuinely helps on each platform, and the best GPU for AI covers buying for headroom.

Context and speed are different problems

A longer context does not only cost memory — it costs time, in two distinct ways. Before the first word appears, the whole prompt must be processed. Then, as the cache grows, each new token has more to read. Generation slows as the conversation fills up.

That second effect is why the same model feels fast at the start of a session and sluggish 50 messages in. Our guide to tokens per second covers what sets generation speed, and why memory bandwidth decides it.

How much context do you actually need?

• Chat: 8K is comfortable for most conversations.
• Documents and RAG: 8K is the practical floor; retrieved passages are larger than people expect.
• Coding assistants: the hungriest common case — agents send whole files and directory listings. Ollama recommends at least 64K tokens for coding agents; our OpenCode local setup guide works out what that costs for eight common models. See the best local LLMs for coding.
• Whole-repository or whole-book work: 128K and up, and at that point plan the KV cache as a first-class part of your memory budget, not an afterthought.

Check a specific model at a specific context with the VRAM calculator, or see what fits each card class on the VRAM tier pages. For the weights side of the budget, how much RAM and VRAM you need covers the rest.

FAQ

What is an LLM context window?

The maximum number of tokens the model can consider at once — system prompt, conversation, attached documents and the reply it is writing, all sharing one budget.

Does a bigger context window use more memory?

Yes, linearly. The KV cache grows in direct proportion to the tokens you allow. Doubling the context doubles the cache.

How much VRAM does a 128K context need?

For a typical 8B-class model, about 16 GB at FP16, 8 GB at Q8 and 4 GB at Q4 — on top of the weights. Larger models with more layers cost more.

Why does my model slow down in a long conversation?

Because each new token reads a larger KV cache. Generation speed falls as the context fills.

Is quantising the KV cache safe?

It is a quality trade, not a free win. Q8 is generally a modest change and can be the difference between fitting and not fitting. Test on your own task.

Sources

• BFWR method: the KV-cache formula above is the one implemented in our VRAM calculator, which counts full-attention and sliding-window layers separately
• Model context lengths quoted in the tier pages and model directory are read from each model’s own config.json
• The 8B weight figure is the Llama 3.1 8B Q4_K_M layer size read from the Ollama registry on 20 September 2026

Updated 20 September 2026: the KV-cache formula was re-checked against the calculator’s source and the 8B weight figure against the Ollama registry on that date. The memory figures are arithmetic you can reproduce, not benchmark measurements.