How Much VRAM Do You Need to Fine-Tune an LLM? (2026 Requirements)

The short answer

Fine-tuning needs far more memory than running the same model, and the gap is not small. Llama 3.1 8B runs comfortably on an 8GB card at Q4_K_M. Fine-tuning every parameter of it with 32-bit AdamW needs roughly 130GB. QLoRA brings that back down to about 5.9GB — which is why almost nobody does full fine-tuning on their own hardware.

Here is what each method costs for the same job: rank-16 adapters on all linear layers, a 1,024-token sequence, batch size 1, gradient checkpointing on.

QLoRA and LoRA figures use an 8-bit optimizer; the full fine-tune column uses 32-bit AdamW. You can change any of those assumptions in our fine-tuning VRAM calculator, which is where these numbers come from.

Why training needs so much more than inference

Running a model holds two things: the weights, and a KV cache that grows with your context. Training holds four.

1. The weights

Same as inference, but usually at higher precision. A BF16 base costs 2 bytes per parameter — 16GB for an 8B model. QLoRA quantizes the frozen base to 4-bit NormalFloat with double quantization, which works out at roughly 0.52 bytes per parameter, or about 4.2GB for the same model. This term does not depend on how much of the model you are training.

2. A gradient for every trainable parameter

Every parameter you update needs a gradient alongside it, normally in BF16 at 2 bytes each.

3. The optimizer’s state

This is the term that surprises people. AdamW keeps an FP32 master copy of each trainable parameter plus two FP32 moment estimates — 4 + 4 + 4 = 12 bytes per parameter. Add the BF16 weight and its gradient and you reach the figure worth memorising: 16 bytes per parameter for full fine-tuning. That is where 8B × 16 = 130GB comes from.

The 8-bit AdamW in bitsandbytes quantizes both moments to a single byte, taking the optimizer from 12 bytes to 6. On a large adapter that is several gigabytes for a quality cost generally reported as negligible.

4. Activations for the backward pass

To compute gradients you need the intermediate values from the forward pass. These scale with sequence length, batch size and model width — and they are the term people most often get wrong.

The mistake almost everyone makes with LoRA

LoRA does not reduce activation memory. It freezes the base model, which removes the gradient and optimizer terms for the 99%+ of parameters you are not training. It does nothing about activations, because the backward pass still has to run through the entire frozen base to reach the adapters.

This matters because activations are what actually blow up. Here is Llama 3.1 8B with QLoRA at rank 16, varying only the sequence length:

At 8,192 tokens, turning gradient checkpointing off costs you 33GB — the difference between a 12GB card and a data-centre one. Checkpointing discards most intermediate activations and recomputes them during the backward pass, trading roughly 20–30% more compute for an enormous memory saving. Leave it on.

Adapter rank barely matters. Sequence length is everything.

Forum threads spend a lot of energy on rank. The memory numbers say it is close to irrelevant. Same model, same everything, varying only rank:

Going from rank 8 to rank 128 — sixteen times the adapter capacity — costs 2.5GB. Going from a 1,024-token sequence to 8,192 without checkpointing costs 32GB. If you are running out of memory, look at sequence length and batch size first, then checkpointing, and only then consider dropping rank.

Gradient accumulation is the tool for the batch-size half of that: run a real batch size of 1 or 2 and accumulate gradients over several steps, and you keep the effective batch size your training recipe wants without the memory.

What you can fine-tune on the card you own

8GB

QLoRA on models up to about 8B, at modest sequence lengths. An 8B model at 1,024 tokens comes to 5.9GB, leaving reasonable headroom. Push to 4,096 tokens and you are at 7.2GB, which is tight if the same GPU is also driving your display.

12GB

QLoRA on 12B–14B models comfortably, and 8B models at long sequence lengths. This is the point where fine-tuning stops feeling constrained.

16GB

QLoRA up to roughly 24B. Also the first tier where plain LoRA on a BF16 base becomes practical for small models, which matters if you want to avoid the quantization step entirely.

24GB

QLoRA on 32B models, LoRA on 8B. A used RTX 3090 remains the value pick here — see our guide to budget GPUs for local LLMs.

48GB and above

QLoRA on 70B, which is where the original QLoRA paper’s headline result sits. Below that, 70B fine-tuning means multiple GPUs.

Our hardware matrix lists what each specific machine can do, and the VRAM tier pages cover the inference side of the same question.

Macs and unified memory

Apple Silicon has no separate VRAM — the GPU addresses roughly 70% of total system memory by default. A 32GB Mac therefore behaves like a machine with about 22GB of graphics memory, which on paper puts QLoRA on a 32B model within reach.

In practice the tooling is the constraint rather than the capacity. bitsandbytes, which provides the 4-bit quantization and 8-bit optimizers that QLoRA depends on, is built around CUDA. The Apple-native path is MLX, whose mlx-lm package includes LoRA fine-tuning and works well, but it is a different toolchain with a different ecosystem of recipes. If you are on a Mac, start with MLX rather than expecting a CUDA tutorial to transfer.

Before you fine-tune anything, check that you should

Fine-tuning teaches behaviour: output format, tone, domain conventions, one narrow task done well. It is unreliable at teaching facts. Training on a thousand statements about your company makes the model more likely to produce text in that shape, with no guarantee any individual fact survives — and worse, it makes hallucinations sound more authoritative, because the model has learned your domain’s voice.

If the problem is that the model does not know something, retrieval is the answer. It is faster to build, updates instantly when the facts change, and can cite its sources. We lay out the full comparison in fine-tuning vs RAG, and the method trade-off in LoRA vs full fine-tuning. If you are building retrieval, our guide to local embedding models for RAG covers the first component you will need.

The practical order: build retrieval first, measure, and fine-tune only if answers come back correct but badly shaped. Starting with the fine-tune means spending days of GPU time before you know whether the problem was ever about behaviour.

How these numbers were calculated

Model dimensions — hidden size, layer count, attention heads, key/value heads, intermediate size — are read from each model’s config.json on Hugging Face. Nothing is estimated. Adapter parameter counts are exact: a LoRA on a weight matrix of shape in × out adds r × (in + out) parameters, summed across the targeted modules in every layer.

Activation memory is the one approximation, and we label it as such. It follows the standard per-layer accounting of roughly 34 bytes per token per hidden unit, assuming flash attention or SDPA — which is what every current trainer uses, and which removes the quadratic attention term that dominated older implementations. Treat it as a planning figure with headroom rather than a guarantee.

Two checks on the model: it reproduces the published QLoRA result that a 65B-class model fine-tunes on a single 48GB card, and it returns exactly 16 bytes per parameter for full fine-tuning with 32-bit AdamW, which is the textbook figure. All sizes are decimal GB, the same convention as our other calculators.

FAQ

Can I fine-tune a 7B or 8B model on 8GB of VRAM?

With QLoRA, yes — about 5.9GB at rank 16 with a 1,024-token sequence and gradient checkpointing. Keep the sequence short and the batch size at 1, and leave headroom if the same GPU drives your monitor.

Why did my training run OOM when the model clearly fits?

Almost always sequence length or batch size. Both scale activation memory linearly, and the weights are already loaded by the time you hit the wall. Halve the sequence length first and see if the error moves.

Is 8-bit AdamW worth it?

For QLoRA, almost always. It halves the optimizer term from 12 bytes per trainable parameter to 6, and the reported quality cost is negligible.

Does the trained adapter need all this memory to run?

No. Once trained, you either load a small adapter file alongside the base model or merge it in, and you are back to ordinary inference memory. Use the inference VRAM calculator for that, and our guide to how much RAM and VRAM local LLMs need for the general picture.

How much training data do I need?

For style and format adaptation, a few hundred high-quality examples often suffice with LoRA. For a genuinely new capability, thousands. Consistency matters more than volume — a small clean dataset beats a large noisy one.

Sources

• Model architectures: config.json from each model’s Hugging Face repository
• QLoRA: Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs — 4-bit NormalFloat, double quantization, paged optimizers
• LoRA: Hu et al., LoRA: Low-Rank Adaptation of Large Language Models
• Optimizer state accounting: the standard mixed-precision Adam breakdown used in the ZeRO / DeepSpeed literature
• Activation memory: per-layer accounting from the Megatron-LM activation-recomputation work
• 8-bit optimizers: bitsandbytes documentation

Updated September 12, 2026.