LoRA vs Full Fine-tuning

Full fine-tuning updates every parameter, which means holding the weights, the gradients and the optimizer state at once — commonly cited at around 16 bytes per parameter with mixed-precision Adam, which puts even a 7B model far beyond any consumer GPU. LoRA freezes the base model and trains small low-rank adapters, typically well under 1% of the parameters, so the gradient and optimizer cost nearly vanishes. QLoRA quantizes the frozen base on top of that. For almost everyone, LoRA or QLoRA is the only option that exists.

LoRA vs Full fine-tuning at a glance

Where the memory actually goes

Training needs far more memory than inference, and the weights are the smallest part of it. You also hold a gradient for every parameter, and the optimizer's own state — Adam keeps two running averages per parameter, conventionally in 32-bit. Add mixed-precision copies and activations for the backward pass.

The commonly used planning figure is around 16 bytes per parameter for full fine-tuning with Adam in mixed precision, against roughly 2 bytes per parameter to run the same model at BF16. That is the whole story: training a 7B model can demand more memory than running a 70B one.

LoRA attacks the largest terms. The base model is frozen, so it needs no gradients and no optimizer state — it is just weights sitting there. Only the adapter parameters are trained, and there are very few of them, so the gradient and optimizer cost is a rounding error. What remains is the frozen base plus activations.

What QLoRA adds

QLoRA quantizes the frozen base model to 4-bit and trains the adapter in higher precision on top, dequantizing each layer only as it is needed. Since the base is never updated, quantizing it costs much less than quantizing a model you intend to train.

That removes the last large term. The base weights shrink to roughly a quarter of their BF16 size, and what is left is activations and the adapter. This is what moved fine-tuning from data-centre work to something you can do on a desktop.

Two things still scale badly and catch people out: sequence length, because activation memory grows with it, and batch size. Most out-of-memory errors during QLoRA come from one of those rather than from the model, and both are adjustable — with gradient accumulation you can keep an effective batch size while lowering the real one.

Adapters are an operational advantage, not just a cheap substitute

A LoRA adapter is a small file that sits alongside the base model. You can hold one base in memory and swap adapters per task, per customer or per tenant, which is impossible with full fine-tunes — each of those would be a complete model.

They are also reversible and cheap to iterate. A bad run costs you a small file, not a retraining budget, and you can keep several and compare them against the same base.

The honest limitation: a low-rank update cannot express arbitrary changes. For genuinely new capabilities, a new language, or a large shift away from the base model's distribution, full fine-tuning has a higher ceiling. For the adaptation work most people actually need — format, tone, domain behaviour, one narrow task — the ceiling is not the binding constraint.

Work it out for your own setup

Common questions

Can I fine-tune a 7B model on a 12GB card?

With QLoRA, generally yes, at a modest sequence length and batch size. With LoRA on an unquantized base it is tight, and with full fine-tuning it is not close. Sequence length is usually what decides it, so reduce that before anything else.

What rank should I use?

Common starting points are 8 to 32, with 16 a reasonable default. Higher rank means more capacity and more memory. For style and format adaptation, low ranks are usually sufficient and higher ones mostly add overfitting risk.

Is a LoRA worse than a full fine-tune?

On the adaptation tasks people usually want, the gap is small enough to be hard to detect. The gap widens when the target is far from what the base model already does. Given that full fine-tuning is out of reach for most hardware, the practical comparison is LoRA against nothing.

Can I merge a LoRA into the base model?

Yes, and it is common before deployment — merging removes the small runtime overhead and produces a single model you can quantize and convert to GGUF. You lose swappability, so keep the adapter file.