Fine-Tuning Memory Math: Why Full Training Needs 16 Bytes per Parameter

Serving a 7B model takes about 14 GB. Fully fine-tuning the same model can take eight times that. The difference is not magic or sloppy engineering. It is a handful of extra copies of the model's numbers that training needs and inference does not. Once you can count those copies, you can predict whether a fine-tuning run fits your GPU before you rent anything, and you can see exactly why LoRA and QLoRA exist.

This guide walks through the standard memory accounting, a worked 7B example, how LoRA and QLoRA change it, and how to estimate the part that is hardest to predict: activations.

The 16-byte rule

A widely used rule for mixed-precision training with the Adam optimizer counts memory per parameter:

Each row has a reason:

• Weights are the model itself, used in the forward and backward pass.
• Gradients hold the result of : one number per parameter saying which way to nudge it.
• The 32-bit master copy keeps tiny updates from rounding away. A 16-bit number cannot represent a very small change added to a large weight, so the optimizer keeps a more precise version on the side.
• Adam's two moments are running averages of the gradient and the squared gradient. They let Adam adapt the step size for every parameter, and they cost two extra full-size copies in 32-bit.

Adam's states and the master copy together make up 12 of the 16 bytes. That is the real price of a good optimizer.

Worked example: a 7B model

• Inference weights alone: 7 billion x 2 bytes = about 14 GB
• Full fine-tuning states: 7 billion x 16 bytes = about 112 GB

Those 112 GB do not include activations or framework overhead, so a real run needs more. No single consumer GPU holds that. Even an 80 GB data-center card cannot, which is why full fine-tuning of even mid-sized models needs several GPUs or sharding.

Activations: the part you can trade

During the forward pass the model keeps intermediate results, called activations, because backpropagation needs them. They scale with batch size, sequence length, hidden size and the number of layers. Double the batch size or the sequence length and the activation memory roughly doubles.

Activations are the one component you can trade for compute. With checkpointing, the model discards most of them and recomputes them during the backward pass, saving a large amount of memory at the cost of extra compute time. covers when that trade is worth it. A smaller batch with gradient accumulation gives a similar effect with a different cost.

Why LoRA changes the math

freezes the base model and trains small adapter matrices added to selected layers. Gradients and Adam states only exist for the adapter, so the 16 bytes apply to a tiny fraction of the parameters. The frozen base only needs its weights.

For a 7B model:

• Frozen base weights in 16-bit: about 14 GB
• Adapter of roughly 0.5% of the parameters, 35 million: 35 million x 16 bytes = about 0.56 GB
• Plus activations, which still depend on batch size and sequence length

QLoRA goes further by storing the frozen base in 4-bit, around 0.5 bytes per parameter, or about 3.5 GB for 7B plus some quantization overhead. The adapter is still trained in higher precision. The quality and speed trade-offs are explained in .

All three exclude activations and framework overhead. Treat the numbers as floors, not totals.

Where the rule bends

The 16-byte figure describes the standard mixed-precision recipe with a standard Adam optimizer. Other setups change it:

• 8-bit optimizers shrink the two Adam states from 8 bytes to about 2, dropping the total to roughly 10 bytes per parameter.
• Pure bf16 training with no separate master copy removes 4 bytes, at some cost in numerical precision.
• Sharding with ZeRO or FSDP splits weights, gradients and optimizer states across GPUs. In the ideal case, 112 GB across 8 GPUs is 14 GB each, before activations and communication buffers.
• CPU or disk offloading moves states out of GPU memory, trading speed for capacity.

Use the 16-byte rule as a starting point and adjust it for the recipe you actually run.

Which method should you choose?

A rough decision guide for a 7B model on a single 24 GB card:

• Full fine-tuning does not fit. You would need multiple GPUs or aggressive sharding.
• LoRA in 16-bit fits the base weights, but leaves limited room for activations, so use a small batch and checkpointing.
• QLoRA leaves plenty of room and is the practical default on a consumer card.

If you have no local GPU, you can still experiment: shows the workflow end to end.

Memory is only one constraint. Data matters just as much, and explains where the needs differ. Before you fine-tune at all, check whether retrieval would do the job cheaper with .

Estimate your own run

Use the . Enter the model, the batch size and the adapter configuration, and it compares the viable methods and shows where the gigabytes go. For a concrete case, read . If the answer says you need a bigger card, price it in the before committing.

Common questions

Why not just use 32-bit everywhere? It doubles memory and slows training, usually without improving the final model.

Does a longer sequence length cost more memory? Yes, mainly through activations, which grow with sequence length.

Can I reduce memory without changing the method? Yes: lower the batch size, enable checkpointing, use an 8-bit optimizer, or shorten sequences.

Checklist

• Count parameters x 16 bytes for full fine-tuning
• Add activations for your batch size and sequence length
• Compare against LoRA and QLoRA before renting bigger GPUs
• Reduce batch size or enable checkpointing before buying more memory
• Re-check with the calculator after every change in model or sequence length