Fine-Tuning Memory Calculator

Serving a 7B model at BF16 needs 14 GB. Fine-tuning the same model needs 112 GB, and the difference is not a rounding error in the framework — it is what the optimizer holds. Every trainable parameter carries four things at once: the bf16 weight, its bf16 gradient, an fp32 master copy that the optimizer actually updates, and the two fp32 AdamW moments (m and v). Two plus two plus four plus eight is sixteen bytes per parameter.

This is why full fine-tuning of even a 7B model is a multi-GPU exercise and why practical guides reach for adapters. It is also why "I have a 24 GB card, the model is 14 GB" is not a plan: the model file is the smallest part of the bill.

LoRA freezes the base model and inserts two small matrices beside each targeted projection. The base stays at 2 bytes per parameter, untouched by the optimizer, and only the adapter parameters carry the full 16-byte training cost. The adapter size is the sum over targeted modules of rank × (dIn + dOut) per layer — two low-rank factors instead of one full matrix.

• Attention-only adapters (q_proj, v_proj) are the cheapest starting point, and need no MLP width from the config.
• All seven projections — q, k, v, o, gate, up, down — is the quality-first setting, and roughly triples the adapter.
• Rank is the real dial: 8 for style and format tuning, 16–64 for new capabilities, and above 128 you usually need better data, not a bigger rank.

QLoRA loads the frozen base in 4-bit NF4 instead of bf16, which takes the base from 2 bytes per parameter to about 0.55 — the half-byte of the values plus the per-block quantization constants. The adapters are still trained in bf16 with a full optimizer state. For a 7B model that moves the base from 14 GB to roughly 4.2 GB, and it is what makes a 70B fine-tune possible on one 80 GB card.

The cost is speed and a small quality risk: every forward pass dequantizes blocks on the fly, so steps are slower than LoRA on the same hardware. Treat the 0.55 figure as an estimate — it varies with the block size and with how many layers stay in higher precision.

Weights and optimizer state are fixed by the model; activations are set by your data. Every layer keeps the intermediate tensors its backward pass needs, roughly sequence × batch × hidden × 34 bytes per layer in bf16 with flash attention. At 4,096 tokens, batch 1 and hidden 4,096 that is about 0.14 GB per layer — and a 32-layer model holds 4.5 GB of them.

Gradient checkpointing is the standard fix: keep only each layer’s inputs (2 bytes per element) and recompute the rest during backward. That cuts the stored term by seventeen times at the cost of roughly a third more compute, and it is why almost every memory-constrained recipe enables it. Turning flash attention off materialises the seq × seq score matrix per head, adding 5 × heads × seq² × batch bytes per layer — tolerable at 512 tokens and ruinous at 8,192.

• Framework differences. PyTorch with FSDP, DeepSpeed ZeRO-3 and Unsloth each shard or fuse differently. FSDP can spread optimizer state across cards, which changes the per-card total far more than any constant here.
• Fragmentation and allocator behaviour. PyTorch’s caching allocator keeps memory it does not hand back, so long runs sit above the arithmetic.
• Throughput. This page answers "how much memory", never "how long". Tokens per second depends on the engine and the hardware generation.
• The ±30% of the activation term. It is a rule of thumb, and the one number a framework can move substantially.