Full fine-tune
Every weight is trained: 16 bytes per parameter.
123.80 GB
- Trainable
- 7,615.6M (100.00%)
- Weights
- 15.23 GB
- Grad + optimizer
- 106.62 GB
- Activations
- 0.33 GB
- Overhead
- 1.62 GB
Fine-Tuning Memory Calculator
Training a model costs far more memory than serving it. Put in the model, the batch and the adapter you have in mind to see the three viable methods compared, and where the tens of gigabytes actually go.
Runs entirely in your browser — nothing you type leaves the page.
Memory by method
Qwen2.5 7B Instruct · 1.0K tokens × 1 · GeForce RTX 4090
Does not fit
LoRA
Frozen bf16 base, small trained adapter.
17.27 GB
- Trainable
- 5.0M (0.07%)
- Weights
- 15.23 GB
- Adapters
- 0.08 GB
- Activations
- 0.33 GB
- Overhead
- 1.62 GB
QLoRA
Frozen 4-bit base, small trained adapter.
6.22 GB
- Trainable
- 5.0M (0.07%)
- Weights
- 4.19 GB
- Adapters
- 0.08 GB
- Activations
- 0.33 GB
- Overhead
- 1.62 GB
- Weights
- Grad + optimizer
- Adapters
- Activations
- Overhead
Adapter parameters
5,046,272
Costed at 16 bytes each, including optimizer state.
Tokens per step
1.0K
Sequence length × micro-batch.
How we calculate this
Everything below is arithmetic on your settings. It is a planning estimate, not a measurement.
- full fine-tune = total params × (2 + 2 + 4 + 8) bytes = 16 bytes per parameter
- optimizer state = 14 bytes (AdamW) / 8 (8-bit) / 10 (SGD + momentum)
- adapter params = layers × Σ rank × (d_in + d_out)
- LoRA base = 2 bytes/param · QLoRA base = 0.55 bytes/param
- activations = layers × seq × batch × hidden × 34 bytes (±30%)
- with checkpointing: layers × seq × batch × hidden × 2 + one full layer
- flash attention off: + 5 × heads × seq² × batch per layer
- overhead = 1 GB + batch × seq × vocab × 4 bytes
- The activation term is the least certain number here; real usage varies by roughly ±30%.
- PyTorch, Unsloth, DeepSpeed and FSDP all differ. Sharded training can divide the state across cards, so treat this as one device's total.
- Adapters already include a mixed-precision AdamW state, so the optimizer choice only changes the full fine-tune column.
- Estimates, not guarantees. Measure on your own setup before committing to hardware.
LoRA vs QLoRA: choosing an efficient fine-tune →Check the model fits for inference →
Why training needs about sixteen bytes per parameter
Serving a 7B model at BF16 needs 14 GB. Fine-tuning the same model needs 112 GB, and the difference is not a rounding error in the framework — it is what the optimizer holds. Every trainable parameter carries four things at once: the bf16 weight, its bf16 gradient, an fp32 master copy that the optimizer actually updates, and the two fp32 AdamW moments (m and v). Two plus two plus four plus eight is sixteen bytes per parameter.
full fine-tune = total_params × (2 + 2 + 4 + 8) bytes
| Optimizer | Bytes per trainable parameter | 7B model |
|---|---|---|
| AdamW (mixed precision) | 16 | 112 GB |
| AdamW 8-bit (approx.) | 10 | 70 GB |
| SGD + momentum | 12 | 84 GB |
This is why full fine-tuning of even a 7B model is a multi-GPU exercise and why practical guides reach for adapters. It is also why "I have a 24 GB card, the model is 14 GB" is not a plan: the model file is the smallest part of the bill.
What LoRA changes: fewer trainable parameters, not smaller weights
LoRA freezes the base model and inserts two small matrices beside each targeted projection. The base stays at 2 bytes per parameter, untouched by the optimizer, and only the adapter parameters carry the full 16-byte training cost. The adapter size is the sum over targeted modules of rank × (dIn + dOut) per layer — two low-rank factors instead of one full matrix.
adapter_params = layers × Σ rank × (d_in + d_out)
- Attention-only adapters (
q_proj,v_proj) are the cheapest starting point, and need no MLP width from the config. - All seven projections — q, k, v, o, gate, up, down — is the quality-first setting, and roughly triples the adapter.
- Rank is the real dial: 8 for style and format tuning, 16–64 for new capabilities, and above 128 you usually need better data, not a bigger rank.
NoteThe frozen base is still in memory
LoRA saves optimizer state, not the model. A 7B base still occupies 14 GB at bf16 before a single adapter byte is counted, which is why a LoRA run on a 24 GB card is comfortable for a 7B model and impossible for a 70B one.
What QLoRA adds: a quantized base
QLoRA loads the frozen base in 4-bit NF4 instead of bf16, which takes the base from 2 bytes per parameter to about 0.55 — the half-byte of the values plus the per-block quantization constants. The adapters are still trained in bf16 with a full optimizer state. For a 7B model that moves the base from 14 GB to roughly 4.2 GB, and it is what makes a 70B fine-tune possible on one 80 GB card.
The cost is speed and a small quality risk: every forward pass dequantizes blocks on the fly, so steps are slower than LoRA on the same hardware. Treat the 0.55 figure as an estimate — it varies with the block size and with how many layers stay in higher precision.
Activations: the term that scales with your batch
Weights and optimizer state are fixed by the model; activations are set by your data. Every layer keeps the intermediate tensors its backward pass needs, roughly sequence × batch × hidden × 34 bytes per layer in bf16 with flash attention. At 4,096 tokens, batch 1 and hidden 4,096 that is about 0.14 GB per layer — and a 32-layer model holds 4.5 GB of them.
Gradient checkpointing is the standard fix: keep only each layer’s inputs (2 bytes per element) and recompute the rest during backward. That cuts the stored term by seventeen times at the cost of roughly a third more compute, and it is why almost every memory-constrained recipe enables it. Turning flash attention off materialises the seq × seq score matrix per head, adding 5 × heads × seq² × batch bytes per layer — tolerable at 512 tokens and ruinous at 8,192.
Watch outSequence length is quadratic, batch is linear
If you are out of memory, halving the batch halves the activation bill; halving the sequence length halves it too but also shrinks the quadratic attention buffer by four. Gradient accumulation keeps your effective batch size while you train on a smaller micro-batch — almost always the right first move.
What this page does not model
- Framework differences. PyTorch with FSDP, DeepSpeed ZeRO-3 and Unsloth each shard or fuse differently. FSDP can spread optimizer state across cards, which changes the per-card total far more than any constant here.
- Fragmentation and allocator behaviour. PyTorch’s caching allocator keeps memory it does not hand back, so long runs sit above the arithmetic.
- Throughput. This page answers "how much memory", never "how long". Tokens per second depends on the engine and the hardware generation.
- The ±30% of the activation term. It is a rule of thumb, and the one number a framework can move substantially.
NoteEverything runs in this tab
The architecture numbers come from each model’s official config.json, and the arithmetic runs in your browser: no request leaves the page, nothing is uploaded and no training data is involved. Where a model does not publish a field the tool needs, it says "needs data" instead of inventing one.
Frequently asked questions
- How much memory does it take to fine-tune a 7B model?
- A full fine-tune needs about 112 GB before activations: 7 billion parameters × 16 bytes each, because every parameter carries a bf16 weight, a bf16 gradient, an fp32 master copy and two fp32 AdamW moments. LoRA keeps the same 14 GB base but trains a small adapter, so it fits in a 24 GB card at a short context. QLoRA loads the base in 4-bit and drops that to about 4.2 GB, which leaves room for a bigger batch or a longer sequence.
- Why sixteen bytes per parameter and not six?
- Because the model weights are the smallest thing in a training step. Start from bf16 weights (2) and bf16 gradients (2), then add what the optimizer needs to take a reliable step: an fp32 master copy (4) so small updates are not lost to rounding, and AdamW’s two fp32 moments (8). That is 16. Anything that quotes a smaller number is either counting only the weights or using an optimizer that keeps less state.
- Does LoRA reduce the memory of the base model?
- No. LoRA freezes the base, so it still occupies its full size — 14 GB for a 7B model at bf16 — and removes only the gradient and optimizer cost for those parameters. The thing that shrinks the base is QLoRA, which loads it in 4-bit. If a 7B base does not fit on your card, LoRA will not save you; quantize the base instead.
- What does gradient checkpointing cost me?
- Time, mostly. It stores only each layer’s inputs and recomputes the intermediate tensors during the backward pass, cutting activation memory by roughly a factor of seventeen at a typical sequence length while adding about a third more compute. For most single-GPU setups it is the difference between running and not running, so the extra time is well spent.
- How should I choose the LoRA rank?
- Start at 8 to 16 for style, format and tone adaptation, and go to 32–64 when you are teaching a genuinely new behaviour or a new domain vocabulary. Rank is linear in adapter memory, so doubling it doubles the adapter — but the adapter is a rounding error next to the frozen base, which is why rank is usually chosen for quality rather than for memory.
- Which projections should I target?
- The attention projections (q and v at minimum, all four if memory allows) capture most of the benefit. Adding the MLP projections — gate, up and down — gives the adapter more capacity to store new facts and roughly triples its parameter count. If you need the MLP widths for those modules and the model’s config does not publish them, this page shows "needs data" rather than estimating.
- Is the estimate accurate for DeepSpeed or FSDP?
- Not directly. Those frameworks shard optimizer state and sometimes parameters across GPUs, so the per-card figure can be far lower than the single-GPU total this calculator reports. Read the total as "what one device would need to hold everything", then divide by the sharding you have configured — and measure, because sharding introduces its own buffers.
Learn this in the course
This tool is the hands-on companion to these stages of the free 11-stage track.
Related tools
- LLM VRAM Calculator →
Pick a model and a card to see how many gigabytes it really takes to serve — weights, the KV cache your context length creates, and the runtime overhead nobody mentions. Every formula is printed on the page.
- LLM API Cost Calculator →
Turn a token count and a traffic guess into a real monthly bill. Set the prompt size, the answer size and how many requests you expect, and every priced model is compared side by side.