全量微调
每个权重都参与训练：每参数 16 字节。
123.80 GB
- 可训练参数
- 7,615.6M (100.00%)
- 权重
- 15.23 GB
- 梯度 + 优化器
- 106.62 GB
- 激活
- 0.33 GB
- 运行时开销
- 1.62 GB
Fine-Tuning Memory Calculator
Training a model costs far more memory than serving it. Put in the model, the batch and the adapter you have in mind to see the three viable methods compared, and where the tens of gigabytes actually go.
完全在浏览器中运行 —— 你输入的内容不会离开本页。
⚠ 此工具尚未完整翻译为中文，因此显示英文版本。翻译正在进行中。
各方法的显存
Qwen2.5 7B Instruct · 1.0K token × 1 · GeForce RTX 4090
装不下
LoRA
冻结的 bf16 基础模型加小型可训练适配器。
17.27 GB
- 可训练参数
- 5.0M (0.07%)
- 权重
- 15.23 GB
- 适配器
- 0.08 GB
- 激活
- 0.33 GB
- 运行时开销
- 1.62 GB
QLoRA
冻结的 4 比特基础模型加小型可训练适配器。
6.22 GB
- 可训练参数
- 5.0M (0.07%)
- 权重
- 4.19 GB
- 适配器
- 0.08 GB
- 激活
- 0.33 GB
- 运行时开销
- 1.62 GB
- 权重
- 梯度 + 优化器
- 适配器
- 激活
- 运行时开销
适配器参数量
5,046,272
按每参数 16 字节计算，含优化器状态。
每步 token 数
1.0K
序列长度 × 微批大小。
我们如何计算
以下全部是对你设置的算术。这是规划估算，不是实测值。
- 全量微调 = 总参数量 × (2 + 2 + 4 + 8) 字节 = 每参数 16 字节
- 优化器状态 = 14 字节（AdamW）/ 8（8 比特）/ 10（SGD + 动量）
- 适配器参数 = 层数 × Σ 秩 × (d_in + d_out)
- LoRA 基础 = 2 字节/参数 · QLoRA 基础 = 0.55 字节/参数
- 激活 = 层数 × 序列 × 批 × 隐藏 × 34 字节（±30%）
- 启用检查点：层数 × 序列 × 批 × 隐藏 × 2 + 一层的完整激活
- 关闭 flash attention：每层 + 5 × 头数 × 序列² × 批
- 开销 = 1 GB + 批 × 序列 × 词表 × 4 字节
- 激活项是这里最不确定的数字，实际使用量大约上下浮动 30%。
- PyTorch、Unsloth、DeepSpeed 与 FSDP 各不相同。分片会把状态分散到多张卡，所以这里是单卡的合计。
- 适配器本身已包含混合精度的 AdamW 状态，因此优化器选择只影响全量微调那一列。
- 这是估算，不是保证。购买硬件前请在自己的环境实测。
Why training needs about sixteen bytes per parameter
Serving a 7B model at BF16 needs 14 GB. Fine-tuning the same model needs 112 GB, and the difference is not a rounding error in the framework — it is what the optimizer holds. Every trainable parameter carries four things at once: the bf16 weight, its bf16 gradient, an fp32 master copy that the optimizer actually updates, and the two fp32 AdamW moments (m and v). Two plus two plus four plus eight is sixteen bytes per parameter.
full fine-tune = total_params × (2 + 2 + 4 + 8) bytes
| Optimizer | Bytes per trainable parameter | 7B model |
|---|---|---|
| AdamW (mixed precision) | 16 | 112 GB |
| AdamW 8-bit (approx.) | 10 | 70 GB |
| SGD + momentum | 12 | 84 GB |
This is why full fine-tuning of even a 7B model is a multi-GPU exercise and why practical guides reach for adapters. It is also why "I have a 24 GB card, the model is 14 GB" is not a plan: the model file is the smallest part of the bill.
What LoRA changes: fewer trainable parameters, not smaller weights
LoRA freezes the base model and inserts two small matrices beside each targeted projection. The base stays at 2 bytes per parameter, untouched by the optimizer, and only the adapter parameters carry the full 16-byte training cost. The adapter size is the sum over targeted modules of rank × (dIn + dOut) per layer — two low-rank factors instead of one full matrix.
adapter_params = layers × Σ rank × (d_in + d_out)
- Attention-only adapters (
q_proj,v_proj) are the cheapest starting point, and need no MLP width from the config. - All seven projections — q, k, v, o, gate, up, down — is the quality-first setting, and roughly triples the adapter.
- Rank is the real dial: 8 for style and format tuning, 16–64 for new capabilities, and above 128 you usually need better data, not a bigger rank.
NoteThe frozen base is still in memory
LoRA saves optimizer state, not the model. A 7B base still occupies 14 GB at bf16 before a single adapter byte is counted, which is why a LoRA run on a 24 GB card is comfortable for a 7B model and impossible for a 70B one.
What QLoRA adds: a quantized base
QLoRA loads the frozen base in 4-bit NF4 instead of bf16, which takes the base from 2 bytes per parameter to about 0.55 — the half-byte of the values plus the per-block quantization constants. The adapters are still trained in bf16 with a full optimizer state. For a 7B model that moves the base from 14 GB to roughly 4.2 GB, and it is what makes a 70B fine-tune possible on one 80 GB card.
The cost is speed and a small quality risk: every forward pass dequantizes blocks on the fly, so steps are slower than LoRA on the same hardware. Treat the 0.55 figure as an estimate — it varies with the block size and with how many layers stay in higher precision.
Activations: the term that scales with your batch
Weights and optimizer state are fixed by the model; activations are set by your data. Every layer keeps the intermediate tensors its backward pass needs, roughly sequence × batch × hidden × 34 bytes per layer in bf16 with flash attention. At 4,096 tokens, batch 1 and hidden 4,096 that is about 0.14 GB per layer — and a 32-layer model holds 4.5 GB of them.
Gradient checkpointing is the standard fix: keep only each layer’s inputs (2 bytes per element) and recompute the rest during backward. That cuts the stored term by seventeen times at the cost of roughly a third more compute, and it is why almost every memory-constrained recipe enables it. Turning flash attention off materialises the seq × seq score matrix per head, adding 5 × heads × seq² × batch bytes per layer — tolerable at 512 tokens and ruinous at 8,192.
Watch outSequence length is quadratic, batch is linear
If you are out of memory, halving the batch halves the activation bill; halving the sequence length halves it too but also shrinks the quadratic attention buffer by four. Gradient accumulation keeps your effective batch size while you train on a smaller micro-batch — almost always the right first move.
What this page does not model
- Framework differences. PyTorch with FSDP, DeepSpeed ZeRO-3 and Unsloth each shard or fuse differently. FSDP can spread optimizer state across cards, which changes the per-card total far more than any constant here.
- Fragmentation and allocator behaviour. PyTorch’s caching allocator keeps memory it does not hand back, so long runs sit above the arithmetic.
- Throughput. This page answers "how much memory", never "how long". Tokens per second depends on the engine and the hardware generation.
- The ±30% of the activation term. It is a rule of thumb, and the one number a framework can move substantially.
NoteEverything runs in this tab
The architecture numbers come from each model’s official config.json, and the arithmetic runs in your browser: no request leaves the page, nothing is uploaded and no training data is involved. Where a model does not publish a field the tool needs, it says "needs data" instead of inventing one.
常见问题
- How much memory does it take to fine-tune a 7B model?
- A full fine-tune needs about 112 GB before activations: 7 billion parameters × 16 bytes each, because every parameter carries a bf16 weight, a bf16 gradient, an fp32 master copy and two fp32 AdamW moments. LoRA keeps the same 14 GB base but trains a small adapter, so it fits in a 24 GB card at a short context. QLoRA loads the base in 4-bit and drops that to about 4.2 GB, which leaves room for a bigger batch or a longer sequence.
- Why sixteen bytes per parameter and not six?
- Because the model weights are the smallest thing in a training step. Start from bf16 weights (2) and bf16 gradients (2), then add what the optimizer needs to take a reliable step: an fp32 master copy (4) so small updates are not lost to rounding, and AdamW’s two fp32 moments (8). That is 16. Anything that quotes a smaller number is either counting only the weights or using an optimizer that keeps less state.
- Does LoRA reduce the memory of the base model?
- No. LoRA freezes the base, so it still occupies its full size — 14 GB for a 7B model at bf16 — and removes only the gradient and optimizer cost for those parameters. The thing that shrinks the base is QLoRA, which loads it in 4-bit. If a 7B base does not fit on your card, LoRA will not save you; quantize the base instead.
- What does gradient checkpointing cost me?
- Time, mostly. It stores only each layer’s inputs and recomputes the intermediate tensors during the backward pass, cutting activation memory by roughly a factor of seventeen at a typical sequence length while adding about a third more compute. For most single-GPU setups it is the difference between running and not running, so the extra time is well spent.
- How should I choose the LoRA rank?
- Start at 8 to 16 for style, format and tone adaptation, and go to 32–64 when you are teaching a genuinely new behaviour or a new domain vocabulary. Rank is linear in adapter memory, so doubling it doubles the adapter — but the adapter is a rounding error next to the frozen base, which is why rank is usually chosen for quality rather than for memory.
- Which projections should I target?
- The attention projections (q and v at minimum, all four if memory allows) capture most of the benefit. Adding the MLP projections — gate, up and down — gives the adapter more capacity to store new facts and roughly triples its parameter count. If you need the MLP widths for those modules and the model’s config does not publish them, this page shows "needs data" rather than estimating.
- Is the estimate accurate for DeepSpeed or FSDP?
- Not directly. Those frameworks shard optimizer state and sometimes parameters across GPUs, so the per-card figure can be far lower than the single-GPU total this calculator reports. Read the total as "what one device would need to hold everything", then divide by the sharding you have configured — and measure, because sharding introduces its own buffers.
在课程中学习
此工具是免费 11 阶段课程中以下阶段的动手实践配套。
相关工具
- LLM VRAM Calculator →
Pick a model and a card to see how many gigabytes it really takes to serve — weights, the KV cache your context length creates, and the runtime overhead nobody mentions. Every formula is printed on the page.
- LLM API Cost Calculator →
Turn a token count and a traffic guess into a real monthly bill. Set the prompt size, the answer size and how many requests you expect, and every priced model is compared side by side.