Full fine-tuning of Llama 3 8B typically needs roughly 60–100 GB of GPU memory once you account for weights, gradients, optimizer states, and activations. Parameter-efficient methods like LoRA bring that down to roughly 16–30 GB, and QLoRA can fit the job on a single 24 GB card with careful settings. The exact number depends on batch size, sequence length, and precision, so the honest answer is a range, not a single figure.
Why Llama 3 8B Memory Math Matters
Llama 3 8B is an "8 billion parameter" model, meaning its learned weights number about 8,000,000,000. Parameters are the numbers inside the model that training adjusts; each one occupies memory, usually 1, 2, or 4 bytes depending on precision. Before you rent a GPU by the hour or buy one, it helps to know that memory is consumed by four separate things during fine-tuning: the weights themselves, the gradients (the corrections computed for each weight), the optimizer states (bookkeeping that the optimizer keeps per weight), and activations (intermediate values saved during the forward pass so the backward pass can use them). Each has its own arithmetic, and the total is what must fit in VRAM, the dedicated memory on your graphics card. Getting the estimate wrong by a factor of two is the difference between a smooth run and an out-of-memory crash twenty minutes in.
Memory math also decides your method. If full fine-tuning needs 80 GB or more, a single consumer card is off the table, and you should be looking at LoRA or QLoRA instead. Knowing the numbers up front turns "can I do this?" into a concrete plan: which precision, which method, which batch size, which GPU. The estimates below are derived from the parameter count and standard training conventions; treat them as reasoned ranges, not measured benchmarks, since actual usage varies with framework, implementation details, and settings.
Counting the Bytes: Weights, Gradients, and Optimizer States
Start with the weights. In 16-bit precision (2 bytes per parameter), 8B parameters take about 16 GB. In 8-bit, about 8 GB; in 4-bit, about 4 GB. For full fine-tuning, the convention is mixed precision: weights stored in 16-bit for the forward and backward passes, with a 32-bit master copy for the optimizer. That master copy costs 4 bytes per parameter, roughly 32 GB.
Gradients add another 2 bytes per parameter in 16-bit, about 16 GB. Optimizer states depend on the algorithm. Adam, the most common choice, keeps two running averages per parameter (first and second moments), each 4 bytes, so about 64 GB. Put together, full fine-tuning in mixed precision with Adam lands around:
weights (16-bit) ~16 GB
master weights (32-bit) ~32 GB
gradients (16-bit) ~16 GB
Adam states (2 x 32-bit) ~64 GB
---------------------------------
subtotal ~128 GB
That subtotal excludes activations, which we cover next, and is why full fine-tuning of an 8B model is a multi-GPU affair: with sharding across, say, four 40 GB cards or two 80 GB cards, it becomes feasible. The subtotal shrinks if you use 8-bit optimizer states (roughly halving the Adam cost to about 32 GB) or skip the 32-bit master copy, but the order of magnitude stays the same. LoRA changes the picture entirely: it freezes the original weights and trains only small adapter matrices, so gradients and optimizer states apply to a tiny fraction of parameters, often under 1% of the total. That is why LoRA runs in the 16–30 GB range.
Activations, Batch Size, and Sequence Length
Activations are the memory used to store intermediate results from each layer during the forward pass, because the backward pass needs them to compute gradients. Their cost scales with batch size (how many sequences you process at once) and sequence length (how many tokens per sequence), and it is the most variable term in the budget. A rough rule of thumb for transformer models is that activation memory grows roughly linearly with batch size and with sequence length, and it is multiplied by the number of layers and hidden dimensions.
Concretely, for Llama 3 8B, training on short sequences (a few hundred tokens) with a small batch can keep activations in the low single-digit gigabytes. Push to 4,096-token sequences with a batch of 8, and activations can climb into the tens of gigabytes. Techniques like gradient checkpointing trade compute for memory: instead of storing every intermediate, the model recomputes some during the backward pass, cutting activation memory substantially at the cost of extra forward passes. If you are tight on VRAM, gradient checkpointing plus a smaller batch or shorter sequences is usually the first lever to pull. In practice, budget activations as a flexible 5–30 GB depending on your settings, and remember that sequence length is often the dominant factor.
Practical Rig: LoRA, Quantization, and Choosing Your GPU
For most practitioners, the practical path is LoRA (Low-Rank Adaptation): freeze the base weights and train small low-rank matrices injected into the model's layers. Memory drops to roughly the frozen base weights plus a small adapter's gradients and optimizer states, plus activations. In 16-bit, that puts you around 20–30 GB; with 8-bit base weights, closer to 12–20 GB.
QLoRA goes further: it quantizes the frozen base weights to 4-bit (about 4 GB for 8B parameters) while training 16-bit adapters, which is how fine-tuning fits on a single 24 GB GPU. Yes, QLoRA reduces memory compared with standard LoRA, because the base weights shrink from 16 GB to about 4 GB; the adapters and activations are unchanged, so the savings are real but bounded. On a 24 GB card, expect to need gradient checkpointing and modest batch sizes. On 40 GB, you have comfortable headroom for LoRA at longer sequences. On 80 GB, LoRA is roomy and full fine-tuning with sharding or 8-bit optimizers becomes plausible. As for batch size: on 40 GB with LoRA and gradient checkpointing, a batch of 4–8 at moderate sequence lengths is a reasonable starting estimate; on 80 GB, 8–16. Treat these as starting points to tune, not guarantees.
FAQ
How much VRAM does full fine-tuning of Llama 3 8B require? Roughly 60–100 GB or more in mixed precision with Adam, driven mainly by the 32-bit master weights, gradients, and two 32-bit optimizer states. With 8-bit optimizer states the low end drops, but you still need multi-GPU sharding in most cases.
Can I fine-tune Llama 3 8B on a single 24GB GPU? Yes, with QLoRA (4-bit base weights plus 16-bit adapters), gradient checkpointing, and small batches or shorter sequences. Full fine-tuning will not fit.
Does QLoRA reduce memory compared to standard LoRA? Yes. The base weights drop from about 16 GB in 16-bit to about 4 GB in 4-bit. Adapter and activation memory are essentially unchanged, so the saving is the quantization of the frozen weights.
How does sequence length affect activation memory during fine-tuning? Activation memory grows roughly linearly with sequence length (and with batch size), so doubling sequence length roughly doubles that term. Long-context training is usually where memory budgets blow up; gradient checkpointing is the standard mitigation.
What batch size fits on a 40GB or 80GB GPU? With LoRA and gradient checkpointing, estimate roughly 4–8 on 40 GB and 8–16 on 80 GB at moderate sequence lengths. These are starting ranges derived from the memory terms above; actual limits depend on sequence length, precision, and framework overhead, so tune from there.
Activations, Batch Size, and Sequence Length
Weights and optimizer states are fixed costs. Activations are the variable that surprises people: every forward pass stores intermediate tensors so the backward pass can compute gradients, and that memory scales with batch size and sequence length. During fine-tuning, activations for a batch of B sequences of length L are roughly:
activation bytes ≈ B × L × hidden × layers × bytes_per_value × checkpointing_factor
For Llama 3 8B, use hidden = 4096 and layers = 32. Without gradient checkpointing, the factor lands around 10–20 (exact value depends on implementation details); with it, roughly 1–3, because only layer boundaries are stored and intermediates are recomputed during backward. bytes_per_value is 2 for bf16/fp16.
Worked example, double-checked: bf16, no checkpointing, factor 16, batch 1, sequence 1024.
B × L = 1024 tokens
1024 × 4096 × 32 × 2 × 16 bytes
= 1024 × 4096 = 4,194,304
× 32 = 134,217,728
× 2 = 268,435,456
× 16 = 4,294,967,296 bytes ≈ 4 GiB
Doubling sequence length to 2048 or batch to 2 doubles this to ~8 GiB; both together give ~16 GiB. That is why long-context fine-tuning fails on cards that handle short sequences fine. Gradient checkpointing cuts activation memory several-fold at the cost of roughly 30% slower steps (an estimate; overhead varies).
Practical Rig: LoRA, Quantization, and Choosing Your GPU
LoRA (Low-Rank Adaptation) freezes the base weights and trains small adapter matrices, so gradients and optimizer states apply only to a fraction of parameters—often well under 1% of the model. Full fine-tuning in bf16 needs roughly 16 bytes per parameter (2 weights + 2 gradients + 4 fp32 master weights + 8 Adam moments): about 128 GB for 8B parameters, before activations. Estimate, not benchmark.
QLoRA adds 4-bit quantization of the frozen base weights (about 0.5 bytes per parameter, ~4 GB), with adapters trained in higher precision. Compared with standard LoRA on bf16 weights (~16 GB base), QLoRA cuts base-weight memory roughly fourfold (estimate); activation memory is unchanged.
Checklist before picking a GPU:
- Budget base weights by method: full bf16 ~16 GB, LoRA bf16 ~16 GB frozen base, QLoRA ~4–6 GB (estimates).
- Add 16 bytes per trainable parameter for gradients and optimizer states.
- Estimate activations with the formula above; enable gradient checkpointing if tight.
- Leave 2–4 GB headroom for CUDA context, fragmentation, and evaluation batches.
- Start with batch 1 and gradient accumulation; scale up only after measuring.
Rough fit (estimates): a 24 GB card handles QLoRA comfortably at short-to-moderate sequences; 40 GB opens standard LoRA with checkpointing; 80 GB supports larger batches or longer context, and full fine-tuning still needs multi-GPU setups.
Frequently Asked Questions
How much VRAM does full fine-tuning of Llama 3 8B require?
Full fine-tuning trains all 8 billion parameters and keeps a full-precision copy of each parameter's gradient and optimizer state, so the memory is far larger than the weights alone. A common starting estimate is roughly 16 bytes per parameter in mixed-precision Adam, which puts you near 120–130 GB before activations. Adding activations, gradients in higher precision, and fragmentation pushes realistic requirements to somewhere around 150–200 GB, which means multiple 80GB GPUs or aggressive sharding such as ZeRO or FSDP. Treat these as order-of-magnitude estimates that scale with batch size and sequence length.
Can I fine-tune Llama 3 8B on a single 24GB GPU?
Yes, but only with parameter-efficient methods, not full fine-tuning. With QLoRA (4-bit quantized base weights plus small trainable adapters), an 8B model typically fits in roughly 10–14 GB at modest batch sizes and sequence lengths, leaving headroom on a 24GB card. Standard LoRA in 16-bit is tighter and often needs careful batch and sequence settings to avoid running out of memory. Expect slower training and limited batch size, but it is workable.
Does QLoRA reduce memory compared to standard LoRA?
Yes. Standard LoRA keeps the frozen base model in 16-bit and adds small trainable low-rank matrices, so the base weights alone occupy about 16 GB for 8B parameters. QLoRA quantizes those frozen base weights to 4-bit (roughly 4–5 GB) while training the same style of adapters in higher precision, cutting the dominant memory cost several-fold. The trade-off is extra dequantization compute, so training is typically slower per step.
How does sequence length affect activation memory during fine-tuning?
Activations are the intermediate tensors saved during the forward pass so gradients can be computed in the backward pass, and their memory grows with batch size times sequence length (and, with attention, often roughly quadratically in sequence length). Doubling sequence length can therefore more than double activation memory, which is why long-context fine-tuning is the usual cause of out-of-memory errors. Techniques like gradient checkpointing trade compute for memory by recomputing activations instead of storing them, and shorter sequences or smaller micro-batches relieve pressure directly.
What batch size fits on a 40GB or 80GB GPU?
There is no fixed number, because it depends on method, precision, sequence length, and whether you use gradient checkpointing; use ranges as planning estimates only. On 40GB with QLoRA and moderate sequences, batch sizes of several examples are often feasible, while 16-bit LoRA leaves much less room. On 80GB you can usually push batch size and sequence length higher, and full fine-tuning may become possible only with sharding across multiple such GPUs. The practical approach is to start with a micro-batch of 1, enable gradient checkpointing, and increase until memory is nearly full, using gradient accumulation to reach your effective batch size.