Running Llama 3 70B at full 16-bit precision needs roughly 140 GB of VRAM just for the weights, so in practice you run it quantized (about 40 GB at 4-bit) or across multiple GPUs. Llama 3 8B is far lighter: about 16 GB at 16-bit, or 5–6 GB at 4-bit, which fits on most modern consumer cards. The exact number depends on precision, quantization, context length, and the KV-cache — the sections below break each factor down.
Why VRAM Matters When Running LLMs
VRAM (video RAM) is the memory on your GPU. When you run a large language model (LLM) locally, the model's parameters — the learned numbers that define its behavior — have to live in VRAM for the GPU to compute with them efficiently. If they don't fit, you have three options: run on the CPU (much slower), offload part of the model to system RAM (also much slower for the offloaded layers), or split the model across multiple GPUs. In practice, VRAM capacity is the single biggest constraint on which models you can run and how fast.
VRAM matters for two reasons beyond raw fit. First, speed: tokens per second roughly scales with how much of the model sits in VRAM and how much memory bandwidth the GPU has. Second, headroom: the model weights are only part of the bill. You also need memory for the KV-cache (explained below), for the framework's overhead, and for the temporary buffers used during computation. A model that "just barely fits" often performs poorly because there's no room left for a reasonable context length.
It helps to separate two questions: can the model fit at all, and can it fit with the context length and batch size you actually want? The first is dominated by weight size; the second is dominated by the KV-cache and grows with context length and concurrent requests.
VRAM for Model Weights: Parameters, Precision and Quantization
The baseline calculation is simple. Each parameter is stored as a number, and the number of bits per parameter (the precision) determines the size. A useful rule of thumb:
weights_GB ≈ parameters × bits_per_parameter / 8 / 1e9
For example, 8 billion parameters at 16 bits: 8e9 × 16 / 8 = 16 GB. Add roughly 10–20% overhead for the runtime, and you get a practical estimate. Common precisions are FP32 (32 bits, rarely used for inference), FP16/BF16 (16 bits, the standard for full-quality inference), and 8-bit or 4-bit quantized formats.
Quantization means storing weights at lower precision — fewer bits per number — to shrink the model. An 8-bit model is about half the size of a 16-bit one; a 4-bit model is about a quarter. Quality loss at 8-bit is usually small; at 4-bit it's noticeable but often acceptable, especially with modern quantization methods that keep important weights at higher precision. Treat these as rules of thumb: actual file sizes vary by format and by how metadata and scaling factors are stored, so always check the model file size rather than trusting the formula alone.
A concrete way to use this: take the parameter count, multiply by bits, divide by 8, convert to gigabytes, then add 10–20% for overhead. That gives you a planning number for the weights, before you account for the KV-cache.
Llama 3 70B, 8B and Other Models: VRAM Estimates
Model weights are the first and largest VRAM cost. The baseline formula is:
VRAM (GB) ≈ parameters (billions) × bytes per parameter
Bytes per parameter depend on precision: FP32 uses 4 bytes, FP16/BF16 uses 2, INT8 uses 1, and 4-bit quantization uses roughly 0.5 (plus a small overhead for scales and metadata, so 4-bit is often closer to 0.55–0.7 bytes per parameter in practice).
Estimates for popular open models, using those assumptions:
| Model | FP16 (~2 B/param) | INT8 (~1 B/param) | 4-bit (~0.6 B/param) |
|---|---|---|---|
| Llama 3 8B | ~16 GB | ~8 GB | ~5–6 GB |
| Mistral 7B | ~14 GB | ~7 GB | ~4.5–5 GB |
| Llama 3 70B | ~140 GB | ~70 GB | ~40–45 GB |
| Mixtral 8x7B (47B active/total ~47B) | ~90–95 GB | ~45–47 GB | ~28–30 GB |
These are weight-only figures. You also need headroom for the KV-cache (next section), the CUDA/ROCm runtime, and framework overhead — typically 1–3 GB.
Worked example. Llama 3 70B at 4-bit: 70B parameters × 0.6 bytes ≈ 42 GB. Add 2 GB overhead → ~44 GB just for weights. That exceeds a single 24 GB or 48 GB card, so you need multiple GPUs (e.g., 2×48 GB or 4×24 GB) or CPU offloading. At INT8: 70 × 1 = 70 GB, plus overhead → ~72 GB.
Rule of thumb for quantized models:
VRAM ≈ params × bytes_per_param + KV_cache + 1–3 GB overhead
KV-Cache and Context Length: The Hidden VRAM Cost
The KV-cache (key-value cache) stores attention keys and values for every token already processed, so the model doesn't recompute them at each step. It grows linearly with context length and batch size, and it can dwarf the weights for long prompts.
Formula:
KV_cache_bytes ≈ 2 × layers × kv_heads × head_dim × seq_len × batch × bytes_per_value
The leading 2 accounts for keys and values. With grouped-query attention (GQA), kv_heads is much smaller than the model's query heads — Llama 3 8B, for example, uses 8 KV heads across 32 layers with head_dim 128.
Worked example (Llama 3 8B, FP16 cache, batch 1, 8,192 tokens): 2 × 32 layers × 8 kv_heads × 128 head_dim × 8,192 × 2 bytes = 2 × 32 × 8 × 128 × 8,192 × 2 = ~1.07 GB
Double the context to 16K → ~2.1 GB; at 128K it approaches ~17 GB, which is why long-context inference on an 8B model can need far more VRAM than the weights suggest.
Checklist before sizing hardware:
- Pick quant level (FP16/INT8/4-bit) and compute weight VRAM
- Estimate KV-cache at your target context × batch size
- Add 1–3 GB runtime overhead
- Confirm total fits on one GPU, or plan tensor parallelism / offloading
- Leave 10–20% headroom for fragmentation and spikes
Frequently Asked Questions
How much VRAM do I need to run Llama 3 70B?
Llama 3 70B has 70 billion parameters, and in FP16 precision (2 bytes per parameter) the weights alone need roughly 140 GB of VRAM, which exceeds any single consumer GPU. With 4-bit quantization the weights shrink to about 35–40 GB, so a single 48 GB card or a multi-GPU setup becomes viable. Add several GB for the KV-cache and runtime overhead, and expect roughly 40–45 GB at 4-bit for short contexts, more as context grows.
Can I run Llama 3 8B on a single consumer GPU?
Yes. At FP16 the 8B weights take about 16 GB, so a 24 GB card runs it comfortably with room for context. With 4-bit quantization the weights drop to roughly 4–5 GB, which fits on 8 GB cards at modest context lengths. Expect total usage of about 6–8 GB at 4-bit, leaving headroom on most modern mid-range GPUs.
How does quantization reduce VRAM requirements?
Quantization stores each weight in fewer bits: FP16 uses 16 bits per parameter, while 8-bit and 4-bit schemes use 8 or 4. Because VRAM for weights scales linearly with bit width, 4-bit cuts weight memory to roughly a quarter of FP16. The trade-off is a small quality loss that grows as precision drops, so 8-bit is nearly lossless while 4-bit is usually acceptable for many tasks.
What is the KV-cache and how does it affect VRAM usage?
The KV-cache (key-value cache) stores attention keys and values for every token already processed so the model doesn't recompute them at each step. Its size grows with context length, batch size, layer count and hidden dimension, so it can add several GB on top of the weights. Long conversations or large batches can make the cache a significant share of total VRAM, sometimes rivaling the model itself.
Does longer context length increase VRAM consumption?
Yes, roughly linearly. Doubling the context length roughly doubles the KV-cache size, since one entry is kept per token per layer. So a model that fits at 4K tokens may run out of memory at 32K unless you shrink the batch, use a smaller cache precision, or rely on paged or sliding-window attention.