How Much VRAM Do You Need to Run an LLM Locally? Quantization Explained

"Will this model run on my GPU?" is one of the most common questions from people who want to run a large language model locally. The usual answers are vague: "should be fine" or "you will probably need to quantize". You do not have to guess. VRAM use can be estimated with simple arithmetic, and the biggest lever you control is quantization. This guide walks through the full calculation with real numbers for Llama 3 8B and 70B, so you can size any model yourself.

The short formula

Total VRAM ≈ weights + KV cache + runtime overhead

• Weights: parameter count × bytes per parameter. This is the largest and most predictable part.
• KV cache: memory the model uses to remember earlier tokens. It grows with context length and with the number of simultaneous conversations.
• Overhead: the CUDA context, temporary activation buffers and framework memory. Budget about 1 to 2 GB.

Step 1: the weights

Bytes per parameter depend on the number format the weights are stored in:

4-bit files such as GGUF Q4_K_M come out slightly larger than a pure 0.5 bytes per parameter, because some tensors are kept at higher precision to protect quality. Treat all of these numbers as estimates and check the actual file size of the model you download.

Example: 8 billion parameters × 2 bytes = 16 GB in FP16. Quantize to 4-bit and the same model needs roughly a quarter of that, which is the difference between needing a workstation card and fitting on a gaming GPU.

Step 2: the KV cache

While generating text, the model stores the keys and values of every earlier token so it does not recompute them at each step. The cost per token is:

KV bytes per token = 2 × layers × KV heads × head dimension × bytes per value

For Llama 3 8B (32 layers, 8 KV heads, head dimension 128) in FP16:

2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes = 128 KB per token

For Llama 3 70B (80 layers, 8 KV heads, head dimension 128) it is 320 KB per token. Multiply by the context length to see how fast it grows:

This is why a model that loads fine can still run out of memory when you paste in a long document. Serve several users at once and you multiply the cache by the number of active conversations. The full derivation is in our .

Step 3: add it up with real scenarios

Scenario A: Llama 3 8B, 4-bit, 8K context, one user

• Weights: ~4.9 GB
• KV cache: ~1 GB
• Overhead: ~1.5 GB
• Total: roughly 7.5 GB. Tight on an 8 GB card, comfortable on 12 GB.

Scenario B: Llama 3 8B, FP16, 32K context, one user

• Weights: ~16 GB
• KV cache: ~4 GB
• Overhead: ~1.5 GB
• Total: roughly 21.5 GB. Fits on a 24 GB card with little room to spare.

Scenario C: Llama 3 8B, 4-bit, 8K context, four users at once

• Weights: ~4.9 GB
• KV cache: 4 × 1 GB = 4 GB
• Overhead: ~1.5 GB
• Total: roughly 10.4 GB. Fits on a 12 GB card.

Scenario D: Llama 3 70B, 4-bit, 8K context, one user

• Weights: ~42 GB
• KV cache: ~2.5 GB
• Overhead: ~1.5 GB
• Total: roughly 46 GB. Does not fit on a single 24 GB card, and is tight even on 48 GB.

Which GPU fits what

Rough planning numbers for a single user:

Apple Silicon Macs use unified memory shared between the CPU and GPU, so a large share of system RAM can act as VRAM. That makes big quantized models possible on a laptop, usually at lower speed than a dedicated GPU.

Memory also decides speed

Generating one token requires reading essentially all the model weights from memory. That makes token generation mostly limited by memory bandwidth, not raw compute. A rough upper bound is:

tokens per second ≈ memory bandwidth ÷ model size in memory

On a GPU with 1,000 GB/s of bandwidth, a 4.9 GB 4-bit model has a ceiling of roughly 200 tokens per second, while the same model in FP16 (16 GB) tops out near 60. Real speeds are lower because of other overhead, but the ratio explains why quantization often makes a model faster as well as smaller.

What if the model does not fit?

• Use a lower-bit quantization. Step down from Q8 to Q5 to Q4. Quality loss is usually small at 8-bit and becomes more noticeable below 4 bits.
• Shorten the context window or reduce the number of parallel requests.
• Quantize the KV cache. Many runtimes support an 8-bit KV cache, which halves that part of the memory.
• Offload some layers to system RAM. The model runs, but generation slows down a lot, because RAM bandwidth is far below GPU memory bandwidth.
• Pick a smaller model. A well-chosen 8B model often beats a heavily squeezed 70B on simple tasks.

Quantization in plain words

Quantization stores each weight with fewer bits. Instead of 16 bits per number, you keep about 4 bits plus a small scale factor shared by a block of weights. When the model runs, the values are scaled back to approximate the originals. You lose a little precision, and you gain a much smaller and often faster model.

You will see several formats. GGUF is the file format used by llama.cpp and tools built on it, such as Ollama, and it is popular for running on consumer hardware and CPUs. GPTQ and AWQ are quantization methods commonly used for GPU inference. The details differ, but the memory math in this guide applies to all of them: bits per weight determine the size.

For most chat, summarization and coding tasks, 4-bit to 8-bit is a good trade. If you notice broken formatting, repeated text or worse reasoning, try the next higher precision before blaming the model.

Local or API: the cost question

A local GPU is a fixed cost, while an API charges per token. Which one is cheaper depends on your volume and how busy the GPU would be. Compare both with the , and use the to check your exact model, quantization and context length. Training needs far more memory than inference, so read before planning a fine-tuning run.

Common mistakes

• Counting only the weights. The KV cache and overhead can add several gigabytes.
• Forgetting context length. A 128K context on an 8B model can need as much memory as the weights themselves.
• Ignoring concurrency. Each active conversation has its own cache.
• Assuming CPU offload is free. It works, but speed can drop sharply.
• Running at 100% of VRAM. Leave headroom, because usage fluctuates and other programs may use the GPU.

Frequently asked questions

Does quantization always hurt quality? A little, and the loss grows as bits go down. At 8-bit it is usually hard to notice. At 4-bit it is acceptable for many tasks. Below that, test on your own prompts.

Can I run a 70B model on a 24 GB GPU? Not fully on the GPU at 4-bit, since the weights alone are around 40 GB. You can offload part of it to system RAM and accept slower generation.

Is system RAM or VRAM more important? VRAM, for speed. System RAM helps when you offload layers or run on CPU, but it is much slower.

Does a bigger context window always cost more? Yes. The KV cache grows linearly with context length, whether or not you use the full window.

Checklist

• Weights = parameters × bytes per parameter.
• KV cache = per-token cost × context length × concurrent users.
• Add 1 to 2 GB of overhead.
• Quantize to 4-bit or 8-bit before buying a bigger GPU.
• Leave headroom, because real usage fluctuates.