LLM VRAM Calculator

When a language model is loaded, the card holds three different things, and only one of them is the model file you downloaded. First the weights: one number per parameter, at whatever precision you chose. Second the KV cache: the keys and values of every token the model has already seen, kept so that the next token does not have to re-read the whole conversation. Third the runtime overhead: the engine, its allocator, kernel buffers and the CUDA context, which cost memory whether or not a single token is generated.

• Weights scale with the number of parameters and the precision. They are the same size at 500 tokens of context and at 50,000.
• KV cache scales with context length, batch size and the number of layers — it is the term that turns a comfortable fit into an out-of-memory crash.
• Overhead is roughly fixed per process, which is why a model that "just fits" on paper usually does not fit in practice.

Weight memory is plain multiplication: parameters × bytes per parameter. A 7-billion-parameter model at BF16 is 7 × 10⁹ × 2 = 14 × 10⁹ bytes, or 14 GB. The same model at 4-bit quantization is roughly 4.3 GB. Nothing about your prompt changes that number, which is why it is the safest figure on the page.

The K-quant rows are averages, not formulas: a GGUF block stores a scale next to its low-bit weights, so a file labelled Q4_K_M is a little larger than exactly four bits per parameter, and the ratio drifts between checkpoints. Hence the approx. on every such row.

Every layer stores one key vector and one value vector per KV head for every token in the conversation, in both the prompt and the generated answer. The formula is therefore exact arithmetic on the architecture: 2 × layers × KV heads × head dimension × context × batch × bytes per element. The leading 2 is the key and the value; the KV-head count is what makes grouped-query attention matter, because several query heads can share a single key/value head.

Worked example — Qwen2.5 7B (28 layers, 4 KV heads, 128-wide heads) at 8,192 tokens, batch 1, FP16 cache: 2 × 28 × 4 × 128 × 8192 × 1 × 2 = 469,762,048 bytes, about 0.47 GB. Push the context to 32,768 and the same cache is 1.88 GB; run four sequences at once and it is 7.5 GB. On a 24 GB card that difference decides whether the model runs at all.

This is also why grouped-query attention replaced multi-head attention so quickly. A model with 32 KV heads instead of 8 would need four times the cache at the same context, and four times the memory bandwidth to read it — the trade-off stage 11 of the course demonstrates with a live simulation.

A sparse mixture-of-experts model keeps every expert in memory but routes each token through only a few of them. Qwen1.5-MoE, for example, has 14.3 billion parameters in total and touches about 2.7 billion of them per token. That splits the two questions this page answers: memory follows the total, because all 60 experts must be loadable, while speed follows the active count, because only the chosen experts are read from memory during a decode step.

Practically, a 14B MoE that needs about 28.6 GB of weights at BF16 can generate tokens at roughly the speed of a 2.7B dense model on the same card — but it still will not fit on a 24 GB GPU.

• Throughput beyond the bandwidth ceiling. The tokens-per-second figure is bandwidth ÷ bytes read per token: a batch-1 upper bound. Real engines lose 20–50% of it to attention over the cache, kernel launches and sampling.
• Tensor-parallel inefficiency. Splitting a model across GPUs duplicates some buffers, so the multi-GPU estimate is optimistic by a few percent.
• Fragmentation. A server that has answered thousands of requests of varying length ends up with a fragmented cache, so production deployments reserve more than the arithmetic asks for.