KV Cache Size Formula: Why Long Context Eats Your VRAM

You loaded a model that fits comfortably in your GPU, sent a long prompt, and ran out of memory anyway. Or it worked for one user and crashed at five. The missing piece is the KV cache: memory that grows with every token of context and with every conversation you serve at the same time. Weights are a fixed cost. The KV cache is the variable cost, and it is the reason long-context and high-concurrency serving needs far more VRAM than the model size suggests.

This guide gives you the formula, worked numbers for Llama 3 8B and 70B, a way to plan how many users fit on a card, and the practical levers that shrink the cache.

What the KV cache stores

During generation, each attention layer needs the keys and values of every earlier token to decide what the next token should be. Recomputing them for each new token would be far too slow, so the model stores them in memory and appends one new entry per token. This is the core idea of , and explains why keys and values exist in the first place.

Every token you add, whether from the prompt or from the answer, adds a fixed amount to the cache. That is why memory use climbs steadily while the model is generating.

The formula

The size of the cache is:

KV cache bytes = 2 x layers x KV heads x head dimension x tokens x bytes per value x batch

The 2 is for keys plus values. "KV heads" is the number of key and value heads, which can be smaller than the number of query heads in models that use grouped-query attention. "Bytes per value" is 2 for 16-bit storage. "Batch" is the number of sequences held in memory at once.

Worked numbers

Llama 3 8B has 32 layers, 8 KV heads and a head dimension of 128. With 16-bit values:

2 x 32 x 8 x 128 x 2 = 131,072 bytes, which is 128 KiB per token.

Llama 3 70B has 80 layers, 8 KV heads and a head dimension of 128:

2 x 80 x 8 x 128 x 2 = 327,680 bytes, which is 320 KiB per token.

Multiplying by context length gives the per-sequence cache:

These figures are per sequence. Ten concurrent users at 32k context on the 8B model need about 40 GiB of cache on top of the weights. A single 128k-token conversation on the 8B model needs as much cache as the weights themselves.

Plan how many users fit

Here is a simple capacity formula:

concurrent sequences = (VRAM - weights - runtime overhead) / KV cache per sequence

Take a 24 GB card running Llama 3 8B in 16-bit. The weights take roughly 16 GB. Reserve a couple of gigabytes for the runtime, and about 6 GB is left for cache. At 8k context that is about one gigabyte per sequence, so only around six conversations fit.

Now quantize the weights to 4-bit. They drop to roughly 5 GB, leaving around 17 GB for cache, which is about 17 sequences at 8k. Same card, nearly three times the concurrency. This is why the change so much with context length, and why is a capacity topic as much as a quality one. All of these numbers are approximate, so treat them as planning estimates and leave headroom.

Why long context also slows you down

Memory size is only part of the problem. At every generation step the GPU has to read the cache for that sequence, so long contexts mean more data moved per token. Decoding speed is often limited by memory bandwidth rather than raw compute, which means a very long conversation generates tokens more slowly than a short one, even on the same hardware. When you choose a GPU, bandwidth matters as much as capacity. compares options by both.

How to shrink the cache

• Grouped-query attention. Most modern models already use it. Llama 3 8B has 32 query heads but only 8 KV heads. Without that design, the same model would need 32 KV heads, and the cache would be four times larger: 512 KiB per token instead of 128.
• KV cache quantization. Storing keys and values in 8-bit roughly halves the cache, and lower precision cuts more. Test quality on your own task, because long-context accuracy can be sensitive.
• Shorter context. Trim old turns, summarize history, or retrieve only the relevant chunks instead of pasting whole documents. See .
• Paged allocation and prefix sharing. Serving engines such as vLLM manage cache memory in blocks, so they waste less on unused space and can share a common prefix between requests. Compare engines in .
• Sliding-window or compressed attention. Some architectures limit how far back attention looks, or compress the cache into a smaller representation. Always read the model's actual configuration instead of assuming the formula above applies unchanged.

Cache memory and API prices

If you use a hosted API, you never see this memory, but you pay for it indirectly. Providers price long contexts higher, and prompt caching exists because rebuilding the cache is expensive. explains how that shows up on your bill.

Try it with your own model

Open the , pick a model and a graphics card, and change the context length. You will see the weights stay constant while the KV cache grows, along with the runtime overhead most guides leave out. Every formula is printed on the page, so you can check the arithmetic yourself. For smaller cards, walks through what fits where.

Common questions

Is the KV cache the same as the context window? No. The context window is the maximum length a model accepts. The KV cache is the memory used to hold the tokens you actually sent.

Does the cache stay allocated between requests? In most serving engines it is released when a conversation ends, unless prefix caching keeps it for reuse.

Why did my model run out of memory halfway through an answer? Because every generated token adds to the cache, and the limit can be crossed during generation, not only while reading the prompt.

Checklist

• Read layers, KV heads and head dimension from the model configuration
• Multiply by your real context length, precision and concurrent users
• Add weights plus runtime overhead
• Decide whether weight quantization, KV quantization or a shorter context is acceptable
• Leave headroom, because memory use fluctuates under real traffic