LLM Tokens per Second: Why Memory Bandwidth Beats TFLOPS

Local LLM speed confuses almost everyone the first time. A GPU with huge compute can still generate text slower than you expect, and a Mac with a modest GPU can beat a desktop card on large models. The reason is simple: generating each new token is limited by memory bandwidth, not by raw compute.

This guide gives you one formula to estimate the speed ceiling of any model on any hardware before you download or buy anything.

TL;DR

• Decode speed ceiling: tokens per second ≈ memory bandwidth (GB/s) ÷ bytes read per token (GB)
• For a dense model at batch size 1, bytes read per token ≈ all the model weights, plus the KV cache.
• Quantization makes generation faster mainly because it shrinks the bytes you must read.
• The formula gives a ceiling, not a guarantee. Real speed is lower.

Prefill vs decode: two different bottlenecks

Every request has two phases, which you can explore in .

Prefill processes your whole prompt in parallel. It is a large matrix-matrix multiplication with lots of math per byte loaded, so it is compute-bound. This is where TFLOPS matter, and it decides how long you wait for the first token.

Decode generates one token at a time. Each step needs a matrix-vector multiplication that touches every weight once, but does very little math with each weight. The GPU spends most of its time waiting for memory, so decode is memory-bound. The ratio of math to bytes moved is called arithmetic intensity, and decode has very low arithmetic intensity.

That is why a chat reply streams at a speed set by your memory, not by the headline compute number on the spec sheet.

The formula

Every decode step must stream the model weights from memory to the compute units. So:

tokens per second ≈ memory bandwidth ÷ bytes read per token

For a dense model at batch size 1, bytes read per token is roughly the size of the weights: parameters × bytes per parameter. A 4-bit quantized 8B model is about 4.9 GB, so a device with 1,000 GB/s of bandwidth has a ceiling near 200 tokens per second.

Worked example: Llama 3 8B

Ceilings from spec-sheet bandwidth, assuming the model fits in memory:

Two lessons jump out. Going from FP16 to 4-bit roughly triples the ceiling, because you read about a third of the bytes. And bandwidth, not brand, sorts the table.

Add the KV cache

Besides weights, each decode step also reads the KV cache for every token already in the context. For Llama 3 8B in FP16 that is 2 (K and V) × 32 layers × 8 KV heads × 128 head dimension × 2 bytes = 128 KiB per token. The full derivation is in our .

On an RTX 4090 with the 4-bit model:

Long context slows generation because every token must also read a growing cache.

Bigger models: 70B

A 70B model at 4-bit is about 40 GB. It does not fit on a single 24 GB card, so you need two GPUs, a high-memory workstation, or a Mac with unified memory.

• Apple M2 Ultra (~800 GB/s): ceiling near 20 tok/s.
• A single H100 SXM (~3,350 GB/s, 80 GB): ceiling near 84 tok/s for one stream.

Datacenter GPUs cost so much largely because of bandwidth, not only compute.

Batching changes the picture

When you serve many users, one step reads the weights once and applies them to every sequence in the batch. Total throughput scales almost linearly with batch size until compute or KV cache memory becomes the limit, while each user's speed stays roughly the same. This is why serving engines like those compared in focus so heavily on batching and cache efficiency.

Mixture-of-experts models

A mixture-of-experts model such as Mixtral 8x7B holds about 47B parameters but activates roughly 13B per token. At small batch sizes you only read the active experts, so speed looks like a ~13B dense model, while memory must still hold all ~47B. In short: memory follows total parameters, and single-stream speed follows active parameters.

How to make generation faster

1. Quantize. INT8 or 4-bit weights shrink bytes per token. See for the quality trade-off.
2. Choose hardware by bandwidth. Compare GB/s, not just TFLOPS or VRAM size.
3. Keep context short. Every extra token in the cache adds bytes to every step.
4. Batch requests when serving many users.
5. Use speculative decoding. A small draft model proposes several tokens and the large model verifies them in one pass. Verifying several tokens costs about one weight read, and with the standard acceptance rule the output distribution matches the large model.

Common mistakes

• Comparing TFLOPS to predict chat speed.
• Forgetting the KV cache at long context.
• Assuming a mixture-of-experts model needs only its active parameters in memory.
• Offloading some layers to system RAM. The slow DDR bandwidth then dominates and speed drops sharply.
• Treating the ceiling as a promise. Kernel overhead, sampling and other costs keep real speeds below it.

Estimate your own speed in three steps

1. Get the model size in GB: use the quantized file size, or use the for weights plus KV cache at your context length.
2. Find your device's memory bandwidth on its spec sheet.
3. Divide bandwidth by size. As a rule of thumb, expect real speed noticeably below the result, often somewhere around 60 to 80 percent on well-optimized setups. Measure on your own machine to be sure.

FAQ

Does a faster GPU always mean more tokens per second? Only if its memory bandwidth is higher. A card with more TFLOPS but similar bandwidth will generate at a similar speed.

How many words is a token per second? In English, one token is roughly three quarters of a word. Paste your text into the to see exact counts.

Why do Macs run large models well? Unified memory gives them large capacity at high bandwidth, so big quantized models fit and stream quickly, even if raw compute is lower.

Does a long prompt slow down generation? It lengthens prefill, which delays the first token, and it makes each decode step read a larger KV cache.

Related reading