11 / 11约 17 分钟阅读难度: Advanced5 张卡片

推理与 KV 缓存

请求发出后发生什么：prefill、decode、采样与显存。

⚠ 本阶段已翻译为中文 2%，其余卡片暂以英文显示。发现有误？欢迎提交翻译修正。

阅读进度0%

Card 1 — two workloads第 1 / 5 张卡片

Prefill and decode are completely different jobs

One is a giant parallel matrix multiply. The other is a tiny serial loop.

Prefill processes your entire prompt at once. Every token attends to every earlier token, so the GPU runs enormous parallel matrix multiplies and is genuinely compute-bound. That phase produces time to first token (TTFT). Then decode generates one token at a time, each depending on the last, so each step is a tiny amount of arithmetic with a large amount of memory traffic — decode is memory-bandwidth-bound. Tokens-per-second is therefore limited by how fast you can read the weights, not by how fast you can multiply.

Analogy

Reading a book vs writing a sentence

Prefill is skimming an entire chapter: fast, parallel, limited by input speed. Decode is writing a sentence word by word, where each word depends on the one before. However quick your hand is, you cannot write the next word before deciding this one.

Phase	Workload	Scales with	Bottleneck
Prefill	All prompt tokens in parallel	Prompt length	Compute (FLOPs)
Decode	One token per step	Output length	Memory bandwidth

NoteWhy batching is the whole game

Because decode is bandwidth-bound, generating tokens for 32 users at once costs barely more wall-clock time than for one — the weights are read once and reused 32 times. That is continuous batching, and it is the single biggest lever on serving economics.

进入下一张卡片前的一个小检查

Why is the decode phase usually memory-bandwidth-bound rather than compute-bound?

5 个知识检查

🔒

实验区已锁定

完成全部 5 张卡片及其知识检查，即可解锁这个模拟实验。

Read the full stage transcript (all 5 cards, no JavaScript required)

Card 1 — two workloads

1. Prefill and decode are completely different jobs

One is a giant parallel matrix multiply. The other is a tiny serial loop.

Analogy

Reading a book vs writing a sentence

Phase	Workload	Scales with	Bottleneck
Prefill	All prompt tokens in parallel	Prompt length	Compute (FLOPs)
Decode	One token per step	Output length	Memory bandwidth

NoteWhy batching is the whole game

Knowledge check: Why is the decode phase usually memory-bandwidth-bound rather than compute-bound?

a. Each step produces one token, so the arithmetic per step is tiny while the whole model still has to be read from memory← correct
b. GPUs cannot do matrix multiplication during decode
c. Because the prompt has already been processed

Arithmetic intensity is low in decode: few FLOPs per byte of weights loaded. Batching amortises that memory traffic across many requests.

Card 2 — the cache

2. The KV-cache: memory that buys speed

Without it, every new token would re-read the entire conversation.

Attention needs the Key and Value vectors of every previous token. Recomputing them for the whole sequence on each new token would be quadratic work for nothing — they never change. The KV-cache stores them, so decode only computes keys and values for the new token and attends against the cache.

KV bytes ≈ 2 · layers · heads · head_dim · tokens · bytes_per_value

The factor of 2 is for K and V. Everything scales linearly with context length and batch size.

# 7B model: 32 layers, 32 heads, head_dim 128, fp16
per_token = 2 * 32 * 32 * 128 * 2      # 524,288 bytes  (~0.5 MB)
kv = per_token * 4096                   # ~2.1 GB  for one 4k conversation

kv * 32                                 # ~68 GB  of KV cache for 32 users

The cache grows with context length × concurrent sequences, not with output length alone.
Grouped-Query Attention shares K/V across query heads, cutting the cache 4–8× with negligible quality loss.
PagedAttention (vLLM) stores the cache in fixed-size pages like an OS page table, removing most of the fragmentation waste of naive contiguous allocation.
Quantising the cache to int8 or fp8 halves or quarters it again, often with almost no quality change.

Watch outThe practical lesson

Long context is not free. Doubling the context window doubles the KV-cache per user, which either halves your concurrency or forces quantisation and paging tricks. This is why "1M token context" and "cheap per-token pricing" are in tension.

Knowledge check: Roughly what does the KV-cache size scale with?

a. Context length and the number of concurrent sequences← correct
b. Only the number of generated tokens
c. The size of the vocabulary

It is per-layer K and V for every token in every live sequence. Long context combined with high concurrency exhausts VRAM first.

Card 3 — the sampler

3. Temperature, top-k and top-p: shaping the dice

The model outputs probabilities. Sampling decides which token you actually get.

A language model does not output a token; it outputs a probability for every token in the vocabulary. Sampling is the policy that picks one. Change the sampler and the same model becomes terse or unhinged, precise or poetic — with no weights changed at all.

Parameter	What it does	Low value	High value
Temperature	Divides logits; rescales the distribution	0 = deterministic argmax, repetitive	>1.5 = incoherent, off-topic
Top-k	Keeps only the k most likely tokens	Very focused, can loop	Turns into plain sampling
Top-p (nucleus)	Keeps the smallest set whose probabilities sum to p	Very safe output	More variety, more drift
Repetition penalty	Discounts tokens already used	Loops and stutter	Strange grammar and avoidance

pᵢ = softmax(zᵢ / T) then keep top-k and/or top-p, renormalise, sample

T below 1 sharpens; T above 1 flattens. Temperature 0 means take the argmax outright.

Analogy

Radio static

Temperature is the static on the dial. At zero the station is locked but plays the same song forever. Just a little static and it sounds alive. Too much and you cannot tell what you are listening to.

Good to knowPractical starting points

Extraction and code: temperature 0–0.2, top-p 1.0. Chat: 0.7 with top-p 0.9. Creative writing: 0.9–1.0 with top-p 0.95. Never raise temperature and top-p aggressively at the same time — the effects compound.

Knowledge check: Your JSON extraction endpoint returns malformed output roughly 3% of the time. What is the most likely sampler cause?

a. Temperature is too high, letting unlikely tokens get sampled← correct
b. Top-p is set too low
c. The context window is too small

Deterministic tasks want temperature 0 with a constrained or schema-validated output format, not creative sampling.

Card 4 — the server

4. Serving economics: batch, page, quantise

Throughput and latency pull in opposite directions. Know which one you are buying.

Continuous batching mixes requests from different users in every decode step, keeping the GPU fed and multiplying throughput.
Prefill/decode disaggregation runs the two phases on separate hardware pools, because they bottleneck differently.
Quantisation (int8, int4, fp8) cuts weight memory 2–4× and speeds decode, which is bandwidth-bound. Quality loss at int8 is usually negligible; int4 needs calibration.
Speculative decoding uses a small draft model to propose several tokens and the big model to verify them in one pass — same output distribution, faster wall-clock.
Prefix caching reuses the KV-cache of an identical system prompt or shared document across requests.

Metric	What the user feels	Improved by
TTFT	Delay before the first word appears	Faster prefill, prefix caching, smaller prompt
Tokens/sec	How fast text streams in	Higher batch, quantisation, GQA, speculative decoding
Throughput	Cost per request	Continuous batching, PagedAttention

Watch outThe classic capacity bug

Sizing a server by weights alone ("7B in fp16 is 14 GB, so a 24 GB card is plenty") ignores the KV-cache. With 32 concurrent 8k-token conversations the cache can exceed the weights. Budget for both, then test at peak concurrency.

Knowledge check: Why can quantising the base weights make generation noticeably faster?

a. Decode is memory-bandwidth-bound, so reading 4-bit weights moves less data per token← correct
b. Quantised models skip attention
c. Because quantisation shortens the context window

Fewer bytes to move means more tokens per second when you are bandwidth-limited — which most single-request decode loops are.

Card 5 — the full lifecycle

5. You just walked the entire LLM lifecycle

Eleven stages, one pipeline, no hand-waving.

1Stages 1–3 (Foundations): a line, a neuron, and the credit-assignment algorithm that trains both.
2Stages 4–6 (Data & Representation): deduplicate the corpus, tokenize it, embed it into a semantic space.
3Stages 7–8 (Architecture & Pre-training): attention mixes context; next-token prediction on trillions of tokens builds the base model.
4Stages 9–10 (Adaptation & Alignment): instruction tuning (with LoRA) makes it useful; preference optimisation makes it well-behaved.
5Stage 11 (Inference): prefill, decode, KV-cache, sampling and VRAM decide what the user actually experiences.

Good to knowWhat to do with this

When you read about a new model or technique, place it on this map. A "new attention variant" is Stage 7. A "better dataset filter" is Stage 4. A "cheaper inference stack" is Stage 11. The map is the real takeaway.

NoteFinish the playground

Run the VRAM sandbox above, then return to the certificate page. With all 11 stages complete — readers, checks and playgrounds — you can generate your verifiable certificate and share it on LinkedIn.

Knowledge check: Which stage of the lifecycle does a technique that reduces KV-cache memory belong to?

a. Inference — it changes serving memory and speed, not what the model learned← correct
b. Pre-training — it changes the loss function
c. Data curation — it filters documents

Grouped-Query Attention and PagedAttention are serving-time and architecture choices that never touch the training objective. Knowing where a change lives is most of the battle.