The intuition: every token asks a question
Read “The animal did not cross the street because it was too tired.” What does “it” refer to? You resolve it by looking back at the other words and weighting the evidence. Self-attention is a learned, differentiable version of that lookup.
Each token emits three vectors. The query encodes what this token is looking for. The key encodes what a token offers as a match. The value carries the content passed along if the match is good. A token compares its query with every key, turns the similarities into weights, and mixes the values accordingly.
“Self” means queries, keys and values all come from the same sequence. Cross-attention, used in encoder–decoder models, takes queries from one sequence and keys/values from another.
The scaled dot-product formula
Stack the token vectors into matrices Q, K and V. The attention output is:
The product QKᵀ holds raw relevance scores: one row per token, one column per token it can see. Softmax is applied per row, so each token’s weights sum to one. Multiplying by V then produces a weighted average of value vectors — the new representation for that position.
| Matrix | Shape | Meaning |
|---|---|---|
| Q | sequence × dₖ | What each token is looking for |
| Kᵀ | dₖ × sequence | What each token offers |
| QKᵀ | sequence × sequence | Raw pairwise relevance scores |
| softmax(…)V | sequence × dᵥ | Mixed context per token |
Causal masks and multiple heads
A language model that generates left to right must not see future tokens during training. A causal mask sets the scores above the diagonal to negative infinity before the softmax, so position 4 can attend to positions 1–4 but never to 5. Encoder models such as BERT omit that mask and attend in both directions.
One attention pattern is limiting, so transformers run several heads in parallel with separate projections. Each head can specialise — one may track nearby syntax, another may link pronouns to nouns — and their outputs are concatenated and projected back to the model width. The heads are not assigned roles by hand; the specialisation emerges from training.
A toy row of attention weights might be [0.10, 0.70, 0.20]: the current token puts 70% of its read on one earlier token. The demonstration below recomputes a full matrix as you toggle the causal mask and change the softmax temperature.
What it costs, and what it does not do
The score matrix has one entry per pair of tokens, so attention scales quadratically with sequence length. Doubling the context window roughly quadruples that part of the compute, and the KV cache grows linearly with the number of tokens. This is the practical reason long-context inference is expensive rather than free.
Attention weights are not a complete explanation. A high weight means a value vector was mixed in heavily; it does not prove that the model’s decision was caused by that token, because later layers, residual streams and MLPs all transform the representation again. Treat attention maps as a useful diagnostic, not a receipt.
Self-attention also has no memory between requests. Everything a model “remembers” about your conversation is text placed back into the prompt, which is why context limits and prompt structure matter so much. For how those vectors are first produced, see “What is an embedding in AI?”