Her token diğerlerine bakar ve neyin önemli olduğuna karar verir.
⚠ Bu aşamanın 3% kadarı Türkçe diline çevrildi; kalan kartlar İngilizce gösteriliyor. Bir hata mı gördün? Çeviri düzeltmeleri memnuniyetle karşılanır.
Okuma ilerlemesi0%
Card 1 — the problemKart 1 / 5
Words need context, and context is far away
By the time a model reaches the end of a sentence, the beginning is a distant memory.
Consider "The animal did not cross the street because it was too tired." What does "it" refer to? The animal or the street? To resolve this, the model must connect a token to another token many positions earlier. Recurrent networks passed a hidden state hand to hand and slowly forgot; self-attention lets every position consult every other position directly, in one step.
Analogy
A committee meeting, not a game of telephone
An RNN is a whispered chain: each person repeats what they heard. Attention is a meeting where every person can hear every other person simultaneously, and privately decides whose opinion matters most for the question they are answering.
NoteThe cost
Directly connecting everything costs O(n²) in sequence length: 1,000 tokens means a million comparisons per layer. That quadratic term is the reason context windows are expensive and why long-context research (sparse and linear attention) is such a big deal.
?
Sonraki kart için tek bir hızlı kontrol
What advantage does self-attention have over a recurrent network?
5 bilgi kontrolü
🔒
Oyun alanı kilitli
Simülasyonu açmak için 5 kartın tamamını ve bilgi kontrollerini bitir.
Read the full stage transcript (all 5 cards, no JavaScript required)
Card 1 — the problem
1. Words need context, and context is far away
By the time a model reaches the end of a sentence, the beginning is a distant memory.
Consider "The animal did not cross the street because it was too tired." What does "it" refer to? The animal or the street? To resolve this, the model must connect a token to another token many positions earlier. Recurrent networks passed a hidden state hand to hand and slowly forgot; self-attention lets every position consult every other position directly, in one step.
Analogy
A committee meeting, not a game of telephone
An RNN is a whispered chain: each person repeats what they heard. Attention is a meeting where every person can hear every other person simultaneously, and privately decides whose opinion matters most for the question they are answering.
NoteThe cost
Directly connecting everything costs O(n²) in sequence length: 1,000 tokens means a million comparisons per layer. That quadratic term is the reason context windows are expensive and why long-context research (sparse and linear attention) is such a big deal.
Knowledge check: What advantage does self-attention have over a recurrent network?
a. Any two positions can interact directly, without information degrading across steps← correct
b. It uses less memory
c. It needs no training data
Attention gives constant-length paths between any two tokens, which is why long-range dependencies survive and why the whole sequence can be processed in parallel.
Card 2 — Q, K, V
2. Query, Key, Value: a search engine inside the model
Ask a question, match it against every available label, then blend the contents.
Each token produces three vectors by multiplying its embedding with three learned matrices: a Query ("what am I looking for?"), a Key ("what do I offer?"), and a Value ("what will I actually hand over if chosen?").
Analogy
Library search
Your Query is the search phrase you type. Every book has a Key: its title and tags. Matching your phrase against every book gives a relevance score. You then take a weighted blend of the books' contents — their Values. Attention is exactly this, performed for every token at once.
Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k ) · V
Scores between every pair of tokens → normalised weights → weighted sum of values.
Q·Kᵀ produces an n×n matrix: how much every token cares about every other token.
softmax converts each row into weights that sum to 1 — a distribution of attention.
Multiplying by V blends the values using those weights: the new, context-aware representation.
The result is a per-token embedding that now contains information from wherever it looked.
Knowledge check: In attention, what role does the Value vector play?
a. It scores relevance between tokens
b. It provides the content that gets blended after the weights are computed← correct
c. It is the query rewritten
Keys determine who is listened to; Values determine what is actually passed along. Queries are the question being asked.
Card 3 — the fixes
3. Why divide by √d, and why softmax bites back
Two small details in one formula make or break training.
Dot products between high-dimensional vectors grow with dimension. With d = 512, raw scores can reach into the hundreds. Feed large numbers into softmax and it becomes a hard argmax: one weight becomes 1.0, the rest 0.0, and the gradient through that softmax is effectively zero. Dividing by √d keeps the scores in a friendly range where softmax stays soft and gradients flow.
softmax(zᵢ) = e^{zᵢ} / Σⱼ e^{zⱼ}
Subtracting the row maximum before exponentiating (a standard trick) prevents overflow.
The second detail is masking. A decoder-only LLM must never see the future, otherwise predicting the next token is cheating. So before the softmax, every position above the diagonal is set to −∞, which becomes exactly 0 after softmax. This is the causal mask, and it is the only thing that turns a bidirectional reader into an autoregressive generator.
Setup
Attention mask
Used by
Encoder (BERT-style)
Bidirectional — everyone sees everyone
Classification, embeddings
Decoder (GPT-style)
Causal — token i sees only ≤ i
Text generation, chat models
Padding mask
Ignores filler tokens in short sequences
Both, in batched training
Watch outDebugging tip
If your loss is flat while training a decoder, check the mask first. A missing causal mask gives suspiciously low training loss and a model that cannot generate coherent text.
Knowledge check: What does the causal mask prevent?
a. A token attending to tokens that come after it← correct
b. Attention from running on a GPU
c. The softmax from overflowing
Masking future positions with −∞ zeroes their weights, so the model can only use left context — which is what makes next-token prediction an honest training signal.
Card 4 — many perspectives
4. Multi-head attention: run the meeting in parallel rooms
One attention pattern cannot capture syntax, coreference and topic at once.
Instead of one attention computation, the model runs several heads in parallel, each with its own Q, K and V projection matrices of reduced size. Their outputs are concatenated and mixed by an output projection. The total parameter count stays roughly constant — you are not adding cost, you are splitting the perspective.
Analogy
Parallel rooms
One committee trying to track grammar, pronouns and topic simultaneously drops balls. Split into four rooms — the syntax room, the coreference room, the topic room, the position room — and let each report back. That is multi-head attention.
Different heads empirically specialise: some track adjacent tokens, some track verbs to their objects, some track rare-token identity.
Head count and head dimension trade off: 12 heads × 64 dims = 768 model width.
Multi-Query / Grouped-Query Attention shares one K and V across heads, shrinking the KV-cache dramatically — the trick that makes long-context inference affordable (Stage 11).
NoteA research oddity worth knowing
Ablation studies find many heads can be pruned after training with little loss. Redundancy is a feature of training, not necessarily of the final artefact — which is why distilled and pruned models can be much smaller.
Knowledge check: What is the main benefit of multi-head attention?
a. Each head can learn a different relationship pattern in parallel← correct
b. It reduces the number of parameters sharply
c. It removes the need for positional encoding
Heads provide multiple independent subspaces for attention, at roughly the same total parameter budget as a single wide head.
Card 5 — the whole block
5. A transformer block is attention plus a small brain
Attention gathers context. The feed-forward network thinks about it.
1Input embeddings are combined with positional information, since attention itself is order-agnostic.
2Multi-head self-attention lets every token gather context from everywhere it may look.
3Residual connection adds the original vector back, so gradients survive deep stacks.
4Layer normalisation re-centres and re-scales activations for stable training.
5Feed-forward network (two linear layers with a non-linearity, typically 4× wider) processes each token position independently — this is where most parameters and most factual recall live.
6Stack this block 12 to 100+ times and you have a transformer.
py
def block(x, mask):
# 1. attention with a residual highway
x = x + attention(norm(x), mask=mask)
# 2. position-wise feed-forward, again with a highway
x = x + mlp(norm(x))
return x
# pre-norm ordering (above) trains more stably than post-norm
# at depth; most modern LLMs are pre-norm.
Analogy
Meeting, then memo
Attention is the meeting where everyone speaks and listens. The feed-forward network is the memo each participant writes afterwards, alone, turning what they heard into something useful. A transformer alternates meeting → memo, again and again.
NoteNext up
You now understand the architecture. Stage 8 asks how it is trained: the loss function, the scaling laws that decide how big to build it, and the dials that keep a multi-billion-parameter run alive.
Knowledge check: What does the feed-forward network in a transformer block do?
a. Processes each position independently after attention has mixed in context← correct
b. Replaces attention entirely in deep layers
c. Normalises the attention weights
Attention mixes information across positions; the MLP transforms each position. Both are wrapped in residual connections and layer norm.