تقطيع BPE

A transformer is a stack of matrix multiplications. It has no idea what a letter is. So every string must first be chopped into tokens, and each token mapped to an integer id from a fixed vocabulary. Feed in "Hello world" and the model actually receives something like [15496, 995].

The full pipeline is: text → tokens → token ids → embeddings → transformer → token ids → text. Notice that the tokenizer sits at both ends: the model's raw output is a probability distribution over token ids, and the detokenizer stitches those ids back into a string.

Whole-word vocabularies break the moment the model meets "unfriendliest" or a new product name, because the id simply does not exist. Character-level vocabularies never break but turn every sentence into a marathon of steps — and attention cost grows with the square of sequence length.

Byte-Pair Encoding finds that middle ground automatically. Start from raw bytes, then repeatedly glue together whichever pair of symbols occurs most often. Frequent strings like "the " become single tokens; rare ones stay spelled out.

1. 1Start with every word split into characters, plus an end-of-word marker (commonly the leading space or a special </w>).
2. 2Count how often each adjacent pair of symbols occurs across the whole corpus.
3. 3Merge the single most frequent pair into one new symbol and record the merge rule.
4. 4Repeat until you hit the target vocabulary size (e.g. 50,000 merges).

corpus = {'l o w </w>': 5, 'l o w e r </w>': 2, 'n e w e s t </w>': 6}

# pass 1: the most frequent pair is ('e', 's') with count 6
# merge it -> 'e s' becomes the new symbol 'es'
corpus = {'l o w </w>': 5, 'l o w e r </w>': 2, 'n e w es t </w>': 6}

# pass 2: now ('es', 't') occurs 6 times -> merge into 'est'
# pass 3: ('l', 'o') occurs 7 times -> merge into 'lo'
# ... continue until vocab_size merges are learned

Merges are applied greedily in the order they were learned. At encode time the tokenizer replays the recorded merge list on the new text — no counting, no statistics, just a deterministic list of rules applied in sequence.

The embedding matrix and the final output projection both scale linearly with vocabulary size: vocab × hidden_dim, twice. For a 128k vocabulary and 4096-width hidden state that is over a billion parameters spent purely on lookup tables.

• Sequence length costs quadratically in attention, so merging frequent strings pays off fast.
• Rare tokens get weak gradients — a token seen 50 times in pretraining never learns a good vector.
• Multilingual fairness: languages poorly covered by the vocabulary produce 3–5× more tokens per sentence, meaning higher cost, slower output and shorter effective context.
• Special tokens (<|endoftext|>, <|im_start|>, chat roles) are reserved ids inserted deliberately, never produced by merges.

• Counting characters is impossible. A model cannot reliably say how many letters are in a word, because it never sees letters — it sees whole chunks. "strawberry" may be two tokens, so "count the r's" is a question about pieces it cannot decompose.
• Numbers tokenize irregularly. "1000" may be a single token while "1001" splits into two, which is why arithmetic is unstable and why chunking digits before feeding a calculator helps.
• Leading spaces matter. "hello" and " hello" are different tokens. Stray whitespace changes the id sequence and can shift output quality.
• Glitch tokens exist. Some rare merge artefacts (famously SolidGoldMagikarp in early GPT-2-era tokenizers) received almost no training signal and produce bizarre model behaviour.
• Prompt length is counted in tokens. A "200-character" prompt can be 30 tokens or 200 depending on language and symbols. Always measure, then budget.