التدريب المسبق والخسارة

Pre-training a large language model optimises exactly one thing: predict the next token. That is it. Feed it a trillion tokens of raw text, mask the future, and ask for the next one every time. Grammar, facts, translation, arithmetic procedures and reasoning patterns all emerge as side effects of getting very good at this single task.

The training signal is self-supervised: the label is simply the next token from the real text, so no human annotation is needed. That is what makes internet-scale training possible at all.

Cross-entropy is minimised when the model puts probability 1.0 on the correct token. Because it uses the logarithm, being confidently wrong is punished brutally: assigning 0.01 to the right answer costs 4.6, while assigning 0.9 costs only 0.105. This asymmetry is exactly the pressure we want — it forces the model to spread probability sensibly instead of guessing.

Pre-training rarely uses a constant learning rate. The standard recipe is warmup then decay: start near zero, ramp up over the first few hundred to few thousand steps, then decay smoothly (cosine) toward the end. Warmup exists because early gradients are huge and unreliable — taking full-size steps immediately destabilises the run.

AdamW maintains a running mean and variance of each parameter's gradient and divides by the square root of the variance, giving every parameter its own effective step size. It is memory-hungry — two extra tensors per parameter — which is why optimiser state is a large part of what you must fit in GPU memory when fine-tuning.

Loss falls predictably as a power law in model parameters, dataset size and compute. That predictability is remarkable: results from small runs let you forecast a large one before spending the money. The practical consequence came from the Chinchilla result — for a fixed compute budget, most models of that era were far too large for the amount of data they saw.

• Compute-optimal means best loss per FLOP. Inference-optimal means best loss per served request — which is why production models are often trained far beyond the Chinchilla point.
• Data quality changes the exponent: cleaner corpora shift the whole curve down.
• Diminishing returns are real but slow — you keep paying linearly for logarithmically better loss.
• Small-scale experiments are genuinely predictive, which is why ablation studies on 100M-parameter models remain standard practice.

1. 1Collect and clean the corpus (dedup, filter, tokenize, shard into fixed-length sequences).
2. 2Shuffle and stream the shards so no epoch ordering leaks into the model.
3. 3Forward pass on a batch, compute cross-entropy on the next token at every position.
4. 4Backward pass, clip gradients, step AdamW on the schedule.
5. 5Log loss, gradient norm, learning rate and throughput. Watch for spikes.
6. 6Checkpoint regularly — a crash at week three without one is a catastrophe.

for step, batch in enumerate(loader):
    lr = schedule(step)                 # warmup -> cosine decay
    for g in opt.param_groups:
        g['lr'] = lr

    out = model(**batch)                # labels shifted inside the loss
    out.loss.backward()

    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()
    opt.zero_grad(set_to_none=True)

    if step % 50 == 0:
        log(loss=out.loss.item(), lr=lr, step=step)