What a loss function does
A model outputs a prediction. A loss function compares that prediction with the target and returns a scalar. Training computes the gradient of this scalar with respect to every weight and takes a step downhill.
Because the loss is the only signal optimised, it is the most consequential design choice after the data itself. A model trained with the wrong loss learns the wrong thing efficiently. Score a spam filter only on accuracy and, when spam is rare, it learns to answer “not spam” for almost everything.
A loss must be differentiable for gradient training. A metric is whatever humans care about — accuracy, F1, latency, factuality — and is often not differentiable. When they diverge, the model optimises the loss and the gap appears in evaluation.
The two you will meet first
For regression the default is mean squared error:
For classification the default is cross-entropy. The model outputs a probability p for the correct class, and the loss is −log(p). Predicting 0.9 for the right class costs about 0.105; predicting 0.1 costs about 2.303. Confident mistakes are expensive, which is the pressure you want.
| Loss | Typical task | What it punishes most |
|---|---|---|
| Mean squared error | Regression | Large absolute errors |
| Mean absolute error | Regression with outliers | All errors equally |
| Cross-entropy | Classification, next-token prediction | Confident wrong probabilities |
| Contrastive loss | Embeddings and retrieval | Relevant pairs ranked below irrelevant ones |
Language models use cross-entropy over the next token at every position. The “loss curve” in a training log is the average of that per-token loss, which is why it starts high and falls as the model learns.
A worked example
Predict a house price of 240,000 when the true price is 300,000. The error is −60,000, so squared error is 3.6 × 10⁹ — a huge number, which is why implementations often scale targets or report root-mean-squared error in the original units. RMSE here is 60,000.
Now predict 298,000. Squared error is 4 × 10⁶, less than a thousandth of the first. That steep ratio is the point of squaring: the model spends capacity on the worst mistakes first. If your data has genuine outliers you do not want to chase, mean absolute error or Huber loss is often the better objective.
The trap: low loss is not the same as good
A model can drive training loss to almost zero by memorising its training set, then fail on new data. That is overfitting, and the loss curve alone will not reveal it. You need a held-out validation set and metrics that reflect the real task.
Loss can also be gamed by the objective itself. A summarisation model trained with next-token cross-entropy is rewarded for fluent, likely text, not for factual accuracy — a mismatch that later appears as confident errors. A reward model in preference tuning is likewise a proxy; optimising it too hard can produce outputs that score well and read badly.
The practical rule: choose a loss that is differentiable and aligned with the decision, then evaluate with something you would defend to a user. Stage 1 builds squared error from scratch; Stage 8 shows next-token loss at scale.