What the algorithm is actually doing
A network makes a prediction, and a loss function turns the difference between prediction and target into one number. The hard question is blame: with millions of weights, which ones made the error worse, and by how much?
Backpropagation answers that efficiently. A forward pass computes the prediction and caches every intermediate value. A backward pass walks the network in reverse, multiplying local derivatives together to get the derivative of the loss with respect to each weight. That derivative is the gradient: the slope of the loss if you nudge that weight.
Backpropagation only computes gradients. Gradient descent, Adam or another optimiser decides how to use them. Keeping the two ideas separate makes both easier to reason about.
The chain rule, with real numbers
Take the simplest model: ŷ = w · x, one input x = 2, one weight w = 0.5, target y = 3, and squared error L = (ŷ − y)². The prediction is 1.0, so the loss is (1 − 3)² = 4.
To find how the loss changes when w changes, multiply the local derivatives along the path from L back to w:
With a learning rate of 0.1, the update is w ← w − 0.1 × (−8) = 0.5 + 0.8 = 1.3. The next prediction is 2.6 and the loss falls from 4 to 0.16. That is one complete step: forward pass, loss, backward pass, update.
| Step | Computation | What it keeps |
|---|---|---|
| Forward | ŷ = w · x, then L = (ŷ − y)² | x, ŷ, L |
| Backward | ∂L/∂ŷ = 2(ŷ − y), ∂ŷ/∂w = x | gradient ∂L/∂w |
| Update | w ← w − η · ∂L/∂w | new w |
Why the learning rate decides whether training works
The learning rate η scales every step. Too small and training crawls, burning compute for a tiny improvement. Too large and the update overshoots the minimum: the loss bounces, then grows, then becomes NaN as numbers overflow.
In deep networks the gradient itself can misbehave. Repeatedly multiplying derivatives smaller than one makes early layers receive almost no signal — the vanishing gradient. Multiplying derivatives larger than one does the opposite: the exploding gradient. ReLU activations, careful initialisation, normalisation layers and residual connections exist largely to keep these products in a workable range.
- The update rule w ← w − η·∂L/∂w is gradient descent; Adam adds per-weight momentum and scaling on top of the same gradient.
- Averaging gradients over a batch trades noise for stability, which is why batch size and learning rate are usually tuned together.
- Clipping the gradient norm stops a single bad batch from destroying the weights.
You write the forward pass; the framework writes the backward pass
Modern libraries record the operations performed during the forward pass and differentiate that graph automatically. This is reverse-mode automatic differentiation, often called autograd. Backpropagation is what it computes for a scalar loss; the same machinery handles shared weights, branches and attention, which are painful to differentiate by hand.
The cost is memory: cached activations must survive until the backward pass, which is why training a model needs far more memory than serving it. The other limit is differentiability. A discrete choice such as “sample this token” has no useful derivative, so preference tuning relies on surrogate objectives or reinforcement learning instead of plain backpropagation.
This is also why backpropagation cannot simply “fix” a wrong fact. Training optimises the loss you defined, and no ordinary differentiable loss compares a sentence with the truth of the world. If you want the arithmetic of the loss itself, read “What is a loss function in machine learning?”