Backpropagation

After a forward pass you know exactly one thing: how wrong the final answer was. But there are millions of weights, each contributing a little to that error. Backpropagation answers, for every single weight, "if I nudge you slightly, how much worse does the final loss get?"

The maths is the chain rule from calculus, applied relentlessly. If loss depends on the output, and the output depends on an activation, and the activation depends on a weighted sum, and that depends on a weight — multiply the local sensitivities together to get the weight's share of the blame.

1. 1Forward pass: inputs flow through the layers, each node caching its output.
2. 2Loss: compare the final output with the target and reduce it to one number.
3. 3Backward pass: walk the graph in reverse, multiplying local gradients as you go.
4. 4Store: keep ∂L/∂w for every parameter — that is the gradient vector.
5. 5Update: the optimiser nudges all weights against their gradients.

# A 2-layer network: backprop by hand
# forward
h = relu(x @ W1 + b1)      # hidden activations
y_hat = h @ W2 + b2        # predictions
loss = mse(y_hat, y)

# backward (chain rule, applied in reverse)
dL_dyhat = 2 * (y_hat - y) / n
dL_dW2 = h.T @ dL_dyhat
dL_dh  = dL_dyhat @ W2.T
dL_dz1 = dL_dh * (z1 > 0)  # ReLU derivative: 1 if z>0 else 0
dL_dW1 = x.T @ dL_dz1

# update
W1 -= lr * dL_dW1
W2 -= lr * dL_dW2

Backprop multiplies a chain of local derivatives. If each is smaller than 1, the product collapses toward zero: vanishing gradients. Early layers then receive almost no learning signal and effectively freeze. If each is larger than 1, the product explodes to infinity: exploding gradients, and your weights become NaN.

• Sigmoid / tanh saturate: derivatives ≤ 0.25. Depth kills them.
• ReLU has derivative exactly 1 when active — the modern default.
• Careful initialisation (Xavier, He) keeps the variance of activations ~1 across layers.
• Normalisation (BatchNorm, LayerNorm) re-centres activations at every layer.
• Residual connections add a highway: gradients flow through the + unchanged.
• Gradient clipping caps the update norm to stop explosions.

import torch

x = torch.tensor([[0.5, -1.2, 3.3]])
w = torch.randn(3, 4, requires_grad=True)

# forward: every op is recorded on the autograd tape
z = x @ w
a = torch.relu(z)
loss = a.pow(2).sum()

# backward: chain rule applied over the recorded tape
loss.backward()

print(w.grad)     # dL/dw, computed for free
w.data -= 0.01 * w.grad

Frameworks build a computational graph during the forward pass, then walk it in reverse. This is why a PyTorch model costs more memory while training than while serving: the cached activations must be kept alive until the backward pass consumes them.

• Learning-rate schedules (warmup, cosine decay) are standard: start small, grow, then shrink.
• Adam / AdamW adapt a per-parameter step size so you worry less about a global value.
• Early stopping watches validation loss and halts when it stops improving.
• Weight decay adds a penalty on large weights to discourage memorisation.