Two modes of the same network
The architecture is identical in both phases. The difference is what you do with it. In training, the forward pass is followed by a loss, a backward pass and a weight update. In inference, only the forward pass runs, and the weights stay exactly as they were.
Training is how a model acquires capability: pre-training on a large corpus, then instruction tuning or preference tuning on narrower data. Inference is how that capability is used: a chat reply, a classification, an embedding, a generated image. One training run can serve millions of inferences.
| Aspect | Training | Inference |
|---|---|---|
| Weights | Updated every step | Frozen |
| Gradients | Computed and stored | Not computed |
| Memory | Activations kept for the backward pass | Weights, KV cache and one forward pass |
| Goal | Samples per second over epochs | Latency and cost per request |
| Hardware | Many GPUs for hours to months | One GPU or a hosted API |
What actually differs inside the model
Two behaviours are easy to miss. Dropout randomly zeroes activations during training to prevent co-adaptation, and is switched off at inference. Normalisation layers use batch statistics while training and frozen running statistics afterwards. Calling a model in the wrong mode gives quietly wrong predictions rather than an error, which is why frameworks expose an explicit train/eval switch.
Autoregressive inference has an extra cost: generating token 100 requires the keys and values of tokens 1–99. Recomputing them for every token would be quadratic work per reply, so the KV cache stores them. That cache is why a long conversation consumes memory that keeps growing, and why serving systems manage it as carefully as the weights.
A practical example: fine-tuning versus calling an API
Suppose you want a support assistant that follows your refund policy. Two different operations are often confused here. Fine-tuning runs training on your examples and produces new weights: the behaviour is baked in and available to every future request without extra prompt text. Prompting with retrieved policy text runs inference only: the weights are unchanged, and the policy lives in the context window for that request.
- 1Fine-tune when you need a consistent format across many calls and have enough labelled examples.
- 2Use retrieval when the knowledge changes often, because updating a document is faster and cheaper than retraining.
- 3Use both when the task needs a learned format and fresh facts — the model learns the shape, retrieval supplies the content.
The misconception that matters most
Chatting with a model does not train it. Your messages live in the context window and, if the product stores them, in an application database. The weights are untouched unless the provider explicitly runs a training or fine-tuning job. A new conversation does not remember the last one unless the product retrieves or injects that history.
The limitation cuts both ways. Training cannot guarantee truth, because it optimises a loss rather than verifying facts. Inference cannot fix a capability the weights never learned. And because providers periodically replace model snapshots, an API response can change between deployments even when your prompt did not.
To plan the memory side, the VRAM calculator estimates weights and cache; the API cost calculator turns token counts into a monthly bill. Stage 11 of the course shows prefill and decode step by step.