What the dial actually changes
At each generation step the model produces one raw score, called a logit, for every token in its vocabulary. Softmax turns those scores into a probability distribution. Temperature T is applied inside the softmax, dividing every logit first:
As T approaches zero, the largest logit dominates and the distribution becomes a one-hot choice — greedy decoding. As T grows, differences between logits shrink and probability spreads towards a uniform distribution over the vocabulary. The sampling method then draws one token from whatever distribution survives.
Most APIs apply temperature to the logits and then truncate with top-p or top-k. The order matters: reshaping the distribution changes which tokens fall inside the nucleus.
A worked example with three tokens
Suppose the next-token logits are [2, 1, 0] for tokens A, B and C. At the default temperature of 1 the probabilities are about 0.665, 0.245 and 0.090. Change only the temperature and the shape moves:
| Temperature | A | B | C | What it means |
|---|---|---|---|---|
| 0.5 | 0.867 | 0.117 | 0.016 | Sharper: A is almost always chosen |
| 1.0 | 0.665 | 0.245 | 0.090 | The model’s own distribution |
| 2.0 | 0.506 | 0.307 | 0.186 | Flatter: B and C get real chances |
At T = 0.5 the top token is about 70% more likely than at T = 2, while the third token becomes roughly twelve times less likely. Over a long response those per-token differences compound: a setting that looks harmless for one word determines whether a hundred-word answer stays on the rails.
What the settings feel like in practice
Low temperature (roughly 0–0.3) is appropriate for extraction, classification, code transformation and anything with one correct answer. Output becomes repeatable and conservative, but also repetitive: the model may loop or refuse to explore an alternative phrasing.
High temperature (roughly 1.0 and above) is appropriate for brainstorming, fiction and diversifying candidate answers in an evaluation harness. It increases variety and also the rate of incoherence, invented details and broken formatting. It does not make the model more creative in the sense of having better ideas; it samples less likely continuations.
There is no universal best value. The same request can need different settings depending on whether you are generating a JSON payload or a poem, and providers differ in how their scales behave.
Misconceptions and limits
- “Temperature 0 is fully deterministic.” Usually it is more repeatable, not guaranteed identical: floating-point summation order, batching and hardware can change results.
- “Higher temperature makes the model smarter.” It makes the output more diverse. Knowledge and reasoning do not improve.
- “Low temperature prevents hallucination.” It reduces random variation, so the model is more likely to repeat its most probable claim — true or false.
- “Temperature changes the model’s confidence.” The model has no calibrated confidence dial; temperature changes the sampling distribution, not the evidence.
Temperature is one control in a sampler. If you also want to cut the long tail of unlikely tokens, top-p does that explicitly; the two are usually tuned together. Stage 11 shows where sampling sits in the inference loop.