How nucleus sampling works
A language model gives every token a probability. Most of that mass usually sits on a handful of sensible continuations, while thousands of tokens share a long, thin tail. Sampling from the full distribution occasionally picks a tail token that derails the sentence; always picking the top token produces repetitive text.
Top-p takes a middle path. Sort the tokens by probability, add them up from the top until the running total reaches p, and keep exactly that set. Discard the rest, renormalise the kept probabilities so they sum to one, and sample.
With p = 1 the nucleus is the entire vocabulary, so top-p has no effect and you are sampling from the raw distribution (after temperature). Lower values cut more of the tail.
A worked example
Take five candidate tokens with probabilities [0.50, 0.25, 0.15, 0.06, 0.04]. They already sum to one. With p = 0.8, the cumulative totals are 0.50, 0.75, 0.90 — so the nucleus is the first three tokens, and the last two are removed entirely.
| Token | Original p | Cumulative | Kept at p = 0.8 | Renormalised |
|---|---|---|---|---|
| A | 0.50 | 0.50 | Yes | 0.556 |
| B | 0.25 | 0.75 | Yes | 0.278 |
| C | 0.15 | 0.90 | Yes | 0.167 |
| D | 0.06 | 0.96 | No | — |
| E | 0.04 | 1.00 | No | — |
Raising p to 0.95 keeps four tokens, because the cumulative total only reaches 0.95 after D. Renormalising by 0.96 gives 0.521, 0.260, 0.156 and 0.063. A single parameter has changed both the candidate set and the relative odds inside it.
Top-p versus top-k and temperature
Top-k always keeps the same number of tokens. When the model is certain, that wastes probability on nonsense; when it is uncertain, k may be too small to allow a valid alternative. Top-p adapts: a peaked distribution yields a tiny nucleus, a flat one yields a larger nucleus.
| Control | What it limits | Behaviour |
|---|---|---|
| Temperature | How sharp the distribution is | Reshapes all probabilities |
| Top-k | Number of candidates | Fixed count regardless of confidence |
| Top-p | Cumulative probability mass | Adaptive candidate count |
Because temperature is usually applied first, it changes the ordering of cumulative mass and therefore which tokens fall inside the nucleus. Tuning both at once makes it hard to attribute a change in output quality to one setting; fix one and vary the other when you evaluate.
Practical trade-offs and limitations
Low p (for example 0.1–0.5) is useful for factual extraction and code, where an unlikely token is usually a bug. High p (0.9–1.0) preserves diversity for creative writing and for generating multiple candidate answers. Many providers expose a default near 1.0, which effectively disables truncation unless you change it.
Top-p is not a factuality control. It removes improbable tokens, not incorrect claims: the most probable sentence can still be false, and a nucleus can contain only wrong options. It also cannot recover a token that should have been likely but was assigned low probability by the model — truncation only ever removes candidates.
If you need reproducible outputs for testing, top-p and temperature are the wrong levers; use a seed where the API offers one and pin the model version. The answer on why ChatGPT gives different answers explains the remaining sources of variation.