The main cause: sampling
A model does not store a finished answer. At every step it scores every token in its vocabulary, softmax turns those scores into probabilities, and the sampler picks one. Temperature and top-p change the shape of that distribution, but even the model’s default settings usually allow more than one plausible continuation.
“The cat sat on the ___” may have 0.6 probability on “mat”, 0.2 on “sofa” and smaller masses elsewhere. Sampling produces “mat” most of the time and “sofa” sometimes. Over a 300-token answer, dozens of such branches compound, which is why two runs can differ in structure and not just in one word.
The same mechanism that makes answers inconsistent makes brainstorming and multiple candidate generation possible. Consistency and diversity are two ends of one dial.
Even temperature 0 is not an absolute guarantee
Greedy decoding — always taking the highest-probability token — removes sampling randomness, but it does not remove floating-point nondeterminism. Summation order differs across batch sizes and GPU kernels, so two mathematically equal logits can compare differently at the last bit. Some inference engines document this explicitly and offer deterministic modes with performance trade-offs.
Seeds help where the API exposes them. A seed makes the pseudo-random stream reproducible, but providers generally describe seeded output as best-effort rather than contractual: a model snapshot change, a different serving backend or a different prompt prefix can still produce different text.
The conversation is part of the prompt
When you type the same sentence a week later, the model may not receive the same input. Chat history, system instructions, memory features and tool results are all inserted into the context window. Editing an earlier message, deleting a conversation or using a temporary chat changes the prefix that the answer is conditioned on.
| What changed | Why the answer moved | Can you control it? |
|---|---|---|
| The sampler drew differently | Temperature and top-p allow several valid tokens | Yes — lower them |
| A new model snapshot | Retrained or re-tuned weights | Yes — pin the snapshot |
| Different history or system prompt | The context prefix is part of the input | Yes — fix the prompt |
| Floating-point ordering | Batch size and kernels change summation | Partly — deterministic mode |
| A seed you did not record | The pseudo-random stream is unreproducible | Yes — store the seed |
Model versioning adds another layer. Providers name snapshots and retire old ones, and a new snapshot can change tone, formatting and refusal behaviour. “Same prompt, different model” is at least as common a cause as “same model, different sample”, especially over months.
How to get enough consistency to test
- 1Set temperature to 0 and keep top-p at 1 when you need the closest thing to deterministic output.
- 2Use a seed if the API provides one, and store the seed with the request.
- 3Pin the exact model snapshot in code rather than a floating alias.
- 4Fix the full prompt: system message, examples, history and tool schemas.
- 5Cache results for expensive calls and evaluate with several runs rather than one.
The limitation is worth stating plainly: reproducibility is an engineering property you approximate, not a switch you flip. For tests, assert on structure and invariants — valid JSON, required fields, bounded length — rather than exact strings. For products, treat variance as part of the design and give users a way to regenerate.
To see the distribution side, read “What does temperature do in an LLM?” and “What is top-p sampling?”. Stage 11 walks through the decode loop where this sampling happens.