Why it happens
Pre-training rewards one thing: assigning high probability to the next token that actually appeared in the training text. That objective produces an extraordinary model of language and of patterns in human writing, but it never contains the instruction “check this claim against a source”.
So when a model is asked about a rare error code, a recent event, or a specific citation, two forces compete. The learned pattern says a citation should look a certain way — author, year, journal, title. The model’s actual knowledge of that citation may be missing. The pattern wins, and a plausible reference is generated.
Grammatical confidence and factual accuracy come from different things. A sentence can be perfectly formed because it is likely, and wrong because it was never grounded in anything.
It is not lying, and it is not a database
Lying requires an intention to deceive. A model has no intent and no separate belief state to compare against its output. It is also not querying a table of facts: knowledge is distributed across weights as statistical associations, which is why paraphrases succeed and precise identifiers fail.
Hallucination is broader than “made up facts”. In a summary, a model can add a claim that is absent from the source even when the claim is true. That is an unfaithfulness problem, and it matters just as much for trust: the output no longer represents the document you asked it to summarise.
A concrete example
Ask for “three peer-reviewed papers on a narrow topic, with DOIs”. The model may answer with three titles in the correct style, correct-looking author lists and plausible years. Check the DOIs and one resolves to an unrelated paper, another does not resolve at all, and the third is a real paper with a different title. The format was learned; the facts were not verified.
The same failure appears in tool use. If a model is asked to call a weather API and the response is empty, a well-behaved system reports the failure. An unconstrained model may narrate a forecast anyway, because a forecast is the likely continuation of the request.
What reduces it — and what does not
| Mitigation | What it changes | Limit |
|---|---|---|
| Retrieval (RAG) | Puts source text in the prompt | Only helps if retrieval finds the right passage |
| Allow abstention | Rewards “not enough information” | Needs training data and prompts that permit it |
| Citations and verification | Lets a reader or program check claims | Adds latency and engineering work |
| Lower temperature | Reduces random variation | Does not add knowledge; a confident error stays |
| Structured outputs | Constrains format and tool arguments | Format correctness is not factual correctness |
Lowering temperature is the mitigation people try first because it is one parameter, and it is the weakest. It makes the model more repeatable, not more informed. Retrieval and abstention change what the model has access to and what it is allowed to say, which is why they move the needle.
None of these eliminate the problem. Evaluation sets built from your own domain are the only way to know the remaining error rate, and high-stakes uses — medical, legal, financial — need human review regardless of how good the demo felt.
See also “Why does ChatGPT give different answers to the same prompt?” for the decoding side, and Stage 10 for how preference tuning shapes behaviour without adding facts.