Because we taught them that guessing pays.
Picture a multiple-choice exam with no negative marking. Right answer, one mark. Wrong answer, nothing. Leave it blank, nothing. Any student worth their salt guesses every time. You can never do worse.
In September 2025 a group of OpenAI researchers argued that this is how language models are trained and graded. Most tests reward a right answer and treat “I don’t know” exactly like a wrong one. So the models learn to guess, confidently.
Their own numbers make the point. On a factual quiz, an older OpenAI model got slightly more answers right than a newer one: 24% against 22%. But the older one almost never declined, so it was wrong 75% of the time. The newer one declined about half the questions and was wrong 26% of the time. On a leaderboard the older model looks better. In a courtroom I know which one I’d want.
Anthropic found something similar from the inside. When they looked at how Claude handles questions, its default was to decline to speculate. It only answers when something switches that reluctance off. A hallucination, as I read it, is that switch flipping when it shouldn’t: the question feels familiar, so out comes a confident answer.
Law is badly exposed to this. A case citation is exactly the kind of neat, pattern-shaped fact a model has seen thousands of times. A plausible name, a plausible citation, for a trite proposition. It feels familiar. It may not exist.
Legal research tools help but don’t cure it. A Stanford study found hallucinations in 17% of answers from Lexis+ AI, 33% from Westlaw’s AI research tool and 43% from GPT-4. Those were 2024 versions, and things have moved since, but nobody has shown the problem gone.
So my rule stays simple. Nothing gets cited unless the judgment is on file, and every quote is checked against it. More on the damage when that step is skipped in Names of non-existent cases are not law.
Sources: Kalai, Nachum, Vempala and Zhang, Why Language Models Hallucinate (September 2025) and OpenAI’s summary (the quiz figures are SimpleQA, from the GPT-5 system card); Anthropic, Tracing the thoughts of a large language model; Magesh and others, Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, published in the Journal of Empirical Legal Studies in 2025.