Language ModelsExplainer

What Is AI Hallucination, and Why You Cannot See It

A hallucination is a confident answer that is wrong. I counted such answers on 1,534 questions. When I made the models repair their failures, the number of wrong answers that look fine went up.

By Vladimir Beskorovainyi

6 min read

Illustration: BESK TECH

An AI hallucination is an answer from a language model that sounds confident and plausible but is not true. The model does not look the answer up. It continues the text with the most likely words, and when it lacks the knowledge it keeps going instead of stopping. A hallucinated answer looks the same as a correct one, so it can only be caught by checking it against a source.

An AI hallucination is an answer from a language model that sounds confident, looks plausible, and is wrong. The model names a book that does not exist, gets a date wrong, or cites a court case nobody decided. Its tone is the same as in a correct answer.

AI hallucination examples

The first is from a courtroom. On November 13, 2026, the Delaware Court of Chancery, in Leiske v. Kidd, sanctioned a lawyer 2,500 dollars for not verifying a brief he had signed.1 The brief contained invented quotations and misdescribed cases. The opinion records that the opposing side found "hallucinated legal propositions" in it. According to the opinion, a generative AI tool was used to help draft and condense arguments in the brief, and the lawyer who signed it filed it without personally verifying the authorities. The cases were real. The quotations attributed to them were not. A quotation like that looks like any other. The only way to tell is to open the source.

The second is from my own measurements. A model, Qwen2.5-Coder-32B, was asked how many owners and how many authorised users are recorded for accounts one to ten. The right answer is two numbers, one for each kind. The model wrote an SQL query that returned one number, the sum of both. The query ran. There was no error message. The result looked like an answer.

Nothing in the second case is invented the way the court quotations were. I count it as the same failure because the reader gets the same thing: an answer that looks normal and is wrong.

Why does AI hallucinate

A language model does not take the answer from a reference book. It continues the text with the words most likely to come next. When it has the knowledge, the continuation matches the fact. When it does not, it continues anyway.

In September 2025, researchers at OpenAI named a cause. The usual ways of training and evaluating models reward guessing and do not reward admitting uncertainty.2 An answer of "I don't know" earns zero points. A guess sometimes earns one. A model selected by such scores learns to guess.

The same article gives an example from one test. One OpenAI model declined to answer 52% of the questions and was wrong on 26%. Another declined 1% and was wrong on 75%. Their accuracy is close, 22% and 24%. By accuracy, they know about the same amount. The second almost never declines, so it is wrong nearly three times as often.

What is the AI hallucination rate

There is no single rate. The pair above shows why: two models with almost the same accuracy differ nearly threefold in wrong answers, because one declines when unsure and the other guesses. The rate also depends on the task. A number measured on trivia questions says little about legal citations or database queries.

So I measure it on a task where an answer can be checked exactly. The model gets a question in plain words. It turns the question into an SQL query, and the database runs it. The result is compared with the correct one. There are 1,534 questions and eight runs: seven models, one of them run a second time through a different provider. All seven are open models that can be downloaded and run on your own hardware. I did not test hosted models such as ChatGPT, so these numbers say nothing about them. The code and the answer to every question are published.3

On this test the best model, Qwen2.5-Coder-32B, returned a wrong result with no error message on 571 of 1,529 questions, about 37%. On another task the number will be different.

A wrong query ends in one of two ways. It may fail to run, and the database returns an error message. That error is visible at once. Or it runs and returns the wrong data. There is no error message, and there is a result.

What happened when I made the models fix their errors

Then I added a check. If the query fails to run, the model gets the error text and tries again, up to two times. The table shows four of the eight runs. A loud error is a query that did not run. A silent error is a query that ran and returned a wrong result.

ModelLoud beforeLoud afterSilent beforeSilent afterAccuracy before, %Accuracy after, %
Qwen2.5-Coder-32B116245716405556
Llama-3.3-70B79176116585556
Llama-3.1-8B3932015676983741
CodeLlama-13B6193745076842631

Five questions whose reference query itself fails to run are left out of the loud and silent counts.

Accuracy went up in all eight runs, in one of them within the margin of error. Loud errors went down in all eight. Silent errors went up, also in all eight.

The check asked the model for one thing: a query that runs. The model delivered that. Some of the repaired queries became correct, and accuracy rose. The rest began to run and return the wrong data. The error stayed, and the message about it was gone.

For the best model, 672 answers were still wrong after the check. 640 of them came from queries that in the end ran without an error, about 42% of all questions. Another 24 failed loudly, and the remaining 8 are the five excluded questions and three that ended some other way. That is 95 of every 100 wrong answers with no error message.

The query from my example is one of them. It was wrong before the check and wrong after it, and it never produced an error, so the check never looked at it.

The check removed far more error messages than errors.

How to check an answer

That an answer arrived tells you nothing about whether it is correct. Correctness has to be checked separately, and that takes work.

Three habits anyone can use.

  1. Open the source behind every link, quotation and case number. If you cannot find it, treat the claim as unverified.
  2. If you gave the model a long document, ask it to copy out word-for-word quotations first and to answer only from them. Anthropic describes this technique in its documentation and says in the same place that it does not remove errors completely.4
  3. Tell the model it may say "I don't know". Without that line it is more likely to guess.

If you build a system on a model, add a fourth. Keep a set of questions to which you know the answers, and run it after every change. That shows how often the model is wrong on your tasks, not on someone else's test.

I do not judge an answer by how it looks. In my measurements, 95 of every 100 wrong answers from the best model came with no error message.

Footnotes

  1. Leiske v. Kidd, Court of Chancery of the State of Delaware, opinion of November 13, 2026. ↩

  2. Why language models hallucinate, OpenAI, September 5, 2025. ↩

  3. Repository nl2sql-onprem-bench, release 0.3.0, DOI 10.5281/zenodo.23445116. Method: arXiv:2606.29733. ↩

  4. Reduce hallucinations, Anthropic documentation. ↩

Sources

  1. Why language models hallucinate, OpenAI
  2. Leiske v. Kidd, Court of Chancery of the State of Delaware, opinion of November 13, 2026
  3. How far do on-premises open LLMs get on Text-to-SQL? arXiv:2606.29733
  4. nl2sql-onprem-bench, code and per-question results
  5. Reduce hallucinations, Anthropic documentation

Vladimir Beskorovainyi

Vladimir Beskorovainyi, Enterprise AI Architect & CTO, author of the Amazon bestseller "Enterprise AI: The KATA Way", founder of Besk Tech (besk.tech)

More in this section

Get new articles by email

One message when something new is published. No other mail, and you can stop at any time.

The address is used for this one letter and nothing else. Privacy

← All articles