AI Hallucinations: What They Are, Examples & Prevention Guide

What an AI hallucination actually is

A hallucination is when an AI tool produces information that is false, made up, or unsupported, and presents it as if it were true. The key word is confidently. The tool does not flag any doubt, because as far as it is concerned there is nothing to doubt.

This is the part that trips people up. A search engine that cannot find something tells you it cannot find it. A large language model almost never does that. It is built to produce a fluent, plausible answer, so when it does not know, it fills the gap with something that sounds right rather than admitting the gap exists.

So a hallucination is not a bug in the ordinary sense. It is a direct consequence of how these tools work, which we get into further down.

The blueberry test: when “PhD-level” AI cannot count

When OpenAI launched GPT-5 in August 2025, Sam Altman described it:

“GPT-5 is the first time that it feels like talking to an expert in any topic – a Ph.D.-level expert,”

It is a genuinely capable model. It is also the model in the screenshot below, getting a question my six-year-old son could answer incorrectly.

The question was “How many bs are in blueberry?” The answer it gave: three. One in “blue”, two in “berry”.

There are two b’s in blueberry. One in “blue”, one in “berry”. The model not only landed on the wrong total, it invented a breakdown to justify it, confidently stating that “berry” contains two b’s when it plainly contains one. There is no hedging, no “let me check”. It is just wrong, and sure of itself.

The letter-counting failure (the more famous version is “how many r’s in strawberry”) happens because these models do not see words as strings of letters the way you do. They break text into chunks called tokens, and counting characters across tokens is exactly the kind of task the architecture is bad at. It is not a sign the model is stupid. It is a sign that “sounds confident” and “is correct” are two completely separate things, and the model only optimises for the first one.

Hold onto that thought, because the gap between confident and correct is where the real damage happens.

When hallucinations get expensive

A miscounted blueberry is funny. The same flaw, pointed at something that matters, is not. Here are two documented cases that went well beyond a party trick.

Air Canada’s chatbot invented a refund policy

In 2022, a passenger named Jake Moffatt asked Air Canada’s website chatbot about bereavement fares after his grandmother died. The chatbot told him he could book at full price and claim a discount retroactively, within 90 days. That was not Air Canada’s policy. The chatbot had made it up.

When Moffatt tried to claim, the airline refused. He took it to British Columbia’s Civil Resolution Tribunal, and in February 2024 the tribunal ruled against the airline. Air Canada had argued, remarkably, that the chatbot was a separate legal entity responsible for its own actions. The tribunal member called that submission exactly that, “remarkable”, and pointed out that a chatbot is simply part of Air Canada’s website, so the airline is responsible for what it says. Air Canada was ordered to pay C$812.02 in damages and fees.

The lawyers who cited six cases that did not exist

In the New York case Mata v. Avianca, two lawyers submitted a legal brief that cited six prior court decisions as precedent. The problem: none of the six existed. They had been generated by ChatGPT, complete with realistic case names, quotes and internal citations, all fabricated.

When the opposing side could not find the cases, the court asked for copies. The lawyers, still relying on ChatGPT, supplied fake “copies” of the fake cases. In June 2023 the judge fined them US$5,000. Notably, the judge was clear that using ChatGPT was not itself the offence. The real problem was doubling down and failing to check before putting fabricated work in front of a court.

Why AI hallucinates in the first place

It helps to know what a large language model is actually doing, because once you do, hallucinations stop being mysterious.

A model like ChatGPT is, at heart, a very sophisticated prediction engine. It has been trained on an enormous amount of text, and its core job is to predict the next most likely chunk of text given what has come before. It is not looking anything up. It is not consulting a database of facts. It is generating the most statistically plausible continuation.

That design has a few consequences worth understanding:

• It optimises for plausible, not true. A fluent, confident answer scores well on “sounds right” whether or not it is right.
• It has no built-in sense of “I don’t know”. Unless specifically trained or prompted to hedge, it will produce an answer rather than admit a gap.
• It does not see text the way you do. Words are broken into tokens, which is why character-level tasks like counting letters fall over.
• Its training data has a cut-off. Ask about something recent and, without live web access, it may confidently describe a world that no longer exists.

How to spot a hallucination

You will not catch every one, but most share a few tells. Be on guard when you see:

• Specific facts, statistics or dates with no source, especially if they seem a little too neat.
• Named studies, books, cases or quotes you cannot independently find. Fabricated citations are a classic.
• Confident answers about very recent events, where the model may be past its training cut-off.
• Anything where the model contradicts itself across a conversation.
• Precise-sounding detail on a niche topic, delivered with the same certainty as a basic fact.

The simplest rule: the more confident and specific the claim, and the higher the cost of being wrong, the more it needs checking against a real source.

How to use AI tools without getting burned

None of this means avoid AI. We use these tools every day at QED, and they are genuinely brilliant for the right jobs. It means using them with your eyes open.

• Use it for drafting, not deciding. First drafts, brainstorms, rewording, structuring ideas: excellent. Final facts, figures, legal or medical points: verify every time.
• Verify anything that carries a cost. If being wrong would cost money, reputation or safety, check it against a primary source before you act.
• Ask for sources, then check they exist. If a tool cites a study or a case, look it up. A real source is real. A hallucinated one vanishes the moment you search for it.
• Use tools with live web access for anything recent, and still confirm the underlying source rather than trusting the summary.
• If you put a chatbot on your own site, supervise it. As Air Canada learned, you are liable for what it tells your customers.

This is also why, when it comes to the content on your website, a human who understands your business still matters. AI can help draft a page in seconds, but it cannot tell you whether the claim in it is true, whether it reflects what you actually do, or whether it will quietly invent a policy you never offered. That judgment, and the responsibility that comes with it, stays firmly with people.

If you would rather have your website content written and checked by humans who use AI as a tool rather than a replacement, that is exactly how we work at QED, as we have one eye on the true cost of AI data centres and their impact

Sources

• Moffatt v. Air Canada, British Columbia Civil Resolution Tribunal, 2024 BCCRT 149, decision issued 14 February 2024.
• Mata v. Avianca, Inc., United States District Court, Southern District of New York, Opinion and Order on Sanctions, 22 June 2023 (Case No. 1:22-cv-01461).
• OpenAI, “Introducing GPT-5”, product announcement, August 2025.
• ChatGPT 5 screenshot, captured by QED Web Design, 2025.