Hallucination is the industry word for an AI stating something untrue with complete confidence. It is not the model lying or malfunctioning. It is the direct result of a system built to produce likely looking text, asked a question where it has no reliable material to draw on. Knowing where it happens is most of the defence.
Why it happens at all
The model is always producing a plausible continuation. If it does not have the real answer, the most likely looking text is still something that reads like an answer, so out it comes. There is no internal check that stops it and says the information is missing.
Citations and sources are a classic case
Ask for references and you may get authors, titles, and journal names that look entirely convincing and do not exist. They follow the pattern of a real citation, which is exactly what the model was producing. Every reference needs checking before you rely on it.
Numbers, quotes and product details
Statistics with a confident source, quotes attributed to a real person, specifications for a particular model of equipment. All of these are places where a plausible looking value is easy to generate and hard to spot as wrong.
It happens most where the material is thin
Very specific, very recent, or very local questions are the riskiest, because there was little in the training data to draw on. A niche supplier, a small local rule, or last month's change is exactly the sort of gap that gets filled with something invented.
Grounding is the strongest defence
Paste in the real document and tell it to answer only from what you supplied, and to say clearly if the answer is not there. This narrows the job from recalling to reading, which is what the tool is good at, and it gives you something to check the answer against.
Common mistakes
- Accepting a reference, case, or source without opening it to confirm it exists.
- Quoting a statistic from an AI answer in a proposal or on your website.
- Asking about a very specific local rule, small supplier, or recent change and trusting the reply.
- Assuming a longer, more detailed answer is a more accurate one.
- Asking the model whether it is sure, and taking yes as confirmation.
- Asking a question about a document without actually giving it the document.
Check yourself
0/41.You ask for three sources supporting a point in a proposal, and get three convincing looking references. What must you do?
2.Which of these questions carries the highest risk of a confidently wrong answer?
3.What is the most effective way to reduce hallucination when you need an answer from a specific document?
4.You challenge an answer and the model apologises, then gives a different answer just as confidently. What does this show?
This is general guidance, not a substitute for advice on your specific setup. Want a hand putting it into practice? Talk to us or see our care plans.