In-Depth
Avoiding the Generative AI Hallucination Trap
Over the last few years, businesses have been increasingly turning to generative AI in an effort to boost employee productivity and streamline operations. However, overreliance on such technologies poses a major risk. These systems, while usually helpful, can experience "hallucinations" in which they produce information that sounds right, but is factually incorrect or misleading. Businesses who act on such information can suffer severe consequences ranging from reputational damage and financial loss to misguided road mapping. Despite serious potential consequences, businesses are increasingly acting on bad AI generated information.
What Is an AI Hallucination?
An AI hallucination refers to a situation in which a generative AI model produces information that sounds correct and is completely plausible, but incorrect. Such information might be slightly misleading or it may be completely fabricated (or fall somewhere in between).
To show you just how bad a hallucination can be, take a look at the information shown in Figure 1. This information was produced by an AI model called Gemma3, which was created by Google. I am running the 27 billion parameter model, which is the most advanced version of Gemma3. In this example, I asked Gemma3 to tell me about myself. The AI produced a biography that is not even close to being accurate. I won't bother going into everything that the AI got wrong, but I will tell you that I am not a she (and never have been), nor have I ever worked as a SANS instructor, or served in the US Air Force.
[Click on image for larger view.] Gemma3 Has Completely Fabricated My Bio.
The information shown in the screen capture clearly fits the definition of an AI hallucination because it's wrong, and yet seems plausible. In fact, if you scroll further down in the output, there are even a series of references that make the information seem more credible, as shown in Figure 2. However, these links point to non-existent pages. One of the things that I found particularly amusing is that the AI output mentioned me being very active on Twitter, when in reality, I don't even have a social media presence.
[Click on image for larger view.] Gemma3 Has Even Fabricated Its References.
Why Do Hallucinations Happen?
The screen captures shown above do not necessarily mean that Gemma3 is hopelessly flawed and unreliable. I have often received good information from Gemma3. Instead, the screen captures point to a problem that seems to exist within all generative Large Language Models, or at least all of the models that I have tried.
Hallucinations happen because most generative AI models do not "know" facts in the same way that a human does. Instead, the output that these models produce is based on patterns and probabilities. When such a model produces text output, it does its best to predict the next word that should occur, based on having been trained on a vast dataset. There are AI models that will try to cross check generative output against a collection of known facts. The problem with this approach is that it is impossible for an AI to know everything.
There are several reasons beyond simple text output predictions why generative AI sometimes experiences hallucinations. For starters, the AI training data is not real time in nature and facts can change without the AI model knowing it. Consider as an example, that someone once gave me an encyclopedia that was written in 1952. Much of the information in that encyclopedia was wrong, even though it was correct at the time that the encyclopedia was written. Consider for example, that in 1952 humans had not yet been to space, Alaska and Hawaii were not yet states, and the PC was decades away from being invented. The same thing can happen with AI, because the rapidly evolving nature of information found online means that AI training data can quickly become outdated.
It isn't just outdated training data that can lead to hallucinations. Sometimes there might simply not be enough training data on a particular subject or the user's prompt could be ambiguous or overly complex. In these types of situations, AI may try to "fill in the blanks" in an effort to be helpful.
One more problem that can lead to AI hallucinations is that AI may incorrectly combine unrelated facts or make logical leaps that lead to a fallacy. In math, there is something called the Transitive Property that allows values to be substituted if they are equal. As an example, if A=B and B=C, then that means that A=C. In the world of generative AI, this transitive property is sometimes incorrectly applied to facts, leading to some bizarre results. This might stem from things like word play or using the same word in a way that has two different meanings. As an example, an AI model might latch onto the old expressions "time is money" and "money is the root of all evil" and therefore incorrectly determine that time is the root of all evil.
Recognizing an AI Hallucination
Although some generative AI models are more prone to hallucinations than others, every LLM that I know of has at least the potential for experiencing a hallucination. Hallucinations are likely unavoidable (at least until technology further advances), so the trick for businesses is to be able to distinguish between a hallucination and good information.
This all starts with education. It's critically important for all employees, from the CEO on down, to be aware of the fact that generative AI is not 100% reliable.
There are also several different ways of helping to spot a hallucination. This might include looking for inconsistencies within the answer that AI gives to a prompt. Data that is overly precise can also sometimes be attributed to a hallucination. If AI gives you a long list of facts, figures, and statistics and it can't point to the sources where the data came from, then the data probably stemmed from a hallucination.
The best thing that a business can do to avoid falling for a hallucination when using a general-purpose generative AI tool is to require the tool to provide references and then take the time to verify those references. That last part is crucial. After all, the example that I showed you earlier included references, but the links within those references were invalid and the references were completely fabricated.
A better approach to avoiding hallucinations is to build your own AI and train it against a carefully curated dataset, such as the organization's own internal business data. Of course this approach requires special care in order to avoid compliance violations or the accidental exposure of personally identifiable information. Keep in mind however, that hallucinations are still possible (though less likely) even when an AI has been trained solely on good data.
About the Author
Brien Posey is a 22-time Microsoft MVP with decades of IT experience. As a freelance writer, Posey has written thousands of articles and contributed to several dozen books on a wide variety of IT topics. Prior to going freelance, Posey was a CIO for a national chain of hospitals and health care facilities. He has also served as a network administrator for some of the country's largest insurance companies and for the Department of Defense at Fort Knox. In addition to his continued work in IT, Posey has spent the last several years actively training as a commercial scientist-astronaut candidate in preparation to fly on a mission to study polar mesospheric clouds from space. You can follow his spaceflight training on his Web site.