Hallucinations remain one of the biggest barriers to enterprise AI adoption. In this article, our Head of Data and AI, Ashwin Shekhar, explores why they occur, why they can’t simply be trained away, and how approaches like Retrieval-Augmented Generation (RAG), verifications and prompting could be some of the steps to build a more trustworthy AI for more reliable real-world applications.
Why does AI Hallucinate
AI is becoming ubiquitous in everyday life and increasingly, people are relying on it without questioning what it says. Yet these systems still confidently invent facts, or more technically, hallucinate. At CulturalPulse we are working in industry sectors such as healthcare and government, where AI must be trustworthy and getting it wrong isn’t an option.
We have had to ask ourselves:
-
Why does hallucination happen?
-
Is it getting better?
-
What can we do now to make it more trustworthy?
-
Will we ever be able to fully trust AI not to make things up?
Let’s look at the Brian Hood defamation case where the mayor of Hepburn Shire in Victoria was named by ChatGPT as being involved in a foreign bribery scandal. Hood later told ABC News the AI chatbot stated he had been “charged with very serious criminal offences” and convicted. In reality, Mr Hood had actually been the whistleblower who alerted authorities to the case more than a decade ago.
So why did this happen?
What many people don’t realise is that large language models or LLMs (the technology behind ChatGPT, Claude, and similar systems) don’t understand information and respond in the way that humans do.
A useful way to think about them is as extraordinarily sophisticated prediction engines (or autocomplete) . Given your prompt (or input), they predict the most “likely” next word, then the next, then the next thereby building an answer word by word from patterns in their training data, not from any “understanding” of truth.
A recent OpenAI research paper argued that hallucinations are not simply a bug that can be worked out, the issue is structural and baked into how the models are trained. The models are graded on a reward structure that favours confident guessing over admitting uncertainty. Imagine a student on a multiple-choice exam where there’s no penalty for a wrong answer or guessing incorrectly, but a penalty for leaving the answer blank.
Is it getting better?
There is good news, with some important caveats.
One of the industry’s benchmarks, Vectara’s industry standard HHEM leaderboard measures and tracks how often models invent facts when summarising documents. Summarisation is the chosen test because the source provides a verifiable ground truth. Therefore if the model introduces anything that isn’t there, it is unambiguously a hallucination.
The latest rankings (May 2026) show a wide range of results. Ant Group’s finix_s1_32b leads at 1.8%, while OpenAI’s GPT-5.5 sits at 9.3% and Anthropic’s Claude Opus 4.7 at 12.0%.
It is interesting to note that counterintuitively, the “reasoning” or “thinking” models often perform worse and it seems that their extra deliberation can sometimes allow them to drift from the source material, introducing information that sound convincing, but actually are not supported by the evidence in the text it is trying to summarise.
Overall, the trend is moving in the right direction.
Better curated training data, learning from human feedback (where reviewers shape model behaviour), and training methods that explicitly reward admitting uncertainty over confident guessing are helping to reduce hallucinations.
But yes, it still happens
Remember, the underlying mechanism that defines how these models work hasn’t changed. At their core, they’re still sophisticated autocomplete and continue predicting words one at a time from patterns. Better training reduces how often they go wrong but it doesn’t change what they fundamentally are.
In January 2026, King’s Counsel Rishi Nathwani apologised to the Supreme Court of Victoria after his AI-assisted submissions in a teenage murder case contained fabricated quotes and citations to non-existent Supreme Court cases. It serves as a cautionary tale for any profession leaning on AI for mission critical or high value work. This is far from an isolated example.
Globally, a public database maintained by researcher Damien Charlotin tracks 1,455 court cases involving AI hallucinations and some of them are in Australia. The technology is evolving and the error rates are falling, but the evidence suggests improvements are containing the problem, not solving it.
What can we do?
The most effective solutions don’t try to change the AI, they change what it’s allowed to work with.
One of the most common approaches is Retrieval-Augmented Generation (RAG). Instead of letting the AI recall facts from its training, you restrict it to a curated set of verified sources or a “constrained knowledge base.” The AI’s job becomes finding and using those sources, not “remembering” based on its own training data. Modern RAG systems retrieve relevant passages by meaning rather than by keyword, using vector databases and vector search to find semantically similar content.
Other techniques can help too, such as requiring citations that can be verified, validation tools that fact-check AI output after the fact and prompting models to say “I don’t know” rather than guess.
At CulturalPulse, these are the foundations of our new AI based health platform, HealthGenie: a multi-agent companion platform to help with many aspects of health. Our first domain is pregnancy and birth, and this is currently in closed trials with expectant mothers and has been successfully trialled with midwives. HealthGenie is restricted to approved documents from trusted clinical sources, with guardrails ensuring its answers stay grounded in verified information. Its agents can also perform approved tasks on the patient’s behalf.
But these “tricks” are engineering or coding solutions that work around the LLM rather than changing how the model works. They are making AI more reliable but they don’t fix the underlying hallucination problem.
So can we ever trust AI?
Hallucinations are unlikely to disappear entirely. But increasingly, they’re becoming an engineering challenge that can be measured, managed and significantly reduced through better models, better data and more reliable safeguards..
The lesson is to use AI with curious confidence, embrace its strengths and understand its limitations. For now AI remains a powerful tool. But it is still a tool and AI does not replace human intelligence. Not yet.
The future of AI won’t belong to the organisations using the most sophisticated or largest language models, it will belong to those building the most trustworthy systems around them.
At CulturalPulse that’s the future we are building.
About the Author
Ashwin Shekhar leads CulturalPulse’s AI and technology team, driving the design and development of trustworthy AI systems for healthcare, multicultural engagement and public sector. His work focuses on Retrieval-Augmented Generation (RAG), multi-agent systems, data science and responsible AI engineering. Combining strategic leadership with deep technical, Ashwin helps organisations harness AI safely, responsibly and at scale to solve complex real world challenges for clients and partners.
