When AI sounds confident and is wrong: governing hallucination risk

What this risk is

AI systems that inadvertently generate or spread incorrect or deceptive information, which can lead to inaccurate beliefs in users and undermine their autonomy. Humans that make decisions based on false beliefs can experience physical, emotional, or material harms.

This subdomain covers both hallucinations (factual errors generated with false confidence) and plausible fabrications (content that is coherent, well-structured, and wrong).

How it occurs · Mechanisms

Causal profile: AI-caused · Predominantly unintentional · Post-deployment

• Training data limitations — Models learn statistical patterns rather than ground truth; they can generate plausible-sounding false content
• Knowledge cutoffs — Models trained on data up to a certain date cannot know more recent information but may not clearly signal this
• Overconfident generation — Models generate false information with the same fluency and confidence as true information
• Context window limitations — In long conversations, models can lose track of earlier context and contradict themselves
• Source contamination — If training data contains misinformation, models can amplify and spread it

—

Real-world incidents

Air Canada Chatbot and Bereavement Policy (2024)

Air Canada’s AI chatbot incorrectly told a customer that the airline offered bereavement fare refunds, providing false information about a policy that didn’t exist. A Canadian tribunal ruled Air Canada was responsible for the chatbot’s misinformation and ordered compensation.

Lesson: Organizations are legally liable for false information provided by their AI systems.

Lawyers Citing Fake Cases (2023 — Multiple Incidents)

In the Mata v. Avianca case, lawyers submitted a brief citing six court decisions that did not exist — all fabricated by ChatGPT. Similar incidents occurred in Australia, Canada, and the UK. Courts began issuing AI disclosure requirements.

Lesson: In professional contexts (legal, medical, financial), hallucinations can cause serious harm and create professional liability.

Google Bard Factual Error at Launch (2023)

Google’s Bard AI made a factual error in its first public demonstration — incorrectly stating that the James Webb Space Telescope took the first images of an exoplanet. The error, visible in a $10M marketing campaign, caused Alphabet’s stock to drop ~7%.

Medical AI and Clinical Decisions

Multiple studies have documented AI systems providing incorrect medical information — wrong drug interactions, incorrect dosing guidance — when used by clinicians or patients without adequate verification protocols.

Mitigations · Governance

• Retrieval-Augmented Generation (RAG) — Ground model outputs in verified, up-to-date knowledge sources rather than relying solely on parametric knowledge
• Citation requirements — Require AI systems to cite sources for factual claims; implement source verification
• Confidence indicators — Design systems to express uncertainty rather than generate false confidence
• Human review for high-stakes outputs — Professional, medical, legal, and financial AI outputs should be reviewed by qualified humans before acting
• Factual accuracy testing — Benchmark models on factual accuracy tasks relevant to deployment context before launch
• Output disclaimers — For high-risk domains, include explicit disclaimers that AI outputs require verification
• Correction mechanisms — Enable users to flag and report incorrect information; use feedback to improve systems

—