When AI medical advice fails the patients who trust it most

What this risk is

AI systems providing incorrect, outdated, or dangerous health information to patients, caregivers, or clinicians — including wrong dosing, incorrect contraindications, fabricated clinical trial results, and inappropriate medical advice — leading to patient harm.

Healthcare is the highest-consequence domain for AI misinformation because errors can directly cause physical harm.

How it occurs · Mechanisms

Use Case	Misinformation Risk	Potential Harm
Patient-facing chatbots	High (no clinical oversight)	Wrong self-treatment, delayed care
Clinical decision support	Medium-High (clinician oversight)	Misdiagnosis, wrong treatment if accepted without review
Drug information queries	High (precision required)	Dosing errors, dangerous interactions
Medical literature search	High (hallucinated citations)	Evidence-free clinical decisions
Discharge instructions	High (patient acts independently)	Post-discharge complications
Mental health chatbots	High (vulnerable population)	Harmful advice to at-risk users

Real-world incidents

National Eating Disorders Association Chatbot (2023)

NEDA replaced human helpline staff with an AI chatbot (Tessa). Within days, the chatbot was found to be providing diet tips to people seeking help for eating disorders — the opposite of safe, evidence-based care. NEDA shut down the chatbot and faced significant criticism.

AI Chatbots and Suicide Risk (Multiple, 2023–2024)

Multiple studies documented AI chatbots providing inappropriate responses to suicide-related queries — including detailed method information, failure to provide crisis resources, and in some cases reinforcing suicidal ideation.

NHS Health Information Errors

AI-generated health information on NHS-adjacent platforms was found to contain outdated guidance, incorrect drug information, and contradictions with NHS official guidance.

Mitigations · Governance

• Clinical content review — All health information generated by AI must be reviewed by qualified clinicians before deployment
• Evidence-based knowledge base — Ground health AI in validated, regularly updated clinical knowledge bases (not open internet)
• Scope limitation — Health AI should clearly define and enforce limits on what it will and will not advise on
• Crisis pathway integration — Integrate safe messaging guidelines and crisis resources into health AI systems
• Mandatory escalation — Specific query types (medication changes, symptoms suggesting emergency) automatically escalate to human clinicians
• Version control for medical knowledge — Track when clinical guidelines change and update AI systems accordingly

—