When clinical AI inherits the inequities of the data it learns from

What this risk is

AI systems used in clinical settings producing systematically worse outcomes for patients from certain demographic groups — through biased training data, flawed objective functions, or proxy variables that encode historical healthcare inequities. Unlike other discrimination domains, healthcare AI bias can directly cause physical harm.

How it occurs · Mechanisms

Healthcare data reflects decades of structural inequities:

• Minority groups historically received less preventive care, creating data gaps
• Clinical trials historically underrepresented women, minorities, and elderly patients
• Healthcare costs — a common AI training label — are not a neutral proxy for health need; they reflect access to care, not just health status
• Physiological measurement tools (pulse oximeters, dermatology imaging) were designed and validated on lighter-skinned patients

AI trained on this data does not inherit healthcare equity. It inherits healthcare inequity.

Real-world incidents

Healthcare Cost Algorithm (Obermeyer et al., Science 2019)

A commercial algorithm used by US hospitals to identify high-risk patients who would benefit from care management programs used healthcare costs as a proxy for health need. Because Black patients historically had less access to care, they spent less on healthcare for equivalent health conditions. The algorithm systematically underestimated Black patient health risk, allocating fewer care management resources to Black patients who were sicker than white patients with the same risk score.

Scale: The algorithm was used for approximately 200 million people per year in the US.

Dermatology AI (Multiple studies, 2018–2023)

Multiple skin cancer detection AI systems showed significantly lower accuracy for darker skin tones — some performing at dermatologist level for lighter skin while performing at below-random for darker skin. Training datasets were predominantly composed of images from lighter-skinned patients.

Pulse Oximetry and COVID-19 (Sjoding et al., NEJM 2020)

Pulse oximeters — standard clinical monitoring devices increasingly integrated with AI health systems — were found to overestimate oxygen saturation in Black patients at nearly three times the rate as white patients. This led to delayed treatment escalation and contributed to worse COVID-19 outcomes.

Sepsis Prediction Model Performance (Wong et al., 2021)

A widely-deployed sepsis prediction AI was found to perform significantly worse for Black, Hispanic, and female patients, with meaningful differences in sensitivity that would translate to delayed sepsis detection for these groups.

Mitigations · Governance

• Disaggregated performance reporting — Mandatory reporting of AI performance stratified by race, sex, age, and socioeconomic status
• Diverse clinical trials — Validate AI systems in patient populations that reflect actual deployment populations
• Proxy variable audit — Evaluate AI features for encoded proxies of race or socioeconomic status
• Health equity impact assessment — Assess potential equity impacts before deploying clinical AI
• Ongoing equity monitoring — Post-deployment monitoring for differential outcomes by demographic group

—