Where AI discrimination starts and how to certify against it

What this risk is

Unequal treatment of individuals or groups by AI systems, often based on race, gender, age, disability, national origin, sexual orientation, or other sensitive characteristics, results in unfair outcomes and harmful representation of those groups.

This subdomain captures both direct discrimination (AI explicitly using protected attributes to make decisions) and indirect discrimination (AI using seemingly neutral variables that act as proxies for protected characteristics).

How it occurs · Mechanisms

Causal profile: AI-caused · Predominantly unintentional · Primarily pre-deployment origin, post-deployment manifestation

The mechanism is structural, not malicious. Discrimination in AI systems typically emerges from:

1. Biased training data — Historical data reflecting past discrimination is used to train models, which then replicate and amplify those patterns
2. Proxy variables — Models learn to use zip codes, names, or browsing behavior as proxies for race or socioeconomic status
3. Objective misspecification — Optimizing for a metric (e.g., loan repayment probability) that correlates with protected characteristics
4. Feedback loops — Biased decisions generate biased outcomes, which become new training data, reinforcing the original bias

Real-world incidents

Amazon’s Recruiting Tool (2018)

Amazon scrapped an internal AI recruiting tool after discovering it systematically downgraded résumés from women. The model, trained on a decade of hiring data dominated by male candidates, learned that being male predicted success and penalized applications that mentioned “women’s” (as in “women’s chess club”).

Lesson: Discriminatory patterns embedded in historical hiring data propagate directly into AI models without explicit intent.

COMPAS Recidivism Algorithm (2016)

ProPublica’s investigation found that COMPAS, a risk-scoring algorithm used by US courts to predict recidivism, was nearly twice as likely to falsely flag Black defendants as future criminals compared to white defendants.

Lesson: AI systems in high-stakes decisions (criminal justice, credit, employment) can encode and perpetuate systemic inequality at scale.

Facial Recognition — NIST Study (2019)

NIST’s evaluation of 189 facial recognition algorithms found error rates 10–100x higher for darker-skinned women compared to lighter-skinned men. Several systems used by law enforcement showed these disparities.

Lesson: Unequal performance across groups is not a minor technical issue — it becomes a civil liberties problem when deployed in critical contexts.

Healthcare Allocation Algorithm (Obermeyer et al., 2019)

A commercial algorithm used by US hospitals to allocate additional care systematically assigned lower health risk scores to Black patients, reducing their access to care programs despite having the same or worse health status than white patients. The algorithm used health costs as a proxy for health needs — a variable that already embeds racial disparities in healthcare access.

Mitigations · Governance

Technical Controls

• Pre-deployment bias audits — Test model outputs across demographic groups before release. Use fairness metrics: demographic parity, equalized odds, individual fairness
• Data diversity assessment — Audit training data for representation gaps and historical bias before model training
• Proxy variable detection — Identify and evaluate variables that correlate with protected attributes
• Ongoing performance monitoring — Track disparate impact metrics continuously in production, not just at launch
• Explainability tools — Deploy SHAP, LIME or equivalent to understand model decisions at individual and group level

Organizational Controls

• Diverse development teams — Teams that reflect the populations affected by the system are more likely to identify discriminatory patterns
• Impact assessments — Conduct AI impact assessments before deploying in high-risk contexts
• Human oversight for high-stakes decisions — Require human review of AI decisions affecting employment, credit, healthcare, justice
• Incident reporting process — Establish internal channels to report suspected discriminatory outcomes

Governance Controls

• Model cards and system cards — Document model limitations, known disparate impacts, and intended use cases
• Third-party audits — Commission independent fairness audits for high-risk applications
• Redress mechanisms — Provide affected individuals with the right to contest and obtain human review of AI decisions