Where AI breaks privacy assumptions GDPR was built on

What this risk is

AI systems that memorize and leak sensitive personal data, or infer private information about individuals without their consent. This includes training data memorization, model inversion attacks, membership inference attacks, and unauthorized inference of sensitive attributes from seemingly innocuous inputs.

Unexpected or unauthorized sharing of data can compromise user privacy expectations, enable identity theft, or result in loss of confidential intellectual property.

How it occurs · Mechanisms

Causal profile: Mixed (AI and human) · Mixed intentionality · Post-deployment

• Training data memorization — Large language models can memorize and reproduce verbatim snippets of training data, including personal information, API keys, or confidential documents
• Membership inference attacks — Adversaries can determine whether a specific individual’s data was included in the training set
• Model inversion attacks — Reconstructing training data from model outputs and gradients
• Attribute inference — Inferring sensitive attributes (health status, political views, sexuality) from seemingly unrelated inputs
• RAG system leakage — Retrieval-augmented systems can inadvertently expose confidential documents in their knowledge base

—

Real-world incidents

Samsung Source Code Leak via ChatGPT (2023)

Samsung employees inadvertently uploaded proprietary source code and internal meeting notes to ChatGPT. The data became part of OpenAI’s training pipeline. Samsung subsequently banned the use of external AI tools.

GPT-2 Training Data Extraction (Carlini et al., 2021)

Researchers demonstrated that GPT-2 could be prompted to reproduce near-verbatim training data, including personal names, phone numbers, email addresses, and physical addresses that appeared in training data.

Healthcare AI and Patient Data Exposure

Multiple hospital systems using AI for diagnostic support have faced investigations after discovering AI systems could be used to reconstruct patient-identifiable information from model outputs.

Mitigations · Governance

Technical Controls

• Differential privacy — Add mathematical noise during training to limit what models can memorize about specific data points
• Data minimization — Do not include personal data in training data unless strictly necessary
• Anonymization and pseudonymization — Apply GDPR-compliant data processing before training
• Output filtering — Monitor and filter model outputs for PII and confidential information
• Access controls on RAG systems — Ensure retrieval systems respect data access permissions

Organizational Controls

• Data governance policies — Define what data can be used for AI training and under what conditions
• Employee training — Prevent accidental data leakage through AI tools (as in the Samsung case)
• Vendor assessment — Evaluate how AI vendors handle training data, especially for foundation model providers
• Privacy impact assessments — Conduct DPIAs before deploying AI systems that process personal data

—