Data Poisoning — Adversarial Attack on AI Training Data

Definition

Data poisoning is an adversarial attack on AI systems in which an attacker manipulates the training data to cause the model to learn incorrect patterns, produce specific erroneous outputs, or behave differently in targeted scenarios. Unlike prompt injection — which attacks the model at inference time — data poisoning attacks the model at training time, corrupting its learned behavior before deployment.

Two primary poisoning strategies exist. Targeted poisoning introduces malicious samples designed to cause the model to misclassify specific inputs or behave in attacker-desired ways on trigger conditions, while performing normally on other inputs (backdoor attacks). Indiscriminate poisoning degrades overall model performance by introducing corrupted or mislabeled data throughout the training set, reducing the model’s accuracy and reliability without a specific trigger.

Data poisoning is particularly concerning in systems that use third-party datasets, web-scraped training data, federated learning architectures, or continuous learning from user interactions — contexts where the full training data pipeline is not under the organization’s control.

Why it matters operationally

Data poisoning matters because it is silent by design. A backdoor attack leaves no visible trace in the model’s normal performance — the model passes all standard validation tests and appears to function correctly. The malicious behavior activates only when the attacker’s trigger condition is present, which may not appear in any test dataset. Organizations can deploy poisoned models, certify them, and operate them in production without any indication of compromise until the trigger is exploited.

For high-risk AI systems, this is not a theoretical concern. AI systems used in security-sensitive contexts — fraud detection, threat assessment, access control, medical diagnosis — are high-value targets for data poisoning attacks. Organizations that source training data from third parties, use publicly available datasets, or deploy continuously learning models that incorporate user feedback have elevated exposure.

Regulatory framework

Framework	Application
EU AI Act	High-risk systems must be robust and resilient against attempts to modify their use or performance by exploiting vulnerabilities. Protection of training pipeline integrity is an implicit requirement of the robustness requirement.
ISO/IEC 42001	Annex A controls on training data include verification of data integrity and provenance, applicable to poisoning detection and prevention.
ISO/IEC 27001	The information security management system covers data asset integrity, including training datasets as critical information assets.
NIST AI RMF	The Measure function includes evaluation of robustness against adversarial attacks, including training data poisoning.

How Zertia evaluates it

Zertia evaluates data poisoning exposure through the AI Model Audit and High-Risk AI Systems Audit. The assessment examines: training data provenance and integrity controls; data pipeline security (who has access to training data, what controls prevent unauthorized modification); third-party dataset due diligence processes; testing for anomalous model behavior on out-of-distribution inputs; and whether continuous learning architectures have safeguards against poisoning through user feedback.

[AI Model Audit] · High-Risk AI Systems Audit