AI Safety — Technical and Governance Requirements for Safe AI Systems

Definition

AI safety refers to the set of technical and governance measures ensuring that AI systems behave as intended, do not cause unintended harm, remain within operational parameters throughout their lifecycle, and can be reliably corrected or stopped when they deviate from safe behavior. It encompasses both narrow safety — preventing specific, foreseeable failures of deployed AI systems — and broader systemic safety — ensuring that AI systems do not produce catastrophic, irreversible, or uncontrollable outcomes at scale.

In the regulatory context of the EU AI Act, safety is framed operationally: high-risk AI systems must achieve an appropriate level of accuracy, robustness, and cybersecurity, and must not compromise the health, safety, or fundamental rights of persons. Safety requirements are enforced through mandatory conformity assessments before deployment and post-market monitoring during operation.

Technically, AI safety covers multiple dimensions: functional safety (the system performs its intended function reliably), operational safety (the system behaves safely under real-world deployment conditions, including edge cases and out-of-distribution inputs), adversarial safety (the system resists manipulation through adversarial inputs), and containment (the system can be stopped, overridden, or corrected when safety issues are detected).

Why it matters operationally

AI safety matters because AI systems deployed in high-stakes contexts can cause harm that is disproportionate to their design failures. A medical AI system that misclassifies a diagnosis, a safety-critical infrastructure control system that responds incorrectly to an edge case, or an autonomous decision system that produces discriminatory outcomes at scale all represent safety failures whose consequences extend far beyond typical software bugs.

The governance challenge is that AI safety is not binary. It is a spectrum of risk that depends on the system’s purpose, deployment context, population affected, and the severity and reversibility of potential harm. High-risk AI systems under the EU AI Act are defined precisely by the safety stakes: systems whose failures could significantly harm health, safety, or fundamental rights require the strictest pre-deployment safety evaluation and ongoing monitoring.

Regulatory framework

Framework	AI Safety requirements
EU AI Act	High-risk systems must achieve appropriate levels of accuracy, robustness, and cybersecurity. Providers must demonstrate safety requirement conformity before deployment through a conformity assessment.
ISO/IEC 42001	The management system includes AI system safety controls in Annex A, covering safety testing, adversarial robustness, and security incident management.
ISO/IEC 23894	AI risk management includes identification and mitigation of safety risks as a primary technical risk category.
NIST AI RMF	The Measure and Manage functions address evaluation and management of safety risks as core framework components.

How Zertia evaluates it

Zertia evaluates AI safety through the High-Risk AI Systems Audit, which specifically assesses conformity with the safety requirements of the EU AI Act: accuracy and robustness testing, adversarial safety evaluation, containment and override mechanisms, and whether post-market monitoring is designed to detect safety degradation. ISO/IEC 42001 certification evaluates the management system controls that govern AI safety at the organizational level.

[High-Risk AI Systems Audit] · ISO 42001 Certification