When AI fraud scales faster than the controls designed to stop it

What this risk is

Using AI systems to gain personal advantage over others through cheating, fraud, scams, blackmail, or targeted manipulation of beliefs or behavior. Examples include AI-facilitated plagiarism and academic fraud, impersonating a trusted or fake individual for financial gain, and creating humiliating or sexual imagery of real people.

How it occurs · Mechanisms

Causal profile: Human-caused · Intentional · Post-deployment

• Voice cloning for fraud — AI clones the voice of a family member or executive to request emergency wire transfers
• Deepfake impersonation — Video deepfakes impersonate executives in video calls to authorize financial transactions
• AI-enhanced phishing — LLMs generate hyper-personalized phishing emails using scraped personal information
• Non-consensual intimate imagery (NCII) — AI generates realistic sexual imagery of real people without consent
• Academic fraud — AI generates essays, assignments, and research papers for submission as original work
• Romance scams — AI powers convincing fake romantic personas to extract money from victims

—

Real-world incidents

CEO Voice Clone Fraud (À920,000 Loss, 2019)

Criminals used AI voice cloning to impersonate a CEO and instruct a subsidiary company executive to transfer £243,000 to a Hungarian supplier. The voice clone was convincing enough that the executive complied.

Deepfake CFO Video Call (HK$200M, 2024)

A finance worker in Hong Kong was tricked into transferring HK$200 million (~$25M USD) after attending a video call with what appeared to be his company’s CFO and other executives — all of whom were deepfakes.

FTC Report on AI-Enhanced Scams (2024)

The US Federal Trade Commission reported a significant increase in AI-powered scams, particularly voice cloning for grandparent scams and romance fraud, with total losses in the billions.

Non-Consensual Intimate Imagery (NCII)

Multiple high-profile cases of AI-generated NCII targeting celebrities and private individuals. In 2024, Taylor Swift deepfakes spread widely on social media before platforms could remove them, prompting calls for federal legislation.

Mitigations · Governance

• Voice and video authentication — Deploy liveness detection and authentication for high-stakes communications
• Out-of-band verification — Require phone callbacks or in-person confirmation for large financial transactions
• Employee training — Train staff to recognize AI-generated voice and video and to verify through secondary channels
• Platform-level NCII detection — Platforms implement hash-matching for known NCII and prioritize takedown
• Academic integrity tools — Deploy AI detection alongside policy frameworks for academic AI use
• Fraud detection systems — AI-powered anomaly detection to identify unusual transaction patterns

—