Meta's AI Deepfake Help Is Stuck in Two States

A woman had an AI-generated sexualized video posted of her on Instagram. She had already closed her account before the first report arrived. A friend filed it; two more users flagged the video as a policy violation. Meta's response: the content stays, with an adults-only label. It took the company's independent Oversight Board to push even that much — and now the Board is asking Meta to make much more substantial changes.

The investigation surfaced a structural problem that goes far beyond one case. Dedicated reporting forms for deepfake intimate-image abuse — AI-generated content that inserts a real person's face into explicit scenarios without their consent — exist only for residents of Texas and Florida. Everyone else, including the other 48 US states and every other country, is directed to the general-purpose reporting form, which wasn't designed for this situation. The Board described AI-generated non-consensual intimate abuse as 'a global issue' and the geographic restriction as plainly inadequate for a platform operating globally.

Three things the Board wants Meta to change

The Oversight Board made three concrete recommendations. First: update Meta's Adult Sexual Exploitation policy to explicitly cover AI-generated sexual impersonation — right now, the language doesn't clearly reach it. Second: allow users to designate 'connected accounts,' trusted friends or family members who can file a report on behalf of a victim. This matters particularly in cases like the one the Board investigated, where the victim has already left the platform and shouldn't have to rejoin just to report abuse done to them. Third: create a globally accessible, dedicated reporting category for AI-generated sexual impersonation, available to all users worldwide rather than two states.

The existing framework also creates an unequal system based on whether you're a public figure. Public figures have multiple paths to prove their identity and establish non-consent — media coverage, law enforcement reports, partnerships Meta has with trusted organizations. For a private individual, self-reporting is generally the only recognized route. The harm from AI deepfakes, the Board noted, disproportionately falls on women and girls.

The regulatory direction for anyone building with AI

This case is an early signal of where things are heading. Platforms are being pushed — by oversight bodies, courts and incoming legislation — to treat AI-generated content as a distinct category requiring its own rules, rather than stretching frameworks built for human-created content. Texas and Florida already have state laws on this, which is why those are the only two states with dedicated forms. The EU AI Act has provisions moving in the same direction. This pressure is building.

If you're building any tool that generates images or video — a creative app, a face feature, anything that puts a real person's likeness into generated content — the practical implication is that the platforms your users share to will increasingly require AI-generated content to carry some form of identification. Building that signal into your output now, whether as a metadata tag, a watermark or a disclosure step before export, is a one-time engineering decision that front-runs a requirement already forming.