Meta’s AI-Powered Deepfake Detection Tool Comes in Two Modes

A woman was shown a deepfake video featuring her in a sexual context on Instagram. By the time her friend reported the incident, she had deleted her account. Two other users reported the video too, but Meta said it wouldn’t take action, and instead placed a “mature audience” warning on the post. Only after intervention from Meta’s independent Oversight Board did they agree to do so. But the Board wants Meta to go further.

During the case review, the Board discovered a systemic flaw. Meta’s reporting process for AI-generated intimate image abuse (using artificial intelligence to superimpose someone’s face onto explicit material without their permission) is only available via specialized forms in Texas and Florida. Everyone else — that means everyone outside of Texas and Florida, which includes all 48 other states in the US and all countries around the world — must use Meta’s generic reporting form, which isn’t tailored to these kinds of cases. The Board called AI-generated non-consensual intimate abuse a “global problem,” and said Meta’s limited approach was “clearly insufficient” for a global service.

1Three things the Board wants Meta to change

In response, the Board issued three clear proposals. First, it recommended that Meta revise its Adult Sexual Exploitation policy to specifically address AI-generated sexual impersonation (currently, it’s not specific enough). Second, it recommended that users be able to specify “connected accounts” (friends or family members they trust) who could report abuse on their behalf. This would be especially useful when someone like the woman in this case leaves the platform and doesn’t want to return to report abuse. Finally, the Board suggested that Meta introduce a global reporting category for AI-generated sexual impersonation, so that users everywhere can use it.

Currently, the system favors public figures. They have several ways to verify their identity and demonstrate non-consent (news articles, police reports, collaborations with trusted entities). In contrast, private citizens usually need to self-report. And as the Board pointed out, “the negative effects of AI deepfakes fall especially hard on women and girls.”

2The regulatory direction for anyone building with AI

This is just the beginning. As oversight bodies, judiciaries and new laws force platforms to consider AI-generated content separately from other types, they’ll need to develop separate policies. Texas and Florida already have laws regulating this kind of activity, so those are the only places where Meta offers specialized reporting forms. The EU AI Act also includes measures that point in this direction. The trend is growing.

So if you’re making a product that creates images or videos (whether it’s a creativity tool, a facial feature or anything else that incorporates a real person’s image), this means that the platforms where your users will upload content will likely soon start demanding that AI-generated content include some kind of identifier. If you want to stay ahead of the curve, consider adding a way to include this information in your generated content — either as metadata, a watermark or a prompt before exporting — during development. It’s a one-off change that lets you futureproof your product against emerging requirements.