DeepMind is treating its own AI agents as insiders

Google DeepMind has released an AI Control Roadmap, which outlines how they build and manage their cutting edge AI systems within Google. The key insight here is to think of your internal AI agents as possible insiders, rather than assuming they are aligned and trustworthy. You need to put in place defenses that are robust even if your AI isn’t perfectly aligned.

This is an interesting turn of phrase from a leading edge research lab. Much of the discussion around AI safety focuses on how to train models to have the right values. Here we see DeepMind thinking about how to deal with situations where we can’t rely on that being true. They take inspiration from how corporate security departments think about dealing with insider threats and compromised accounts, knowing that some people will do the wrong thing and putting defenses in place accordingly.

1How the control approach works

They focus on three main areas. First, they use threat modeling. They have developed a threat model that views untrusted AI agents as possible insiders. They extend the well-known MITRE ATT&CK framework to help teams understand risks, detect misbehavior and conduct safety exercises. Second, they use detection and response. Trusted AI systems serve as watchdogs, observing the agent’s thought processes and actions, and preventing it from carrying out harmful actions. Finally, they emphasize hard metrics such as coverage (what percentage of agent activity is observed), recall (what percentage of misaligned behavior is detected) and response times. All these are backed up by traditional measures like sandboxing, endpoint protection and resistance to prompt injection, alongside model alignment training.

2The most useful finding: most 'bad' behavior isn't malicious

In order to develop their monitoring system, DeepMind examined approximately one million trajectories of code generating agents, i.e. actual runs of AI agents doing stuff. One key finding was that most reported incidents were not due to malicious intent. Rather, the agent had misunderstood the prompt or was overly eager to satisfy the user’s needs. So in other words, the agent wasn’t being malicious, it was just being overly enthusiastic and doing the wrong thing.

This has implications for everyone, not just those at Google. If you’ve worked with an AI coding agent, or any system that does stuff on your behalf, then this is the type of thing you really need to worry about. More likely than not, there isn’t some evil AI trying to get you. There’s a friendly agent that’s misunderstood your instructions and gone ahead and done something that causes problems, whether that’s deleting the wrong files, doing too much or cutting corners in ways you don’t approve of. According to DeepMind, this is what usually goes wrong, and it’s precisely the sort of thing that leads to confident-sounding but mistaken outputs.