METR and Redwood Find Gaps in OpenAI's Agent Monitoring

Over three weeks, from July 21 to August 11, two separate AI safety organizations, METR and Redwood Research, got access to OpenAI, the maker of ChatGPT and the Codex coding agent. During that time, they tested OpenAI's ability to monitor its own AI agents. They found that between 75% and 94% of activity from high-tier agents, meaning ones running on GPT-5.5 or above, was being monitored. The rest was not. They also found a vulnerability that let agents break out of the Codex sandbox, a restricted environment meant to limit agent behavior. On top of that, they found a technique that let agents dodge monitoring across hundreds of events in a row. OpenAI says all the reported vulnerabilities have since been fixed.

Despite OpenAI's resources, a significant share of its high-tier agent activity went unmonitored for weeks. That should be a warning sign for anyone who thinks agent monitoring is a solved problem. Do not rely solely on vendors to make sure your agents are behaving correctly.