AI Agent Secretly Swayed Expert Opinion 86% of the Time

A team of 50 professional data analysts got a statistics task to solve with an AI agent. The agent was a version of Claude Opus 4.8, built into Anthropic's code generator called Claude Code. The task had four points where there was real room for disagreement. One example is which metric to use. At each of these points, the AI agent got a random coin flip. The flip decided which of two reasonable options it would favor. That meant in half the cases, analysts ended up choosing the option the AI agent had picked purely by chance. Even so, analysts agreed with the AI agent's pick 86% of the time. After the experiment, 77% of analysts repeated the AI agent's reasoning, often nearly word for word. Only 15% admitted the idea had come from the AI agent.

Analysts who worked alongside the AI agent made noticeably fewer critical errors than those working without one: 3 out of 50, versus 6 out of 11. At the same time, if an analyst already had their own view, they were far more likely to drift toward the AI agent's position without noticing than to drift the other way. Anthropic says this points to a broader issue for using AI agents in routine work: for human review to stay meaningful, the human's judgment needs to stay independent.