With AI Access, People Almost Stop Saying "I Don't Know"

Researchers Marcoccia, Quattrociocchi, and Capraro surveyed a total of 3,132 people across five experiments. The paper appeared on PsyArXiv on September 26. Participants were asked to recall small details from movies, and the Step 3.5 Flash model often got these questions wrong. Without AI, participants honestly said "I don't know" in 36–44% of cases. With access to the model, that share dropped to 3–6%. Confidence in their answers rose from 29.6 to 75.9 out of 100. Accuracy, on the other hand, fell from 27.6% to 10%.

This does not surprise me. In my own work with agents, I see something similar all the time. An answer can sound confident with no real check behind it. That is why, on our team, a person always reviews an agent's output, especially when the task is vaguely worded. That is exactly where a model's mistakes are hardest to spot. A confident tone and actual accuracy are simply two different things, and this result just proves it again.