AI Access makes “I don’t know” virtually obsolete.

In a paper published on PsyArXiv on September 26th, researchers Marcoccia, Quattrociocchi, and Capraro conducted 5 separate studies with a combined sample size of 3,132. They presented participants with various movie trivia questions, many of which the Step 3.5 Flash model would answer incorrectly. When not using AI, participants correctly responded “I don’t know” 36–44% of the time. However, once given access to the model, they only responded “I don’t know” 3–6% of the time. Their confidence in their response increased from 29.6 to 75.9 on a 100-point scale, but their accuracy decreased from 27.6% to 10%.

I’m not too surprised by these results. In our experience building agents, we often notice this happening. It’s easy for an agent to give a confident response without doing much verification. This is one reason why we always have a human check over our agent’s responses, particularly if the prompt is ambiguous. It’s easiest for a model to make errors in these situations. Being confident and being correct are not the same thing, and this study is yet another reminder of that.