AI Chatbots Read X-Rays — and Get It Wrong With Confidence

The RadLE 2.0 benchmark, created by the CRASH Lab at Ashoka University (India), evaluated 16 different AI models (such as Claude Fable 5, Gemini 3 Pro, Muse Spark 1.1, Grok 4.5) on a set of 200 x-ray images. It measures three things: diagnostic accuracy, confidence, and awareness of being outside the model’s expertise.

The best performing AI got 758/2000 points, whereas a human radiologist achieved 988.7. While the headline result isn’t the difference in raw scores, many highly accurate models performed worse than their less accurate counterparts due to overconfidence in incorrect answers. This is especially true for models that give wrong answers with high confidence, which RadLE 2.0 strongly discourages. Almost all open source and domain specific models tried everything, and paid dearly for it.

“Many models would have scored significantly higher had they abstained more often.” Claude Fable 5 performed best in the “calibrated confidence” category, and Gemini 3 Pro achieved almost human performance, but none of them could keep up with the human radiologist. As the authors emphasize, AI models need to be able to say “I don’t know, ask a human” before we let them make independent decisions.