AI Judges of Research Novelty Turn Out Unreliable

The Allen Institute for AI (AI2) ran an experiment to check whether language models can reliably tell if an idea is original or not. The researchers compared ideas published at ICLR 2026, one of the major AI conferences, with ideas generated by AI. They asked each model about the same idea twice, phrasing the question differently each time. OpenAI's GPT-5.4 dropped from 93.5% accuracy to 40.9%, which is worse than random guessing. Another model, Anthropic's Claude Opus 4.6, shifted its verdict by 21 percentage points.

You cannot trust a single language model to decide whether an idea is original. The answer depends on how the question is phrased, not on the content of the ideas themselves. So if we want to automate this task, relying on just one language model is not a good idea. But we found that sending a simple prompt to Claude Opus 4.6 three times and taking the majority vote beats two other tools built specifically for this job: Idea Novelty Checker and Sakana AI's AI Scientist, an AI lab based in Tokyo. On top of that, Claude Opus 4.6 turned out to be much cheaper: $2.59 versus $30.66 and $82.05.