Arena Proposes Judging AI-Written Apps by Actually Running Them
Researchers from Arena and three universities propose to evaluate apps written by AI with an agent that uses a computer, not by reading the code text.
Researchers at Arena, a company that runs a public leaderboard of AI models, have proposed a new way to grade apps written by AI. Scientists from Carnegie Mellon, UC Berkeley and UCLA worked with them. The judge does not read the code. It is a computer-use agent, an AI program that clicks, types and watches the screen the way a person does. In the authors' test, AI models fixed broken apps more often when they checked their work this way. They repaired 75.5% of the problems, against 55.4% when they checked only the code.
The idea addresses a familiar gap. Code can look correct while a button on the screen does nothing. An agent that actually uses the app is more likely to notice. In the study, it caught 71.1% of broken features. A judge that only read the code caught 63.0%. For anyone choosing an AI tool to write software, this kind of test is closer to how people actually use what gets built.
The authors name the main limit themselves. Today's models are too slow to interact in real time, which matters for things like games. All the numbers come from the researchers' own paper, and nobody has checked them independently yet. The method is worth watching as a new way to rank coding models, but for now it is a research result, not a standard.
Source: arena.ai
Free course
Stop reading about AI — start building with it
The free Claude Code course: your first site, tool or game — no coding. No upsells, no cross-sells — nothing to buy here.
Start free →▌ Related guides

Author
Evgenii Arsentev
PhD · Founder & CEO, Dad4Kids
Articles · Latest articles