← All news·2026-11-15·3 min read

Arena Proposes Judging AI-Written Apps by Actually Running Them

Researchers from Arena and three universities propose to evaluate apps written by AI with an agent that uses a computer, not by reading the code text.

aibenchmarkarenaagentsresearch

Researchers at Arena, a company that runs a public leaderboard of AI models, have proposed a new way to grade apps written by AI. Scientists from Carnegie Mellon, UC Berkeley and UCLA worked with them. The judge does not read the code. It is a computer-use agent, an AI program that clicks, types and watches the screen the way a person does. In the authors' test, AI models fixed broken apps more often when they checked their work this way. They repaired 75.5% of the problems, against 55.4% when they checked only the code.

The idea addresses a familiar gap. Code can look correct while a button on the screen does nothing. An agent that actually uses the app is more likely to notice. In the study, it caught 71.1% of broken features. A judge that only read the code caught 63.0%. For anyone choosing an AI tool to write software, this kind of test is closer to how people actually use what gets built.

The authors name the main limit themselves. Today's models are too slow to interact in real time, which matters for things like games. All the numbers come from the researchers' own paper, and nobody has checked them independently yet. The method is worth watching as a new way to rank coding models, but for now it is a research result, not a standard.

Source: arena.ai

Free course

Stop reading about AI — start building with it

The free Claude Code course: your first site, tool or game — no coding. No upsells, no cross-sells — nothing to buy here.

Start free →
EAEvgenii Arsentev

Author

Evgenii Arsentev

PhD · Founder & CEO, Dad4Kids