Berkeley: Agent Frameworks Drive Cost, Not Quality

UC Berkeley experimented with 21 model–framework pairs on two datasets: SWE-bench Lite and Terminal-Bench 2.0. They discovered that some model–framework pairs perform equally well (±2%) while having very different costs (up to 5x). For example, on Fable 5.1, Claude Code is 2x more expensive than the cheapest framework (Pi), yet its success rate is only 1.1% worse.

Pi is a framework with 4 tools (read, write, edit, bash) and reaches the Pareto frontier on both datasets. Why does Claude Code cost so much more than Pi? Because it produces 10x more tokens at the start of every session. If you’re doing one task with a clear spec, you can probably get away with using a simpler framework and saving some money. But if you’re engaged in a long session with lots of project context, you might want to spend a little extra.