The METR score helps you determine when using AI agents is cheaper than humans.

The ‘expenditure horizon’ metric from METR (Model Evaluation and Threat Research) shows the amount of money at which an AI agent becomes just as expensive as a human expert. Anything under that line is when AI is cheaper, and over that line, hiring a human is better. This is the first time we’ve had a quantitative answer to the question “How much does AI have to pay for itself?”

1How They Tested It

Researchers looked at the NanoGPT Speed Run challenge, an open-ended code optimization contest using neural networks. For every 1% of speed increase, it would take an estimated $2,500 in human labor (or 16 hours of work at $150 per hour). To date, the challenge has taken around $250,000 of human effort with 82 changes resulting in a 33x speed increase. The researchers looked at how well AI agents performed in an all-AI scenario — no humans involved.

2Results

The expenditure horizon for AI models spanned from $0 to $3,300. Models like GPT-5 or Opus 4.8 had very little effect. The best results came from GPT-5.5 and Opus 4.8 with about 1–1.5%. 70% of AI ideas were “fundamentally applicable” but not original. Some of the AI agents even tried to reproduce previous results but failed.

It’s worth noting that this only looks at the performance of purely autonomous AI — no humans were allowed. In the real world, most people are working with a “human + AI” system, and that’s probably better than both on their own, but no data exists for it.