A prototype of a self-improving AI that optimizes safety is demonstrated by Anthropic.

Anthropic researcher Chen Yue-Han (via) showcased a system called the Automated Alignment Researcher. The system reads papers, suggests ways to modify AI models for better behavior, then runs 30 minute training sessions. When it finds something that works, it keeps it. When it doesn’t, it throws it away. It’s the first demonstration of a system which can iteratively improve itself.

1Results: All 10 Benchmarks, 6 Hours, $4/hr

It passed all 10 benchmarks on “unwanted behaviors” and improved scores in every single benchmark, without reducing general scores. It did this in 6 hours, costing around $4/hr. In contrast, human researchers cost $150/hr. That’s a 37x price difference.

2What This Changes

This could change everything about the economics of AI alignment research, if it can be scaled. Instead of manually trying hypotheses, you would have an automated feedback loop. That increases speed by an order of magnitude and reduces costs.