Anthropic Warns It Is Close to a Self-Acceleration Threshold

Anthropic is the AI startup behind the Claude family of models. On Monday, November 2, it updated its IPO filing. In that filing, Anthropic admits it may soon reach a point it calls "dramatically accelerated AI-driven progress" (its own phrase). Another name for this is RSI, recursive self-improvement, when an AI gets good enough at building AI that it starts rapidly improving itself. Anthropic says none of its current models have reached RSI yet. But it also admits that in the past, it only noticed RSI after the fact. Even so, the company says it intends to keep humans in control of how future models get built.

Here is where it gets interesting. Under Anthropic's own Responsible Scaling Policy, the company might not react to RSI in time. And Anthropic admits it does not know how to guarantee that humans will stay in control of model development once models start building models better than themselves.