Commentary: Some Anthropic engineers think AI might end us. Why race ahead?

A CONTRADICTION

At the heart of these developments is a contradiction best encapsulated at Anthropic. It was founded to be more safety-conscious than OpenAI, yet it clings to that mission statement while building ever more powerful agents – even if that same technology could also, potentially, lead to mass destruction. Commercial imperatives also feed its desire to keep advancing: Anthropic is preparing for an initial public offering that could value it at US$2 trillion.

Coxon’s big concern was about recursive self-improvement (RSI), or AI systems that can develop by themselves their own more capable successors. He lamented in one television interview about how strange it was to do so little programming himself when building frontier models. So much of his work was now being outsourced to other AI systems.

Anthropic already addressed concerns about RSI in a June blog, revealing the unease of its engineers. “On days where everything works well,” one said, “I can’t help but think nothing I do matters, everything is automated and better and faster than I ever will be.” Slowing down RSI would be good, the company added, but if that “lets the least cautious actors catch up technologically, it could leave everyone less safe.”

The thinking goes that whoever reaches RSI first will experience an “intelligence explosion”, says Amjad Masad, chief executive officer of California-based Replit. That would be such a meaningful leap in capability that the first model to get there “might be the last”.