Anthropic calls for coordinated pause as it warns AI could outpace human control

WASHINGTON: Artificial intelligence firm Anthropic has urged leading AI developers to consider a coordinated and verifiable slowdown in building more advanced systems, saying rapid progress could soon produce machines capable of improving themselves faster than institutions can manage the risks.

In a blog post published on Thursday, the company behind Claude said the amount of work AI systems can perform independently has been doubling about every four months and is moving toward what it described as recursive self-improvement, a stage where systems can upgrade themselves without human intervention. Anthropic co-founder Jack Clark and Anthropic Institute lead Marina Favaro said society should prepare for that possibility even if it has not yet arrived.

The company said stronger safeguards would become increasingly important if AI systems gained the ability to design their own successors. It argued that a pause in development could create space for society to grapple with the consequences of such a shift.

Clark and Favaro wrote that the threshold had not yet been reached and was not certain to be reached, but warned that institutions may be less prepared than the pace of development requires.

Anthropic said in the post:

"If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behaviour all grow much more important,"

The company added that a pause would allow society to "deal with its immense implications"

Clark and Favaro also wrote:

"We are not there yet, and recursive self-improvement is not inevitable. But it could come sooner than most institutions are prepared for,"

Concerns that increasingly capable AI systems could escape meaningful human control and inflict wider social harm have grown alongside advances in the technology. Earlier this year, Anthropic’s own Mythos model drew strong attention in sectors including banking and software because of its ability to identify weaknesses in existing code.

Regulation and industry response

Regulation has lagged, particularly in the United States where many of the top AI laboratories are based. Earlier this week, the Trump administration issued an executive order that placed responsibility on the companies themselves, asking them to voluntarily submit their most advanced models for US government cybersecurity testing before releasing them publicly.

Calls for a pause in AI development are not new. In 2023, Elon Musk, who owns xAI, backed a Future of Life Institute initiative seeking a six-month halt in AI work to create time for safety protections, but the effort achieved little.

Anthropic has for years presented itself as a company focused on AI safety. Earlier this year, it declined to allow the US military to use its models for domestic surveillance and fully autonomous weapons. Decision triggered backlash from the government, which placed the company on a national security blacklist due to take effect later in 2026. Reuters reported on Friday that the standoff appeared to be easing in some parts of the US government.

At the same time, Anthropic has continued to introduce more powerful systems. In February, it softened an important safety commitment, saying it would no longer withhold potentially dangerous AI if competing firms were close to reaching similar capability levels.

Anthropic was recently valued at $965 billion in a major funding round and confidentially filed for a US initial public offering on Monday, putting it ahead of OpenAI in valuation and in the contest to secure fresh funding.

Need for joint action

Anthropic said an isolated or badly coordinated slowdown could prove counterproductive if other, less cautious companies continue advancing, potentially making the overall environment less safe. Any meaningful pause would require agreement among multiple well-funded frontier labs, along with clear rules on what would trigger or end such a measure and who would supervise it.

The company wrote:

"A unilateral pause by one lab, by contrast, is achievable immediately, but accomplishes much less: it would change who the front-runner is, but it would not create the wider deliberative process that is currently missing,"

Anthropic said its research arm, the Anthropic Institute, will examine what systems would be needed to support a slowdown. In the coming months, it plans to bring together policymakers, researchers, civil society organisations and rival AI companies for discussions on managing risks including recursive self-improvement.

OpenAI, xAI, Alphabet, Meta Platforms and France’s Mistral did not immediately respond to requests for comment on whether they would support the proposal.