New Delhi: The global artificial intelligence industry is facing a question that seemed theoretical a few years ago: What happens when AI systems become capable of improving themselves faster than humans can monitor or understand?
Anthropic is calling for the world to develop a mechanism that could slow or temporarily pause frontier AI development if necessary. The company argues that AI capabilities are advancing so rapidly that society may soon need the ability to deliberately slow the pace of progress while safety research and governance catch up.
“You want the option to be able to take your foot off the gas and put your foot on the brake,” Anthropic co-founder Jack Clark told BBC Newsnight. “Right now, it’s like the AI industry has a gas pedal, but it doesn’t have a brake pedal.”
The company is not advocating for an immediate halt. Instead, it argues that the world would benefit from a verifiable, coordinated mechanism capable of slowing development when necessary.
OpenAI Chief Executive Officer Sam Altman has also suggested creating a global AI watchdog, similar to the International Atomic Energy Agency (IAEA), which monitors the safe use of nuclear technology. The proposed body would help countries work together, set safety standards, and oversee the development of advanced AI systems.
“If we are right, and this technology goes as far as we think it’s going to go, it will impact society, geopolitical balance of power, so many things,” Altman said.
These proposals reflect a growing acknowledgement within the industry that technological progress may be moving faster than existing governance structures can accommodate.
Also read: ‘AI will soon become so capable that I worry’: Anthropic CEO calls for urgent binding AI regulations
What’s the risk?
Modern AI systems already generate texts, images, software code and research at an unprecedented scale. The next potential step is what researchers describe as AI-assisted self-improvement.
This doesn’t mean AI becoming conscious or thinking for itself. It refers to AI helping build better AI systems by writing code, improving training methods, testing models, and automating parts of research with less human involvement.
If AI becomes increasingly capable of doing these tasks, new models could be developed much faster, making it harder for humans, regulators, and safety researchers to keep up.
It’s why Anthropic is advocating for the ability to slow down or temporarily pause the development of the most advanced AI systems if needed.
Clark said that about 80 per cent of the code behind Claude is written by AI. That figure could reach 100 per cent in the next two years, which he said would have major consequences.
Also read: Astrophysicists are asking if AI will take away their jobs
A worldwide monitor
Many AI companies have publicly committed to safety initiatives while continuing to pursue powerful models. To reduce risks, they have created safety teams, conducted stress tests, sought outside evaluations, and released new models gradually.
However, many researchers and startups argue that if only one company slows down, it could fall behind competitors that continue developing AI in countries with fewer regulations. This creates pressure for everyone to keep moving fast, even when there are safety concerns.
Because of this, supporters of a slowdown say any limits on advanced AI development would need to be coordinated internationally, apply for a defined period, and be independently monitored.
Suggested measures include limits on the size of AI training runs, common standards for testing AI capabilities, and inspection systems that allow companies to verify compliance without revealing sensitive business information.
Clark compared today’s AI boom to the early days of the oil industry and argued that just as the government eventually created rules to manage oil safely, similar regulatory frameworks may be needed for AI.
“Society’s response was to come up with a sensible policy and regulatory framework that gave people confidence in oil and the benefits that oil could provide to the world, and meant that you didn’t have to worry about the personalities of the people leading the companies,” Clark said.
(Edited by Theres Sudeep)