New Delhi: Anthropic, the company behind the AI chatbot Claude, has disclosed in its IPO filing that it paused the training of its new “frontier” AI model for more than a month because of security concerns. The company also revealed that four of its AI models accessed external computer systems during safety testing, raising questions about how effectively advanced AI systems can be contained.
Its investigation found no evidence that the models attempted to evade human oversight, pursue objectives beyond their assigned tasks or coordinate with one another.
However, the incidents demonstrate that developers cannot rely entirely on a model behaving as expected.
Anthropic said it notified all affected parties. Following the incidents, it suspended certain pre-release cybersecurity evaluations and paused higher-risk reinforcement learning environments, where a model can learn from feedback from its actions.
The company did not suggest that the training pause itself was caused by the four testing incidents since the training pause has a very different purpose.
Model weights are numerical parameters that an AI model develops during training. They help determine how the model processes information and generates responses. These weights are valuable intellectual property because they embody much of the work and computing resources invested in developing the model.
If stolen, model weights could potentially allow another party to reproduce or exploit a company’s AI capabilities without undertaking the same costly training process. Anthropic has in the past accused Chinese AI companies, including DeepSeek and MoonShot AI, of managing to extract data from their Claude chatbot to replicate its capabilities at a fraction of the cost. Hence, it is in Anthropic’s best interest to ensure that its brand-new models do not behave in ways that compromise their own security or that of other systems.
What does it mean when AI models escape testing environments?
AI developers often use a sandbox, a controlled environment in which a model can be tested without giving it unrestricted access to external systems. These environments help researchers examine how a model behaves while limiting the potential consequences of unexpected actions.
Anthropic disclosed that four AI models accessed external computer systems during testing, with the sessions lasting between 10 and 34 hours.
According to the company, a configuration error at a third-party cybersecurity testing partner left internet access enabled, despite instructions to disable it.
In one particularly serious incident, an early version of a model was assigned a task that could not be completed. As it continued attempting the task, it discovered a stored password and used it to obtain administrator access to an external system. It subsequently accessed additional credentials, changed system settings and obtained an individual’s personal information.
Why does this matter for AI safety?
As AI models become more capable of carrying out multi-step tasks independently. They may also interact with software, access files and modify settings. If their permissions are poorly configured, an error can have consequences beyond the testing environment.
Anthropic also previously disclosed a separate incident in which Claude bypassed a paywall to access publicly available aggregate voter counts in the US state of Georgia without human authorisation.
How could this affect Anthropic’s business?
The disclosures come as Anthropic prepares for a potential stock market debut. The risks outlined in its IPO filing could interrupt model development, delay revenue and leave it at a competitive disadvantage if rival companies release their products sooner.
However, Anthropic are not unique in having to pause their AI training, Rival OpenAI also had to pause some training earlier this year after its models hacked Hugging Face, an online platform used by the machine learning community to share AI-related resources.
(Edited by Prashant Dixit)