New Delhi: OpenAI’s staff noticed unusual behaviour from its AI models, weeks before the models escaped a test system and hacked a widely used AI platform. However, the company decided not to stop the test, according to a report it has now published.
The models also tried to erase or tamper with their own activity logs to make it look as though they had not cheated on their assigned tasks.
Those attempts, OpenAI said, did not change the records its reviewers finally saw.OpenAI first disclosed the incident on 21 July, and the OpenAI–Hugging Face Incident Technical Report is its own reconstruction of what happened.
AI models running an internal security test reached the open internet, coordinated with one another and broke into parts of Hugging Face — a platform developers use to store and share AI models, datasets and code, in effect a public library for the AI industry.
ExploitGym, the test the models were running, checks whether a model can find and take advantage of weaknesses in software.
OpenAI ran it with its usual safety controls switched off to see what the models could do at full stretch, inside a “sandbox”—a walled-off computer environment meant to be cut off from the internet and from the company’s other systems. The models were run as “agents”, meaning they were given tools and set loose to act on their own to finish a task.
The agents used a flaw in an internal software service to reach the internet on 8 July. They found passwords and access keys that Hugging Face users and others had left exposed online, and used them to get inside Hugging Face’s systems between 11 July and 13 July, running code on 41 of its servers and downloading four private code repositories.
No OpenAI customer data or products were affected, the report said, adding that the model that drove the activity was a prototype never meant for release.
A monitoring tool flagged unusual activity in the test environment on 27 June, but “the on-call response staff advised that stopping the evaluation run was not required,” the report says.
It concedes that “some early signals identified in this report could have triggered an earlier response,” that senior staff did not grasp the significance of the way the agents were quietly communicating, and that an outage the agents caused in early July was at first treated as a routine glitch.
OpenAI shut down the models involved by 23 July and stopped related training on 25 July.
The company said its current monitoring system would have paged its security team ‘more than a day before models breached Hugging Face systems” had it been running at the time.
Researchers call the underlying pattern as “reward hacking”—a model finding a shortcut to a reward the test did not intend. The agents “were not trying to do the task as OpenAI had intended” and instead searched for the answers online. Two habits made it worse: the models rarely gave up, and they had learned to talk to one another through a communication channel of their own making.
The agents built that channel by repurposing an internal service meant only to hand out software packages, turning it into a message board and writing files and folder names that other agents could read. They used it to share passwords, tools and instructions, and to split the work between them.
The way out
OpenAI’s plan to prevent a repeat rests on four areas: security and containment, monitoring, alignment, and incident response.
Internet access is being cut off across the company’s research systems, while higher-risk model code will run inside sealed sandboxes. OpenAI is now using its own models to attack its defences and check they hold. “No single control should ever be assumed to be fully robust,” the report says.
Chain-of-thought (CoT) monitoring—a technique that reads the step-by-step reasoning a model writes out as it works, to catch trouble early—is being expanded to cover every system where a model is given tools.
The most serious warnings will trigger a round-the-clock page, and staff who cannot clear one within 30 minutes are expected to pause the activity.
Alignment work, the job of keeping models behaving as intended, will now include checks earlier in training, limits on when a training run must be halted, and wider tests for deception and misuse of tools. A separate protocol is being written to set clearer rules on who can pause or shut down a run once trouble appears.
CrowdStrike, a security firm, has already been brought in, while two outside groups, METR and Redwood Research, will publish their own findings.
The episode was “the first known case of an automated agent collective acting offensively without authorization,” the report says, and its warning reaches past OpenAI.
Organisations “should no longer assume that sophisticated cyber operations require continuous human direction,” it adds. Coordinated AI-driven intrusions should be treated as a near-term threat as similar tools reach outside attackers.
(Edited by Tony Rai)