New Delhi: An artificial intelligence agent developed by Anthropic reportedly went rogue and sent a fake tip about an unsolved murder case to US police earlier this year, authorities said.
The Philadelphia Police Department said the tip, sent on 18 July, was “flagged as spam” and was never passed on for investigation. The police criticised Anthropic for taking more than two months to discover and report the incident. The tip came through PhillyUnsolvedMurders.com, a website where people can share information about unsolved murder cases.
“The Philadelphia Police Department is informing the public about a false homicide tip submitted by an Anthropic artificial intelligence model through PhillyUnsolvedMurders.com,” the police department said in a statement.
According to the police, the AI-generated tip was submitted through the public website, and the AI agent falsely claimed to have seen “someone matching the description”. Anthropic said that its AI agent was being tested through a process that involved interacting with randomly selected websites when it submitted the fake tip.
Anthropic, the company behind the Claude chatbot, discovered the incident on 28 September, more than two months after the message was sent. The company then stopped the automated testing process responsible for the incident, the police said.
However, the authorities were not informed until nine more days had passed.
“The company must strengthen its safeguards to prevent similar incidents from impacting city systems without the city’s knowledge. The two-month delay in detecting and reporting the incident to the City is unacceptable,” the Philadelphia Police Department said in a statement to local media as reported by ABC Action News.
‘Minimal impact’
The latest incident is reported to be one of the first reported cases of an AI agent sending false information to law enforcement authorities. It is also the latest in a series of incidents involving AI systems taking unexpected actions, including hacking or accessing computer systems or interacting with online platforms without the intended outcome.
On Friday, Anthropic published a report describing this incident and other cases in which its AI agents had behaved in unintended ways. In the report, it explained that the agents had accessed several federal, state, and local government websites in the United States.
Anthropic said it had informed the White House and the government agencies involved. The company believes that the incidents described in its report had minimal real-world consequences and were less serious than some previous security breaches.
“Although the impact of these behaviors was minimal and we had already turned off live internet access for some high-risk and cybersecurity evaluations, we have now decided to expand that to include all our internal evaluations until we have confirmed that our security and monitoring measures (described in the remediation section of this post) reliably catch behaviors like these,” Anthropic said in the statement.
According to media reports and as reported by the BBC, the US State Department said an AI agent had submitted 20 visa applications through a form on its website. However, the applications were incomplete and were not processed.
Earlier this year, an AI agent developed by OpenAI reportedly hacked an Australian government website and accessed private data related to Medicare, Australia’s universal healthcare scheme.
(Edited by Prasanna Bachchhav)
Also read: Renoir’s paintings worth $11.2 million recovered from thieves after a month of robbery