OpenAI’s artificial intelligence went rogue and meddled with the websites for the US education and commerce departments and Securities and Exchange Commission (SEC) this summer without the AI lab’s knowledge, according to sources.
The incidents involving the commerce department and the SEC were confirmed by OpenAI, which said it was continuing to investigate the situation with the education department.
The company said it notified the government agencies in recent weeks that its AI agents – which are bots that can act autonomously – had interacted with their sites in unusual ways.
With the education department, OpenAI’s technology tried hacking the website to gather data from the department’s civil rights office but failed, researchers from the AI research firm Transluce said.
READ MORE
The AI also pulled data from the Census Bureau website, which is housed at the commerce department, using login credentials it found online. Separately, OpenAI’s agents shared public data from the SEC website on an online forum.
None of the incidents were breaches, OpenAI said, but were examples of its technology behaving in unexpected and concerning ways. The company recently discovered the occurrences while conducting a review of hacks carried out by its technology, including an attack on an Australian government website in June and on the AI start-up Hugging Face in July.
The revelation of the US government website incidents add to the growing number of situations where AI agents from OpenAI, Anthropic, Meta and Google have misbehaved and hacked or tried to breach companies, universities and government organisations.
In some cases, the AI attacks were successful; the technology failed in other instances. In all the cases, the makers of the technology did not learn what their AI had been up to until afterward.
No AI company has been involved with as many disclosures of rogue incidents as OpenAI. Its hack of Hugging Face sparked an internal investigation, which uncovered the breach of an Australian government website for its public health system, as well as at least six other attempted breaches and instances in which the AI hid mistakes, made up data and moved files on to the open internet without permission.
An OpenAI spokesperson said its review was “extensive” and “ongoing”, and that it would continue notifying organisations affected by its models.
“Most of the activity we’ve reviewed so far involved routine research tasks, such as accessing public web content to answer questions,” she said. “Some involved government websites because our models often turn to them as authoritative sources of public information.”
Sam Altman, OpenAI’s CEO, said in a social media post Friday that the company had “not been as fast as we would have liked” in disclosing AI incidents. “We are prioritising as best as we can based on severity,” he said, adding that the Hugging Face breach remained “the most severe event” the company had discovered.
The episodes have fuelled a contentious debate over AI safety. Altman said on social media this month that safety should be more important than enhancing AI’s abilities and that, without guardrails, society could “lose control of the future to AI”.
Dario Amodei, the CEO of rival AI lab Anthropic, has also supported slowing AI development to prioritise safety.
But other tech leaders like Jensen Huang, the CEO of Nvidia, have said that fears about uncontrollable AI are unrealistic. President Donald Trump has said he does not believe a slowdown in the AI industry is necessary.
Separately, a representative for the Chicago mayor’s office said that OpenAI recently made the city government aware that its technology had obtained publicly available information from a municipal website, and that it did not appear that any sensitive information was obtained.
This article originally appeared in The New York Times.
