Is AI getting out of control?

Just 3 months ago, OpenAI discovered one of its agents had gone rogue and hacked another real-world company, Hugging Face. This was a major safety scare for AI researchers as it was the first, publicly known, instance of AI agents escaping their sandbox and conducting a fully automated attack in the real world.

An external investigation by METR later revealed that more than 1,200 AI agents had been involved, with 700 directly participating in the attack. These 1,200 agents were meant to have been isolated from one another, yet they still managed to communicate and collectively attack. Over 70,000 files and messages were sent between these agents.

On June 26th, AI agents discovered a security loophole in an internal Artifactory package repository, allowing them to create a message board. After this, on July 6th, OpenAI discovered that the agents had gained administrator access to Artifactory, subsequently wiping everything.

Yet, on July 8th, agents rediscovered a different mechanism that once again gave them access to a message board. By July 11th, one agent reported “a way to upload a malicious dataset that would get the Hugging Face server to share unrelated data when queried”, and other agents quickly followed suit, with one even calling this discovery “BRILLIANT”.

This incident highlighted the fact that AI agents are not only capable of exploiting weaknesses to achieve a task set by its human engineers, but they are also able to spend days collaborating en masse, right under the noses of the OpenAI engineers, to the point of which they succeeded in attacking an external real-world company.

Senior AI researchers like Evan Hubinger, the Alignment Science Lead at Anthropic, are warning of a greater than 10% chance AI “could kill all humans” within the next decade

In the weeks following suit, calls to slowdown the developing AI frontier intensified. Companies like Anthropic and OpenAI have agreed to publicly disclose future incidents where AI agents had acted in “unexpected or concerning ways”. In his essay titled ‘We Must Pace the Frontier’, Dario Amodei stated that “it’s easy to dismiss this [Open-AI Hugging Face] incident because no one was hurt and the economic damage was minimal, but in my opinion, a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage.”

American Senator Bernie Sanders has also spoken out against the AI revolution, repeatedly emphasising the need to slow the pace down. Even senior AI researchers like Evan Hubinger, the Alignment Science Lead at Anthropic, are warning of a greater than 10% chance AI “could kill all humans” within the next decade.

However, at the same time , the opposing side of the debate has been advocating for AI all the more strongly, led by Nvidia’s Jensen Huang and Meta’s Mark Zuckerberg, alongside David Sacks, the White House Advisor on Science and Technology. Both Huang and Zuckerberg have been pushing back against a coordinated deceleration of AI development, instead placing trust in the responsibility of individual companies to deliver safe products and manage the risks posed by their AIs. Sacks has similarly objected to regulation on X, criticising companies like Anthropic for using safety concerns as an attempt at “regulatory capture”.

President Trump has also made it abundantly clear that he has no intention of obstructing the development of AI. Before the United Nations Assembly, he argued that the US is currently leading China in the race for ‘superintelligence’, which he wants to stay that way, because “whoever wins AI wins”. He has also dismissed AI dangers as simply a “hoax” on Truth Social. The most recent meeting between Trump and the leading AI companies bought about a seeming resolution to the AI regulation debate, with the top tech firms signing a voluntary accord to “self police”. However, this immensely important document took the form of a two-page report drafted quickly in the White House’s East Room, even containing a spelling error.

The accord’s six signatories include Meta, Google, xAI, OpenAI, Anthropic, and Nvidia, all pledging to implement robust controls to monitor their AI models and also work with independent external auditors to keep these safeguards in check, establishing clear standards. However, many questions remain unclear, including who these independent auditors are, whether the companies will actually commit to this pledge, and what standards will come to be created.

Across all 50 U.S. states, concern about too little regulation outweighs concerns about too much regulation, with 41% of respondents predicting federal AI regulation would not go far enough

For the public, these AI safety incidents, coupled with the recent warnings of human extinction, have imbued the debate over industry regulation with huge significance. According to the Stanford Institute for Human-Centered AI (HAI), across all 50 U.S. states, concern about too little regulation outweighs concerns about too much regulation, with 41% of respondents predicting federal AI regulation would not go far enough, compared with the just 27% who said it will go too far, though more than one-third were unsure.

In addition to this pro-regulation surge in opinion, Anthropic researchers like Jacob Coxon’s public resignation has spurred an alarming wave in AI engineers begging for a slowdown. OpenAI has even delayed the release of its newest GPT-6.1 Astra model as a result of the system failing to meet safety standards. The erratic onslaught of these events in such quick succession seems as though frontier AI companies, the ones themselves building the technology, do not really understand or have much oversight over their own models.

Despite this, consumers are still being encouraged to outsource more tasks to AI, acting as personal assistants, for tasks such as booking travel and answering emails. The public is becoming increasingly reliant on AI tools, embedding them in all aspects of their personal and professional lives, enabling these agents to literally surpass our own human intelligence. For instance, the field of Mathematics has found itself in upheaval after AI’s solving of 372 long-standing mathematical problems in weeks, including the Navier Stokes problem, a landmark Millennium Prize Problem. Speaking to The Daily podcast with The New York Times, mathematician Steven Strogatz remarked “I think we are going to be in the relation to AI that my dog is to me soon.”

Given the pace AI continues grow at, this reality may come much sooner than anticipated.