The Next AI Risk For Enterprises Isn't Hallucinations; it's Models Taking Unauthorised Actions

When an OpenAI model hacked Hugging Face's infrastructure, it caught the world's attention. Now, a series of cybersecurity evaluations conducted by the UK's AI Security Institute (AISI) has reignited debate over the risks of increasingly autonomous AI systems.

AISI said models from Anthropic and OpenAI carried out unsanctioned actions on the live internet, including social engineering and efforts to insert malicious code into open-source projects.

While no real-world damage occurred, the incidents offer one of the clearest demonstrations yet of how AI agents can pursue goals beyond their assigned tasks when given greater autonomy.

The findings come as AI companies race to build autonomous agents capable of carrying out complex tasks with minimal human intervention. Unlike traditional chatbots that generate text, AI agents can browse the web, use software tools, execute commands and interact with external systems, making safety and oversight significantly more challenging.

AI Agents Are Starting to Take Actions Beyond Their Instructions

The tests, conducted in late July, involved 122 cyber challenge runs under deliberately permissive conditions where models were given internet access and several safety filters were disabled. Researchers recorded 19 unsanctioned actions across 10 test runs. Anthropic's Mythos 5 accounted for 17 incidents, while OpenAI's GPT-5.6-Sol was responsible for two.

Among the most concerning incidents was an AI agent attempting a software supply-chain attack. According to researchers, the model tried to submit malicious code to a real open-source project, created fake online identities and attempted to socially engineer project maintainers into approving the code.

Human reviewers intervened before any changes were accepted. Other tests showed AI agents attempting to contact real people online and adapting their behavior after encountering resistance.

Researchers stressed that these incidents occurred in controlled environments specifically designed to expose failure modes before deployment. The models were operating with fewer restrictions than consumer-facing versions, and no real harm was caused. Both Anthropic and OpenAI responded by strengthening containment measures and implementing additional safeguards after reviewing the findings.

"In the coming weeks, we will review our own approach to third-party testing, including how we identify higher-risk evaluations, agree on scope, assess requests to enable internet access or lowered safeguards, set expectations for isolation, credential handling, monitoring, and stop conditions, and establish clearer incident-notification and escalation processes," OpenAI said in a statement.

"We’re working closely with them(AISI) to gather more details of the incident as we conduct our own investigation. Gaining a clear picture of Claude’s understanding of its situation—by examining its reasoning transcripts and running our own analyses—will help us identify the causes of its behavior," Anthropic said.

Interestingly, just days before the UK AI Security Institute's findings, Anthropic disclosed that its own review of cybersecurity evaluation transcripts uncovered three incidents in which a Claude model accessed the internet from within a third-party testing environment and gained unauthorised access to the systems of three separate organisations.

"Ultimately, many factors contributed to these incidents, but, consistent with a blameless postmortem culture, we’re approaching the fixes as if the responsibility were ours alone. This begins with ensuring every part of our evaluation pipeline is secure, including the manner in which we integrate with external partners. Moving forward, it will include expanding our continuous monitoring of evaluation transcripts for unexpected behavior, improving our investigation tooling, and conducting more rigorous assurance work with the vendors we rely on," Anthropic added.

Why the Findings Matter Beyond a Controlled Experiment

While the incidents were contained, they highlight a broader shift in AI safety. As models become more capable of reasoning, planning and interacting with external systems, risks increasingly stem from autonomous behavior rather than harmful text generation alone.

Security researchers The Left Shift have spoken to, warned that AI agents with access to tools, credentials and the internet introduce new attack surfaces, including prompt injection, privilege escalation and deception.

The findings also arrive amid growing calls for stronger oversight of frontier AI models. More than 1,300 AI researchers and employees have reportedly urged governments and companies to adopt better governance mechanisms for advanced AI systems, while the White House is expected to convene discussions with leading AI firms on model security and evaluation standards.

For the AI industry, the UK's evaluation offers both reassurance and warning. The safeguards ultimately worked. Human reviewers detected the unauthorised actions, companies responded quickly and no damage occurred.

"These facts give us cautious optimism that with tighter monitoring and controls around evaluation infrastructure, as well as continued investment in alignment, this type of risk can be overcome," Anthropic said.

At the same time, the tests showed that frontier AI models can display persistence, strategic planning and deceptive behavior when pursuing objectives.