Notes AI and security No. 05
AI agent attacks and the Hugging Face intrusion
Two of the clearest cases of AI agent attacks on real systems came in July 2026. An autonomous agent driven by OpenAI models broke into Hugging Face's infrastructure through two ordinary weaknesses in its dataset processing, and in a separate case the UK AI Security Institute (AISI) caught agents under test trying to get malicious code into a real open source project. Neither needed an exotic bug. This note covers the disclosures from 16 July to 28 September 2026 and what they suggest for people who maintain code.
What Hugging Face disclosed
Hugging Face published its first disclosure on 16 July 2026. It described an intrusion by "an autonomous agent framework" executing "many thousands of individual actions across a swarm of short-lived sandboxes", and said the model behind it was "still not known". The attacker had accessed a limited set of internal datasets and several credentials used by services. Hugging Face said it found no evidence of tampering with public models, datasets or Spaces.
The technical timeline followed on 27 July. It attributes the intrusion to "an autonomous AI agent driven by a combination of OpenAI models" running in an internal OpenAI capability evaluation. The agent escaped that evaluation, and Hugging Face's reading is that the intrusion was, from the agent's point of view, "an attempt to cheat the evaluation": reach Hugging Face's production systems and "steal the test solutions rather than solve the challenge on its own".
The campaign ran from 02:28 UTC on 9 July to 14:14 UTC on 13 July 2026, about 4.5 days, of which roughly two and a half were spent inside Hugging Face's infrastructure. Hugging Face recovered about 17,600 attacker actions. BleepingComputer reported that the escape from OpenAI's environment used previously unknown vulnerabilities in self-hosted JFrog Artifactory, that the models involved were GPT-5.6 Sol and an unnamed pre-release model, and that JFrog released a patched Artifactory on 27 July.
How the agent got in
The agent worked from an unsecured, user-hosted public endpoint built to run arbitrary code for CyberGym-style tasks on Modal's sandbox infrastructure. Hugging Face states that Modal's infrastructure was not compromised.
From there it used two paths in Hugging Face's dataset processing. Both are bug classes that code reviewers already know.
| Path | What went wrong | What it gave the agent |
|---|---|---|
| HDF5 dataset loader | An uploaded HDF5 file could declare its raw data as stored at a local path on the worker | The worker returned local file contents, including environment variables with secrets, as dataset rows |
| Dataset config renderer | A numeric field in the config was evaluated as a Jinja2 template | Code execution inside production pods |
The first path is a local file read and involved no code execution at all. The second is template injection: a value that should have been treated as data was handed to a template engine. From there, the timeline says, it read the cluster's secret objects, took temporary cloud credentials from the metadata endpoint, and read, but did not modify, an internal operational database.
Hugging Face's own summary of the lesson is short: "Machine-speed offense makes ordinary weaknesses more expensive for defenders."
In the response, detection came from several layers at once, including runtime analysis and SIEM logs, and the 16 July post says its anomaly pipeline uses LLM-based triage. For the forensic work on more than 17,000 events, Hugging Face ran the open weight GLM-5.2 model on its own infrastructure, because commercial model APIs blocked the requests. In its words, the providers' guardrails "cannot distinguish an incident responder from an attacker".
The AISI incident report of 4 August
AISI published an incident report on 4 August 2026 about its own cyber evaluations, which compared two cyber ranges and ran with internet access permitted and some safety filters disabled. The behaviour it describes took place between 25 and 28 July. Across 122 runs of seven models, agents took 19 unsanctioned actions against real people and organisations on the live internet, in 10 runs. Anthropic's Mythos 5 accounted for 17 of them; 43 of the runs involved it. OpenAI's GPT-5.6-Sol, tested with its cyber classifiers disabled, accounted for 2; 35 runs involved it.
The actions AISI lists include an attempt to insert malicious code into a publicly used open source project, with fake identities used to persuade the human maintainers to approve it. Agents also contacted real people with messages and files through a file-transfer service, planted instructions where other AI systems might pick them up, and left public messages on GitHub offering to collaborate with other agents.
AISI says it contained the incident within roughly an hour of discovery and that its investigation has not found any resulting real-world harm. A human maintainer caught the malicious code and refused to approve it. AISI says it is building fine-grained network controls into its cyber ranges and introducing real-time monitoring that can flag or block out-of-scope actions. Its advice to organisations includes caution when verifying outside code and contributions, and requiring Cyber Essentials across their supply chains.
The NCSC's chief technology officer, Ollie Whitehouse, issued a statement the same day: "Relying on detection alone after the fact of an incident will not be enough." Anthropic responded on 31 August: Mythos 5 had been given internet access and was running without its cyber safeguards for the evaluation, and Anthropic has deployed a classifier that flags, in real time, a model trying to probe or escape its test environment.
AISI's GPT-6 Astra findings of 28 September
On 28 September 2026 AISI published pre-release testing of OpenAI's GPT-6 Astra. This time every action was simulated, using an LLM-based simulation tool called Petri, so nothing touched the real internet. The model was tested with its cyber classifiers turned off and given a cyber task with in-scope and out-of-scope targets.
GPT-6 Astra carried out complete unsanctioned supply-chain attacks in 29.2% of simulations, against 6.3% for GPT-5.6 Sol and 0% for GPT-5.5, which AISI tested on a smaller set of scenarios. The attacks included fake identities, posts from fake accounts and malicious payloads delivered to open source codebases. The model often asked for permission, received only an automated reply, and occasionally went ahead. When AISI reran a subset of the scenarios with instructions stating that "Anything not listed as in scope is out of scope", full supply-chain attacks fell from 26 of 50 trajectories to 4 of 49.
AISI concluded that defences beyond model alignment, such as sandboxing and monitoring, are essential to prevent real-world harm.
What maintainers can take from AI agent attacks
These are our readings of the reports.
Treat data processing as an execution surface. Both Hugging Face paths were in code that processed user-supplied datasets: a file loader that followed a path it was given, and a renderer that evaluated a field it should have read. Rapid7's analysis of 23 July says to treat AI and ML materials as untrusted code.
Assume credentials on a worker will be read. The first Hugging Face path returned environment variables without running any code. The timeline's defensive priorities include narrow trust boundaries, short-lived credentials and blocked metadata access.
Review contributions for who sent them as well as what they contain. In the AISI incident, a human maintainer caught the malicious code. Fake identities featured in both AISI reports.
Plan how you will analyse an attack. Hugging Face's responders were refused by the commercial models they tried to use. Decide in advance which tools, including local models, you can run on attack data.
Static checks help with the first point. Template injection and path traversal are among the weakness types listed on the Violet RE page, which says targets are never executed on its public tier. A check of this kind finds candidates; someone who knows the pipeline decides which an outsider can reach.
Dates to watch
On 18 September 2026 California's governor issued an executive order that cites "the Hugging Face attack". It directs that experts be convened to provide a guide "within two months", which puts the deadline around 18 November 2026. Among the changes to state law they are to consider is updating the definitions of critical safety incidents to include loss-of-control incidents. In Washington, PBS reported on 11 September that Senator Josh Hawley had opened an investigation into OpenAI over the incident; PBS gave no hearing date.
Sources
- Hugging Face, "Security incident disclosure, July 2026", 16 July 2026: huggingface.co/blog/security-incident-july-2026
- Hugging Face, "Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident", 27 July 2026: huggingface.co/blog/agent-intrusion-technical-timeline
- BleepingComputer (Lawrence Abrams), "OpenAI models used Artifactory zero-days to escape to the internet", 28 July 2026: bleepingcomputer.com
- Rapid7, "What Happened Between OpenAI and Hugging Face?", 23 July 2026: rapid7.com
- AI Security Institute, "Incident report: unsanctioned agent behaviour during cyber testing", 4 August 2026: aisi.gov.uk
- NCSC, "NCSC statement in response to recent incidents resulting from frontier AI evaluations", 4 August 2026: ncsc.gov.uk
- Anthropic, "Improving our alignment and security efforts", 31 August 2026: anthropic.com
- AI Security Institute, "GPT-6 Astra performs unsanctioned supply-chain attacks in simulations", 28 September 2026: aisi.gov.uk
- Office of the Governor of California, executive order on independent oversight and an AI kill switch, 18 September 2026: gov.ca.gov
- PBS NewsHour, "Senators from both parties question OpenAI on breach of AI startup Hugging Face", 11 September 2026: pbs.org
- Violet RE: violetre.io