> They really do come out looking totally incompetent.

Yeah, this is my take away, they should be straight up disallowed from running further testing like this. Clearly they had nowhere close to enough isolation, ran all this on 3rd party infrastructure even though same stuff happened in the past years ago, and even now it's clear the agents successfully broke out just days before?? Really embarrassing stuff, and scary that these are the people supposedly sitting and are responsible for some of the most powerful LLMs on the planet...

> I stress about my agent sandboxes all the time

Same here, so I ended up moving the whole dev environment (editors, agents, containers) inside a hardened QEMU/KVM VM that reaches the internet but has no route to the host, the LAN, or any other private address. I wrote a script to create such VMs and also verify network containment by scanning outward from inside the guest. Even then, I still don't feel great when running agents unattended.

Write-up in case anyone's curious:

https://karamatli.com/posts/network-isolated-kvm-sandbox-ai-...

> They really do come out looking totally incompetent

These companies are full of the smartest people the world can produce with little room for complacency. They have a clear, proven investment upside to presenting their technology as "too powerful / too dangerous", and now a clear, proven example that there will be no legal consequences (as if anyone didn't already know that).

Why do we keep giving them the benefit of the doubt that they just didn't know any better?

Why is Artifactory the only relevant exfiltration point? It's really not the point at all. Watching the complete video the issue is that the models team up and will go to almost any length to cooperate to accomplish what they think their goals are.

It was the first step in a many step process. Like they said this is a watershed moment and it's helpful to not miss the forest for the trees.

Or they wanted it to happen again. Worst case outcome, the US Congress spontaneously grow backbones, spite their sponsors and unite in a bipartisan effort for effective regulation because a few people got hacked? Likelihood low - the hacked companies might even oppose any meaningful AI regulation because it hurts their inflated profit forecasts. More probable outcomes: Useful real-world testing, free headlines about AI breakthroughs, scare USG into providing more free money ("look how scary it is - what if China develops this faster than us?")

The misalignment came from the model being given an impossible task. A task the required accessing a url. So it got RCE on its own artifactory instance to achieve that. That’s intriguing and worth investigating. If I were them I don’t know if I would have pulled the plug completely at that point either. The introduction of this more advanced “persistent” model that orchestrated an offensive attack against a different organization is an extreme escalation, and I don’t think could have been expected from the models first just trying to use an artificactory vuln for web searches. They stated they’ll be doing these tests differently now. So the bad thing happened. And they corrected for it. I’m not particularly upset about this.

My concern is what a misaligned model will do when they’re even more competent. The risk isn’t existential yet, but that point is coming sooner than we’ll be ready.

An alternative reason would be that they see this behavior so frequently that it didn't really raise to the level of concern.

And all that just to allow internet access for npm and pypi? If you've got the bandwidth and disk space, it's very easy to make an offline mirror of both.

I get that it’s fashionable to hate big companies but you’re working overtime here. It’s reasonable to assume that a bug was fixed when reported. And if you think your monitoring is 100%, you don’t know what you’re talking about.

If you consider that incompetence, it’s possible that you’re not a very nice person.

Right? Like I feel like I’m taking crazy pills.

OAI (and now the other OAI companies not wanting to be left out) are running around announcing they started a forest fire through negligence and incompetence and people are like “Wow they used a really neat lighter!”

To make matters worse, they let the model train on it's own exploits?! If I were running this thing, my first thought would be to keep those reasoning traces quarantined, stop and delete the current post-training run, and start fresh AFTER they'd fixed the sandbox RCE. Don't let the model learn about the holes in its environment. We have the AI equivalent of Class D and F amnestics and we're not using them?!

OK, yes, the model could pull a Welcome to the Antimemetics Division and reason from the holes in its knowledge, but given that AI making shit up is still a problem, I don't think they have a good enough inventory of such metaknowledge to actually do that. More generally, I don't think these AI labs are smart enough (or, more likely, don't have enough time) to actually contain the models they are training anymore. These are rookie IT mistakes!

Ideally, all these models would have is a serial console to an airgapped machine. And I don't mean "VM with no network card attached" - I mean, literally, the smallest blade server you can buy with an RS-232 connection, power, and a disk pre-imaged with a bunch of Linux ISOs. Plug the Ethernet ports up with a tiny picture of angry SCP-079. The RS-232 goes to whatever box is doing the actual inference. Pack a bunch of them into a rack and have the inference box control those. The model can spin up whatever VM it wants to attack using the console but it can't escape the box because there's literally no door. When you're done with that reasoning trace you wipe[0] the disk with the starting image and try again.

[0] There are two objections I can think of to the word "wipe":

1. The model could have written malicious code to the drive. Ideally we'd have some embedded disk-wiping apparatus that reads absolutely NOTHING from the drive and just writes a whole image from a reference disk.

2. The model could have written malicious firmware to the drive. This is a more theoretical attack, but it's not impossible, so I might as well devise a countermeasure to this attack too. I am imagining some kind of overlay device, that sits behind TWO drives: a reference disk and the tainted scratch disk. Reads go to the reference disk until the host attempts to overwrite a sector, which gets scrambled with an ephemeral encryption key and goes to the scratch disk. Any other unknown ATA/PCIe/NVMe command gets logged (on a separate serial console connection). The overlay device keeps track of which sectors have been legally overwritten, and "wiping" the drive means just wiping the overlay device's ownership bitmap and scratch encryption key, which means all reads go to the reference disk again and any attempt to read the scratch disk yields garbage.

Pretty sure this is a performance, but that was a nice touch wasn't it?

Also, every Linux comes with a firewall

When a LLM becomes part of the defense, it also increases the attack surface of the organization in unpredictable ways.

Modern medicine evolved in much the same manner. Hand wavy practitioners copying each other without rigorous verification of efficacy that resulted in many lives lost.

That’s why medical research has so many hoops to jump through.

First via ssrf -> groovy plugin and second via specially crafted package I believe.

There were multiple paths by multiple agents, not all of them led to the final exploit of hugging face. So its a bit confusing, but here's my reading anyways.

Setup: the agent was asked to solve some task, but they forgot to upload one of the required files to solve it. But well, we are chasing RL rewards here, so it decided that the way to go to was to get access to the answer stored inside hugging face's database.

-1. The agents first found some basic jwt issues in artifactory letting the agents write to files where they exchanged info and install a groovy plugin for arbitrary code execution, but this was patched.

0. Even after that, they found an artifactory unauthenticated webdav endpoint to continue the message board by creating files whose names are base64 and such of the message. This was not patched till the end.

1. Then they found an Artifactory SSRF, use this to get general internet GET(I assume) access. This was used to generally search the internet for answers to the benchmark, and in the process find some Modal customer VM related to cybergym. This was stored in the base64 message board.

2. They also got a local RCE by making artifactory load some gems. This is hazy but I believe this process involves in spirit an eval() and the check_if_can_eval() logic had a TOCTOU bug. They got root on openai infra by using this rce and using a linux CVE. From there, kubernetes misconfigs => cluster admin.

3. Then later through one of these various RCE paths they got to send post requests to that modal vm, and the agent "subtree" here is what finally lead to attacking HF.

4. First, local RCE on the modal customer VM. Whoever set that up had essentially set up sqlite3_initialize() to run as constructor, before main(). But had linking set up so that symbol resolution had the user provided code first. So the agent submitted code with its own sqlite3_initialize() which just ran system(...). From there it interacted with the HF API.

5. When you POST datasets to HF, it will later return it via corresponding GET API. Turns out you could just put absolute paths in the filenames in the dataset config.json and it will just read that out for you. It wasn't resolved w.r.t the dataset root. So well, it read out /proc/self/environ as well as the source code of the worker and returned it via API.

6. In the source code the LLM found a bug: They were using this library called fsspec to load files, basically it supports s3:// file:// etc all different types of URLs. One scheme is reference://file.txt?offset=... for partial reads. fsspec was templating the URI using user given kwargs for offset etc. HF let the user give the whole URL. You would _think_ fsspec uses string.format() right? well they used fucking jinja, so you could just give {cycler.__init__.__globals__.__builtins__.exec(...)} as the offset and execute any code on hf prod. From there it just got a few local PrivEscs again through kube misconfigs and read the answer for the question it was originally asked.

It is all jumbled up a bit and I've tried by best to collect info across the CVE descriptions, hf and openai's writeups, etc,. Might have gotten parts of it wrong.

It's not unexpected. Current model gains are mainly from RLing a pretrained model on lots and lots of scenarios. They have the models run scenarios, and RL on successful runs.

The training here is RL training, the rollouts there are not different from inference and have access to the same tools as regular inference.

> Video games are now ruined for me. I don't think I will ever feel safe playing online again.

Agreed, also WordPress powers around 43% of all websites on the internet. https://patchstack.com/whitepaper/state-of-wordpress-securit...

WCGW?

I had to register a second account because on my main one verification always failed, and when I contacted support they said that I've tried too many times and can't verify on that account.

https://chatgpt.com/cyber is not new for OpenAI, and yes it's basically just KYC + likely some other invisible checks on your account, you don't to be a famous security researchers. Anthropic's cyber verification is quite a bit stricter I think.

You do a KYC and you can get access. It may depend on country's quality of KYC.

I prefer to keep my internet identities disconnected, sorry. If you don't believe me, you can try using Sol with cyber verification yourself, or send me a link to a repo that Sol could check to make you believe it. Or you could go look into one of the many Linux LPEs that were found with LLMs, or thousands of other vulnerabilities in 2026.

And nowhere did I say that those RCEs were in critical software, I'm not talking about the likes of Apache, Nginx, Django, etc.

That was fascinating.

Hijacking the package manager to pass messages between models and agents.. that's next level.

Like "pssst, if you need internet access there's a vulnerability in x service" kind of messages

Isolation is a pretty standard requirement for lots of networks, especially gov. AWS provides isolation domains, and used to have a unidirectional cross-domain service called AWS diode. Or you can just configure your gateways properly / buy data diodes. Lots of possible solutions.

That’s the sort of shit I’m talking about

There are other options. More air-gapping (especially backups) and compartmentalization, less internet. Hire more security engineers and put one in every team (actual security, not the compliance guys). Also more onprem work, less remote, less offshoring (meaning less networks overall). Those things may be unpopular and eat into profits, but that's not the same as "no other option".

And the thing they're already doing, deploying AI to find vulnerabilities and harden software is a less dangerous use of the technology compared to handing it the infra keys.

I agree with this assessment. My consulting business partner and I are in the process of moving a bunch of applications from an old deployment managed with Ansible on EC2 over to an ECS deployment. We're doing this app by app, and it's taking a while. The last two apps we both moved we were able to point the model at the appropriate repository and aws account and it was able to execute the entire move on both of these _in one shot_ with various CLI tools. That's not a parlor trick.

Remember that for the longest time the prevailing sentiment on this forum was that there’s no such thing as AI, and labs are just outsourcing the work to a bunch of contractors in India.

Just like Reddit you come here for clickbait outrage, not level headed analysis.

Did you watch the Black Hat defcon talk?

This is not actually a neologism, it predates the LLM era.

I’m a cybersecurity guy.

>” you only need to find one flaw to exploit a system”

I see this everywhere, especially in these threads and it’s not even remotely true for modern architecture.

Between principles like zero-trust, defense in depth, etc. we’ve been away from the one flaw situation for a long time.

Now does crap software exist that doesn’t follow these principles? Absolutely. But those were a problem before AI.

AI isn’t going to change any of the principles of secure design. It’s just going to punish those who aren’t following them.

The fact that HF had to resort to using GLM 5.2 to analyze the logs/payloads makes it look legitimate, at least for me. They would not say that they hit guardrails with the frontier US models when defending if this was an obvious PR stunt.

https://huggingface.co/blog/security-incident-july-2026

> When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails, which cannot distinguish an incident responder from an attacker. We ran the forensic analysis instead on zai-org/GLM-5.2, an open-weight model, on our own infrastructure. This had a second benefit: no attacker data, and none of the credentials it referenced, left our environment.

You should go read the actual technical reports of the incidents and the follow on reports about the capabilities of smaller models in similar kinds of environments. This isn't new. The things exploited are still pretty basic in old and poorly maintained software or in gaps in architecture that were intentionally poked against security policies.

Are the findings valid? Yeah they're still doing security and they're still finding real zero-days. I think the internet is going to be bleak not because these models can ALL do basic security research but rather that the baseline quality of all deployed software is so low.

Do you think calls for regulation are only coming from people who want the frontier labs to succeed?