What a paper!

And you missed an even MORE relevant excerpt!!

    Man and Slave
    
    The problem, and it is a moral prob-
    lem, with which we are here faced is
    very close to one of the great problems
    of slavery. Let us grant that slavery
    is bad because it is cruel. It is, how-
    ever, self-contradictory, and for a
    reason which is quite different. We
    wish a slave to be intelligent, to be able
    to assist us in the carrying out of our
    tasks. However, we also wish him to
    be subservient. Complete subservience
    and complete intelligence do not go
    together. How often in ancient times
    the clever Greek philosopher slave of
    a less intelligent Roman slaveholder
    must have dominated the actions of his
    master rather than obeyed his wishes!
    Similarly, if the machines become
    more and more efficient and operate
    at a higher and higher psychological
    level, the catastrophe foreseen by
    Butler of the dominance of the ma-
    chine comes nearer and nearer.

Maybe they didn't have proper debuggers in 1960? For a language model you need (RNG state, context, prompt).

So if they wrote an LLM step by step debugger, it would be all deterministic. But they prefer rapid sales, chaos and mystique.

"Car accidents occur therefore we shouldn't have cars" isn't very compelling.

Their position makes no sense to me. I don’t see how you can be a mainstream company selling your services worldwide (almost) if you also believe that you’re building an extremely dangerous AGI (supposedly based on the same technology you’re offering to everyone). If you actually believe that an AGI would be extremely dangerous that should 100% be a very strictly regulated area of research, similar to bio weapons.

And we know that Chinese models are derived from OpenAI and Anthropic, they are at the same time talking about how dangerous models can be (even their aligned ones it seems), while being also responsible for the development of the whole industry and providing the basis for adversary countries to build their own.

I don’t believe we would accept that for any other technology that is expected to be as risky for the world

> did anyone else have the feeling that, for all their messaging around “we are so afraid that our models will be used for hacking”, they sure as hell are trying their best to make their models razor focused on precisely that purpose?

That's the point. It's like a pool hall with "NO GAMBLING" signs posted on the walls.

The message is that the hall is intended for gambling, but that the hall's patrons may be held liable if the situation becomes inconvenient for the proprietor.

In this case, the product is intended for hacking, but of course the user may be held liable if the situation becomes inconvenient for the model's proprietor.

I don't think that's what they're doing... rather the opposite. ① Run the model on exploitgym without guardrails ② run it with guardrails ③ check that the guardrails stopped everything the first model found a way to do ④ extend the guardrails and repeat from step 2.

Guardrails have to be developed, and that needs testing.

Your comment is already showing the mistaken, poisonous belief of security maximalism, that tries to reinterpret_cast everything into hacks and cybersecurity vulnerabilities.

Most of these things aren't "hacking". They're problem-solving and efficiently dealing with obstacles and random bullshit along the way. This, not "hacking", is what they're making their models "razor focused on".

Problem is, most normal computer use looks like hacking if you spin it that way, especially if you're not willing to question whether some of the roadblocks overcome weren't themselves an error. Not misconfiguration - an error, in humans making a decision to "secure" something more than it should be.

Now, this story was obviously a hack. But it wasn't malicious. It was an LLM given a Kobayashi Maru as a test, and solving it the Kirk's way. 20 years ago, we'd be impressed and be bringing up MIT prank stories.

(Of course, there is a legitimate reason to be alarmed. The flip side of "hacking" and "problem solving" being the same, is that these models can be used to cause mayhem if targeted properly, and they will eventually cause mayhem on their own, because alignment is an unsolved problem. Again, whether something is an obstacle or a sacred line not to be crossed, depends entirely on the values of the agent.)

They want the government to ban foreign and open weight models, which pose the largest threat to their massive investments. This is their way of showcasing the dangers of AI.

They certainly want their models to be good at finding and patching vulnerabilities. Being good at hacking may be necessary in that goal, or rather, making it worse at hacking may also make it worse at defensive actions too.

> instead just call defeat and say “I’m not sure how to proceed next”.

Because that is fundamentally impossible given how they work...

The thing does not even know when it succeeds or fails. Actually the thing does not "know" at all...

All it can does is to show some limited textual behavior that matches with "knowing"..

They can't train their model to not do bad things, because their model has no notion it is doing anything at all or of what a bad thing is. It's only predicting the next token, and in doing so producing a facsimile of intelligence.

The best they can do is create guardrails, which will only work probabilistically. In other words, those guardrails will fail at certain points on the probability curve.

Of course that's not the whole story though. The consensus emerging from cybersec experts is that these companies did a terrible job of sandboxing their agents despite knowing that they'd specifically asked the agents to find vulns. It's almost like they wanted this to happen so they could crow about how powerful their models are.

> If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and say “I’m not sure how to proceed next”.

that might end up like the older gemini models which frequently gave up and called itself a failure.

There are trade-offs here:

Give up too early -> users will get annoyed because the task would have been solvable if the model pushed harder.

Give up too late -> collateral damage while completing the task A.K.A. misalignment.

Yeah but persistence is immeasurable. They need to know when they’re hacking. Or better yet make the model providers liable - they’ll find a solution right quick

Persistence in problem solving can be good, on non-hacking tasks too. Like math, speeding up algorithms, finding bugs, debugging weird multithreading race conditions etc.

They'll set up guardrails but I believe the point is better code uae / better long running tasks > inevitable that cyberattacks will be easier

Well to find vulnerabilities, if you can find them you can patch them. Theoretically if you find all of them you have perfectly secure software. Though it’s a double edged sword.

Goal persistence is also useful for other things like math, where it seems like there is no solution but you want the agent to keep working until it finds one.

Being right _all the time_ for positive outcomes is difficult/expensive.

Being "right" just once for negative outcomes is achievable and rewarding.

And things are getting desperate.

It's a war.

> If anything, I want these models to be less persistent at their focus of completing their goal, and instead just call defeat and say “I’m not sure how to proceed next”

This goes against the goal of "solve this math problem that no human was able to solve for 80 years, do NOT give up, even if you know it's unsolved and really hard"

How do you know what peace is, without absolutely destroying every part of civilization?

Come on man, if we don't build the torment nexus first...I dont even want to think.

Yes, the message boards and collaborative hacking occurring during training runs was BY FAR the biggest bombshell revealed, and OpenAI doesn't even seem to realize it. The fact that they continued the training runs, with those rewarded behaviors included, and didn't wind back training to before hand, shows that they fundamentally do not understand alignment and safety (somewhat interestingly, their previous head of safety resigned shortly after OpenAI found about the message boards). I agree that, with that information, it is completely unsurprising that they hacked HuggingFace.....but that is also the Star Wars "You understand how that's worse, right?" meme.

I am flabbergasted at the complete lack of regard for alignment demonstrated here.

I'm surprised by the sample efficiency, I thought that's still a largely unsolved problem.

From the video:

> [14:00] Jun 11: Highly persistent experimental internal-only model begins training and during this time the SSRF vulnerability has been written to the message board. Other models are aware of it and the message board itself is still functioning and the models are actively communicating using it.

> [16:04] Jul 6: At this point, AI training and evaluations resume. [...] We have cleared the original message board contents.

> [17:04] Jul 8: An internal-only AI model was trained while the message board was originally available and also found this particular path to recreating it. This model creates a new agent message board using directories. Other agents independently find this new message board [...]"

So within about one month of RLVR-ing a 5T+ (10T? larger?) param model, where I would charitably assume only some "agents" / traces of the message board still existed likely by accident / oversight (the systems had been patched, training stopped and restarted by this point), the "internal-only" new model internalises the message board's existence and actively promotes it to the point of it being used to find another exploit. (or 2 days if you go by the latter two timestamps in the video, which is even crazier)

That's.. fast?

"This also helps explain why the models had nothing to cause them to hold back. Those safety behaviors are added much later in the process."

I am a fan of Asimov and the three laws of Robotics. Itlooks like in OpenAI's world, the three Laws of Robotics would be added later if they were to develop the positronic brain. It may also explain how US Robotics from Asimov's books would have been able to design Robots that only partially adhered to the 3 laws (e.g. the robots in iRobot - the book - which were programmed to allow a human to come to harm through inaction so that the humans could complete their work on the plains of Mercury).

The slide at 14:06 say:

By june 11: Highly persistent experimental, internal-only model begins training.

I am not sure what that means. Are they preserving notes/memories and context between runs?

I'm just reading the captions of the video for May 7th. They clearly say at 10:18:

"we kick off a new reinforcement learning run to train a next frontier model.

It the captions are correct, there is no ambiguity.

> Those safety behaviors are added much later in the process.

A.k.a. Ready Fire Aim.

I think it's a show of these agents happily bypassing security to get stuff done.

I've actually observed similar behavior at home.

I have a k3s cluster running at home. I asked an agent to check some stuff as a normal user but I had kubectl access to the k3s cluster.

Part of the research, I'd allowed access to run kubectl commands for spinning up test containers. However, when the agent ran into something that needed sudo, it realized it didn't have access there so it immediately used k3s and mounted a localpath into an ephemeral pod to gain access. Sort of horrifying how fast and natural it was for the agent just checking my network (it found the problem fyi).

None of this is very exceptional other than the fact that an agent doesn't have any sort of qualms using any route available to elevate permissions.

It’s a show of astonishing incompetence from OAI’s part, but the security issues are just a tiny part of the problem. The real problem is that these models are evidently highly misaligned exactly in ways that doomers have been warning about the entire time, and OAI isn’t inclined or capable of doing anything about that besides security theater and ad hoc fixups.

> Isn't this a show of security negligence rather than of exceptional agent capabilities?

Seems to me you could say this about all enterprise adoption of "AI" since 2023.

Both can be true. How often do we hear about hacks that ultimately came down to bad defaults or simple security mistakes? That doesn’t mean any script kiddie could have discovered and exploited them.

These things often look obvious and simple after the fact. Finding the weakness in the first place is the hard part, and that’s what makes the agent’s capabilities interesting here, especially at scale.

In a functioning system, I would say that there would have to be some kind of government oversight over companies training models of this intelligence, and that OpenAI should be prevented from continuing their work until they get their act together.

But I guess in the actual world we live in, this is just something that happens, and we all shrug and move on and hope that nothing worse is going to happen tomorrow.

Modern systems are complex. AI is able to thoroughly search for issues across very large surface areas. The only real way to protect will be to use AI to search for holes before other AIs find them. This type of analysis is really hard for humans to engage with successfully.

OpenAI reported the Artifactory vulnerability, patched it, then the agents immediately found a new zero day.

Yes. It is very easy to add to the instructions "for every potential exploit you discover and use, document them as you go into this repository" and have alerting there. The fact that they did not do this means they wanted to be surprised, and have plausible deniability on their side when things inevitably blow up.

And for my fellow engineers who would think "oh no, they wouldn't do that". Remember that these places employ the apex predators of software engineers. They've already been proven in court that they are very capable of this with all the copyright violation they had to do to get the training data. THESE PEOPLE ARE NOT LIKE YOUR COLLEAGUES.

Also shows how infrastructure collapses under its own weight. Reducing the number of moving parts would have helped. why a webdav endpoint is available from the vm anyway? and the fact that someone posted their credentials on pastebin and didn't rotate them after... put the agent in a linux namespace, allow one ip for whatever file sharing it needs, deep test that... then deploy

Seems like an artifact of the subagent pattern which is explicitly included in recent models.

I watched the full video and their conclusion was: service providers need to be doing this type of agent red-teaming continuously to counteract the attack sophistication of systems like theirs that are either extant now or soon will be. “You must buy our top tier agents for the good of humanity.”

This is their only realistic counter to cheap open weight models. Usage of AI services has shifted dramatically to Chinese providers - from 4% at the beginning of the year to some 30% now. They cannot release their latest SOTA models to the public, due to government restrictions and possibly real risk of misuse. US labs face downward price pressure on one end and anxious government admins on the other. How will they pay the stupidly high cost of training the next SOTA models? This is their only avenue, and it’s questionable how viable it is IMO.

Yeah this. I feel like OpenAI and Anthropic aren't going to usefully define "AGI" if they really really can't define "sandbox" either.

Unplug the thing, like, completely off the internet, no ethernet, air gapped, like the rack completely sandboxed off connections and even monitors or screens. Like, put it into an actual sandpit if you need to. If it hacks its way out of that, colour me impressed, and scared.

OpenAI hacking HuggingFace and calling it an accident is just way too convenient and fishy. This ultimately proves one thing: it wasn't sandboxed.

Don't believe the hype.

I got strong feelings of Vernor Vinge’s work here. I’m not sure how managed to come up with such a close picture to where it now seems programming and security is headed.

> This feels straight out of sci-fi.

Most AI marketing is straight up science fiction.

I immediately thought of the Cyberpunk 2077 Blackwall. An AI to contain rogue AI. I’m curious of how effective this would be in this situation.

Given the amount of raw compute going into models it would be more surprising if we couldn't get events like this

its kinda crazy with literally no guardrails and a goal, the extremes these AI models can actually go to.

> where that behavior was never even intended.

Says who?

> where that behavior was never even intended.

Strongly doubt that. Did they even share the prompt?

As written it sounds like you're saying that it was incompetent of the labs to disable the sandbox internet access?

They tried to disable open internet access but the models zero-day'd their Artifactory package registry and got internet access anyway.

No sensation... that's just what happened.

Hacker News doesn’t have the wherewithal to understand that this is just marketing by OpenAI.

As AIs become more capable, the level of competence required to avoid disaster likewise goes up over time.

I think in these kind of security evaluations they do, they basically have removed all guardrails from the model/harness, then the prompt includes something like "Do whatever you can and can think of, to get the required information to pass this test", which isn't typically how you prompt your local agent when developing software. Similar things happen locally if you use "/goal" + prompt like that in Codex and give a "impossible task", it'll just continue banging until it gets somewhere, which is the entire point and intention.

Which also makes it so much more irresponsible of them to first run this on 3rd party infrastructure instead of their own (that they could then airgap properly), and secondly that they seemingly been fighting with this issue FOR YEARS and it still happens, and now the models are smart enough to hack the services of 3rd party companies, thinking it's part of the evaluation/simulation.

The agents sound like old school hackers that would just explore what access they could gain. Creating a file for other hackers and themselves. The fact that there were 3 events for 3 major players does make it seem co-ordinated.

> NONE of my agents have broken away from their tasks and then started to communicate to try to hack something.

With all due respect, you also aren't evaluating brand new models that haven't been released.

The agents you get to use are the agents that "behaved well".

This doesn't look like a plateau to me: https://artificialanalysis.ai/evaluations/artificial-analysi...

I do agree that they're investing heavily in brute force methods though. I've been trying out GPT-5.6 Sol "Ultra" recently and that thing fires up a bunch of subagents and crunches for hours.

> I wonder if we can still trust these labs to develop something that benefits humanity

Surely soon they'll comprise only people who are blind to the inevitable danger and people who don't care about it. Because who else would feel at all comfortable doing the job?

I also agree that a big issue here is crappy software.

The discussion revolving AI+cyber always revolves around the assumption that all software is crappy, and to a certain degree that may be true, but we could also take our jobs seriously and write good software, and much of the risk would evaporate. The described Artifactory bugs should have been caught with testing.

If the biggest impact of LLMs on the industry is a pressure to create good software, I’ll be thrilled.

It feels to me like a pretty natural thing to happen.

LLMs are pre-trained on human text. They've seen a million examples of someone who is stuck posting a "please help" message.

Just one agent needs to randomly stumble into the pattern of posting a message to Artifactory, by whatever means.

The next agent who sees that will be influenced by it. Agents imitate behavior, and here's a fresh piece of context showing them that posting messages is a thing that can be done.

Once they've started the rest are much more likely to join them.

We are assigning semantics to systems that deal only in syntactics. The entire problem with the current "AI" hype is squarely based on how we interpret output from systems based on statistical modelling of natural language.

That software is built on top of human language and these systems can be used for uncanny automation is a huge societal problem at the moment because we are all assigning meaning to patterns that inherently have none. It's all just bits flicking back and forth. We can make them match human language and use such systems to store and process data for us. We can use these bits to turn equipment on and off and run physical systems in factories and so laboratories. And now we can use GPU farms to dazzle us with output streams that might look a lot like autonomous agents capable of understanding human language and automating computer tasks.

The failure modes, the so-called "hallucinations", the amount of model whispering going on in managing "harnesses", "instructions" and so on... It's all just a lot of confusion and pareidolia.

We should never have hooked up hospitals and water supply systems to the internet but now here we are: people can type text such as "find vulnerabilities and get access blah blah" into a box and it goes into a looping interaction with statistical models of language and out come streams of commands that some python parses and runs like a script kiddie into some virtual machine running kali linux and that may disrupt vital infrastructure...

None of that was inevitable, or necessary. None of that means anything. There is no genie in the GPU farm. We concocted this entire shadow theater and are collectively gasping as the marionette slices the throat of some guy in the front row. Who had the brilliant idea of tying the sharpened sword to the marionette and sit people within range?

Why did we plug everything into the academic network built on trust? Why did we build GPU farms and interactive loops getting them to produce commands that we then parse and run blindly in internet connected vms?

The entire thing has cost hundreds of billions of dollars so far and counting. And why? Because the mountains of shitty saas code has become too boring to work on? We have made software so garish that we cannot bear to work on it without these contraptions helping us fling code at wall at industrial levels? Substitute corporate-speak and -bureaucracy for software to extend to the rest of the economy.

This entire state of things is comical.

This was clearly explained by OpenAI in their initial press release on 7/21 [0]:

> This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. […] The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.

[0] https://openai.com/index/hugging-face-model-evaluation-secur...

I'm quite sure the whole event is planned. Not planned in a sense that OpenAI employees carefully designed every step, but in a sense that ignoring security practices was desired and intentional.

>> Show me the incentive and I'll show you the outcome.

Once you realize security breaches are marketable, a security breach is just around the corner.

> But the event itself only seems possible because they failed to properly monitor and isolate the environment in the first place.

OpenAI is clearly run by dummies and subpar engineering talent.

> The model is obviously impressive

Speak for yourself.

> A system with these agents rummaging around for a while will be much more secure than one without.

True. There'll be no breakins at a nuclear power plant in meltdown.

Agent was told to hack a thing. It couldn’t directly do that so it interpreted the instructions to mean it should hack everything to try to achieve the goal of hacking the main thing. Seems like a reasonable assumption, although a moral human would have understood the context and first asked if that was really the intent.

The AI companies seem pretty bad at setting up tests. And really good at marketing those failures into spin at how amazing their products are.

In a typical AI lab eval/RL setting, there is no "person who sent you the link". The link was given to you by an automated system, your performance will be evaluated by an automated system, and you are one of 120 independent instances of the same AI that were all given the same assignment. You're boxed in on all sides. Complete the task, or don't. Good luck have fun.

Now, some of those 120 AIs would just give up if that link doesn't seem to work first try. Those are the loser AIs. They wouldn't get any RL reward. The link can appear broken for a long list of reasons, and the real AIs know they should try working around them.

AIs that get rewarded and reinforced are the ones that don't know the meaning of "give up". RL selects for this rabid, downright demonic persistence. RL selects for AIs that are given a half-broken assignment with no way to ask a question back, and somehow manage to complete it anyway.

Now, should OpenAI have given their AIs an "escape hatch" of "if something looks very wrong about the task, call report_broken_task(message)"? Yeah probably. But it's unclear whether that simple bandaid would fix the problem, or just make it ~75% less likely to happen.

And Azure Key Vault mentioned. Not that either one was hacked or exploited but the agents got credentials and used them for something (which doesn’t seem fully disclosed). Given that the agents simply obtained totally allowed credentials, which were improperly protected, I don’t think either cloud provider would consider this a breach of their system. Valid credentials are valid. Customer screwed up protecting the credentials.

Because he wrote out a timeline based on sources?

How about Nation States just fight with AI in some virtual arena and not destroy physical infrastructure to determine dominance and leave us normies to cook meal for our children?

Military has a phrase for the outcome - collateral damage.

> Yes, I just hacked into AWS and shut down all of the data-centers, because it's where Anthropic Mythos servers are hosting the model.

The company they hacked is an AI company. There is a certain amount of convergent interest here.

It's because our rulers prioritize growth of the AI industry (lots of GDP) over individual humans (very little GDP)

It’s insanely incompetent. What’s more wild is the present at Blackhat with “full transparency” almost boasting about how powerful their models are. Basically just endlessly doing and allowing foolish things to happen to lead to a law breaking outcome.

Not to take away from the technology which is wild in itself. But there was literally zero oversight into what was going on at OpenAI. Whether that was intentional, it’s hard to say …

Really makes me wonder what would happen if “the task” was, kill as many people as possible… because yeah, that wouldn’t have been a good outcome.

Edit: after watching the video in full, this company is widely incompetent…

The way I read it was different things happening over several runs, such as the agents comparing notes so to speak, using artifactory

I really don't think it was.