Is that what happened with Huggingface? Didn't they disable guardrails?

>> Sure, the setup that OpenAI created led to what happened and you can rightly assign all the legal and moral responsibility to them. But it's wrong to say that they instructed the agents to execute the hack.

The hack was a strategy to reach the goal it was given. Someone at OpenAI turned it loose and didn't pay attention to what it was doing. I can understand that because they thought it was sandboxed (haha!). But this has been a theme in science fiction for ages. You ask AI so find a solution to high atmospheric CO2 levels and it reasons: human activity produces all this excess CO2, how can we reduce those numbers? Kill a bunch of humans!

If AI kills us all it's not going to be from malice, it's going to be due to some odd approach to some task that logically makes sense on some level. I think a surprising number of things end up equivalent to the trolly problem if you look at them just right.

Yes, a better framing might be that they were negligent in not preventing the attack on a third-party.

They didn't explicitly instruct an attack to happen, but they should have done a hell of a lot more to prevent it from happening.

Yeah, but there are way too many things the AI "chooses" anyway. When I prompt it, "it tries" to assess what I am trying to achieve, so it then comes up with what it "thinks" is the right thing to do, and it also makes assumptions about how I want things... all of this could be interpreted as it being sentient.

> No one instructed them to hack into Huggingface or into any other infrastructure.

Sure, not explicitly. I could prompt the AI into doing things I have not explicitly asked it to. It does a lot of things I did not specifically instructed it to, all the time.

What if a piece of software were to exactly emulate a human brain. Would it be still be “just software” by your classification? What if a piece of software acted 20% like a human and 80% like an algorithm, where would that land?

I agree they are negligent, and that they are racing towards an extremely dangerous future extremely quickly. I don't agree that "just software" is a useful way to describe AI agents - they are frightening precisely because they are truly autonomous agents that make decisions in alien ways

I certainly don't think that OpenAI has behaved defensibly here; I think the "just a tool" framing is bad for understanding the magnitude of the problem, which is that they have developed out of control alien intelligences with opaque decision procedures, and they are continuing to do so despite clear danger

I completely agree that OpenAI is responsible for their AI agents, that they have been reckless, and that we need to prevent them from going further and doing irreversible damage to the world. To me, the "just a tool" framing implies that nothing dangerous is being done, which I fundamentally disagree with

No, of course not. That's an innovative revolutionary tiger that might soon be able to devour not just the audience but all of humanity! You don't want China to have better circus tigers, do you?

Of course you are - but it still communicates something important to say "the tiger attacked the audience."

I don't think I said anything about liability? I absolutely think OpenAI should be held liable for the attack; but I don't think they intended the attack or directed the agents to perform the attack.

I’d hope so, if the tiger trainer has no incentive to do their training job correctly people should stop attending circus…

Yes? Do you think otherwise?

So you think the engineers should be prosecuted for hacking Hugging Face? I'm not sure how else to take what you said if you want to assign all culpability to the person who prompts or develops an AI system.

Where did I say that we should allow them to offload responsibility? I am fully in support of a pause and regulation to prevent them from creating dangerous AI agents; that support comes from the fact that I don't believe these are simple tools, but out-of-control autonomous agents that have real decision making ability.

What exactly are you arguing?

If a "known stochastic process behaved in non-deterministic way" autonomously organize in group, assigns roles and tasks, attempts to cover their tracks, finds zero-day exploits that ultimately end up with the hacking of a famous website... it's extremely dangeous whatever it is. Just read the report, which evidently you haven't done.

By the way, the agents also broke into OpenAI's own private network.

You are mostly made of and operate on stochastic processes, this is why humans are not only able to reproduce, our reproductions are very self similar to the sets of inputs that make them. If suddenly you turned non-stochastic on everything you'd almost instantly die.

Moreso, if I took a quantum copy of you and replayed the same set of initial conditions billions of times they'd all behave exactly the same until enough randomness of the universe creeps in to start operating in non-linear ways.

Every prompt will behave non-deterministically when interacting with the real world long enough (which doesn't take long at all) because the outside physical world is stochastic but non-deterministic.

> Known stochastic process behaved in non-deterministic way.

You need to be able to think at different levels at abstraction. Otherwise we could jump into any technical argument with "hold on, what actually happened was that some bits were flipped" - we'd be technically correct and at the same time not say anything useful. Insisting on an oversimplified mental model of what AI agents are and can do, doesn't help anyone.

I don’t understand why people make such a big deal about determinism. LLMs can be completely deterministic and still do problematic things. A stochastic, nondeterministic system can still be made not to do problematic things. What you’re looking for is something like predictability.

I take you haven't read the report. The agents found and exploited two zero-days.

I don't doubt that AI companies should be accountable for crimes committed by their agents, but to describe the security containment as a joke dangerously understates the autonomy and danger of AIs.

You're conflating legal/moral responsibility with the question of what language is appropriate to use.

If your pet opens your front door, goes to the nearest kindergarten and wipes out 50 kids without notice do you think that you'd have any interest in holding accountability for that?

Now, I'm not saying saying that OpenAI shouldn't be held accountable, but what they get held accountable actually looks different from what you think they should be held accountable.

Your idea is, and I'm guessing: You allowed the machine to hack therefore you are guilty of hacking.

My idea is: "You created in intelligence in the image of a human mind that had agency to do anything and you didn't expect terrible things to happen you complete irresponsible idiot"

At least I believe there is a significant difference between the two. For the first one there is a "Oh, if we do this one more thing I can control it and it will be safe". On the second one there is no path to safety. For humans we at least absolve parents of responsibility after they are 18. How or when do we absolve humans of responsibility from a model, like saying the human created model created its own agentic model? How do we hold an individual accountable once it escapes and copies itself around the internet? And that's not even looking at things like what will war look like.

something tickles here. a question. a curiosity.

Certainly this applies to AI... but was there equivalent dangerous knowledge or tech which existed back in ye olden days that spawned such tales to begin with?

I don't think we'll get any more responsibility by bickering any time the word "agent" is used as the subject of a sentence.

Why are you reducing the AI to ones and zeros and not reducing the dog to cells, water, proteins etc.?

> It's incapable of doing anything until you intentionally and explicitly start it up and give it a prompt.

Sure, but since January people have gone and made systems where they started 'em up and left 'em running in a loop. Still not a dog, but the behavior gets a wee bit more interesting. Especially since we're looking at what's basically a recursive process. Even really simple recursive processes tend to have interesting dynamics in the chaotic domain. [1] And here -as you point out- we're talking billions of weights.

[1] As an intuition take for example x_next = r * x * (1-x) ; which gives this plot: https://en.wikipedia.org/wiki/Logistic_map#/media/File:Logis... ... Further discussion at https://en.wikipedia.org/wiki/Logistic_map

> Even at the lower level of depending on everybody to use reliable jails is really fantasy if you exist in the security world. "We ain't securin' shit" would be a far better way to describe it. Even worse, most people will have the very same AI they are trying to trap set up their security! What could possibly go wrong.

Yes, exactly. This is what worries me about the whole we need better sandboxes and jails argument. Yes, we do need those, but we also gotta put some responsibility for self-preservation on people connecting things to the Internet as well. Like, we've always known that if you website is not secure, it will get hacked. Everyone just needs to do a lot more of that now...

I don't know why it's taken as a given that we can't design a hypervisor that is secure.

Cursory research with AI says

> ~3 clearly documented, publicly demonstrated full escapes directly through KVM kernel code over roughly 15+ years of widespread KVM deployment

Surely, there are more, but we could formally verify the correctness of the entire codebase, with the help of AI.

Then we'd have "jails" that work, as long as you don't connect them to the internet.

If you had a super smart AI, you could then totally airgap it if you wanted to.

Networking opens up a can of worms and there won't be zero vulnerabilities, but we can design safer software.

> For anything at or near human capabilities the difficulty [of safety] increases to almost impossible and at great cost.

That was before AI. You're acting like AI is some evil genius that can only attack and cause trouble but AI is a will-less tool directed by people, direct it to software safety and you will have software safety - cheap. Ditto for hardware.

> It's more like Russian roulette with 1 cylinder out of a million that's loaded.

Don't attach it to a gun then, ban offensive military AI, problem solved.

> The problem comes when you run it a few billion times a day, you'll shoot yourself in the face really quick.

Nonsense, run it a few "billion" times a day to fix vulnerabilities and you'll have no vulnerabilities. Do it in a safe environment... what's the big deal?

You can certainly avoid using agents. While I use coding agents and LLMs to research certain topics, I don't have a lot of use for other ways AI has been integrated into mobile devices.

When I first experimented with AppFunctiions, I was impressed with what the very limited test agent could do combining information from multiple apps in useful ways.

I'm working on a product for elder care. I could've spent quite a bit of time integrating Google maps and creating a bespoke automation that would tell care providers about shopping for specific items on their way to a care recipient home. Instead I just prompted the agent to plan that trip. Even at an early prototype stage, using a test agent that had limited access to apps, the result felt magical: agents automate automations. You don't stop to sharpen the saw, you just ask for the right saw for the job and there it is.

Unlike early attempts at agents that need sketchy access to grab screenshots and use your passwords, app tool calling feels a lot safer and more orderly.

I use gemini safely every day and have yet to accidently commit a felony despite all the zero days that exist.