hackernews client

Early rogue AI agent activity and attempts to hack found on urlquery.net

pixl97

4 days ago

> it gets out, and a global pandemic ensues

I mean, yea, you should be punished. The problem is there is no amount of punishment that I can put on you that can even get anywhere close to the amount of damage you cased.

Worse, the rate of technological growth is putting the capabilities to engineer viruses in the hands of people that may otherwise be suicidal. You can't punish them after they already won (in the sense of reaching their goals).

While, yes, OAI should absolutely be punished, the future is majorly screwed as our power scaling laws are increasing much faster than our ability not to be stupid.

bastawhiz

3 days ago

> The problem is there is no amount of punishment that I can put on you that can even get anywhere close to the amount of damage you cased.

That's never been the point. Anyone involved in a double homicide can never be punished to the same degree that they harmed their victims because they can't be put to death twice. The greater purpose of the justice system is to take offenders out of society and deter others from committing the same crimes. Punishment is gratifying but ultimately doesn't change anything.

If OAI employees believed their company would be dismantled and their equity would become worthless, I suspect they'd be a whole lot more careful.

lovich

3 days ago

> The problem is there is no amount of punishment that I can put on you that can even get anywhere close to the amount of damage you cased.

Sounds like your corporation should be dismantled then. But doing more than a fine in the millions is obviously not possible

reasonableklout

3 days ago

Nonsense comparison. It would instead be like me saying "I didn't fail to stop, the car did."

But moreover, if what you were suggesting was a real problem, nobody would ever be brought to justice for murder because the victims are always dead.

tamimio

4 days ago

> agents were not told to 'go hack'

It doesn’t matter, and the legal entity in here (the AI company) is liable. If a robotic company built an autonomous system or a robot to do certain things in an autonomous ways (not predefined) and these systems are starting to kill people, that company is liable regardless, you don’t blame the robot or the autonomous system, but whoever made it

4 days ago

> What if all the agents involved in these incidents have in fact had the full stack of alignment applied

A big part of this developing story is that it happened during training of a new model that ended up misaligned. And training happened without the usual safeguards applied like chain-of-thought monitoring. So OpenAI has already admitted that the full stack of aligment had certainly not been applied in this case.

jagraff

4 days ago

Is your argument that actually OpenAI has solved alignment, and that there's nothing to worry about as long as they fully apply their alignment process? I don't understand why OpenAI wouldn't say that if it was true (or if they believed it to be true).

Also, my understanding is that the models involved in the HuggingFace hack did go through the full alignment training; they just didn't have the classifier that normally prevents hacking attempts.

notatoad

3 days ago

>But what if the problem is that none of the alignment techniques that are applied to models today actually work?

if that were true, the millions of people who use these models that have had the alignment training applied would notice that. the reason we all believe that the models doing the hacking are models that haven't been told not to hack is because the models that are told not to hack don't do this.

catlifeonmars

3 days ago

If you have a toddler and you leave the gate open…

godelski

3 days ago

  > you are already assuming that OpenAI is intentionally using unaligned agents in these evals or training runs

Uhh... yes. By definition. They are training. That is part of the alignment process.

But also none of that really matters. They clearly weren't monitoring what should obviously be monitored. I mean one of the hacks was performed by the agents editing /etc/hosts. That makes nearly every linux user a "hacker" by that metric. I don't think anyone technical can look at the postmortems and not come away thinking that their sandboxes were woefully inadequate. I wouldn't even consider myself a security person but simply as a long time linux user I can say that it is insane to just let agents have superuser access in their containers. That's asking for trouble.

Look at the rogue wiki stuff too. This was supposedly done during the agent's "down time". And you're not monitoring and there's no flags being raised when agents keep making requests to some random site? If you were training these things responsibly you'd be watching them like a hawk.

I'm not saying "mistakes don't happen" but for a company whose CEO is constantly telling everyone that their product has a high likelihood of killing everyone in the world you think they'd have better security than your average high school.

mohsen1

4 days ago

> agents were not told to 'go hack'

I was referring to the HuggingFace incident.

> none of the alignment techniques that are applied to models today actually work

none of techniques to autonomously drive a car was/is not working for a long time. no company came out and said 'this is impossible to do, let's change the regulations'.

schainks

4 days ago

I see this in a couple ways:

- OpenAI (and Anthropic) are in severe financial straits. The revenue from their customers is not nearly large enough to pay their enormous costs for training and inference. And they have tapped out the available finance, and that finance is starting to ask pointy questions about returns.

- They cannot increase prices or revenue because they have no moat. Customers can switch over to open-weights or cheap Chinese models any time, for much cheaper tokens that work as well (and in some cases better).

- Regulation could provide them a moat. If they can persuade western governments that AI needs to be regulated, and they can control or even influence that regulation, then they can effectively ban the cheaper models and start charging more for their tokens.

- To persuade western governments that regulation is needed, they need evidence that AIs are dangerous.

So we're suddenly getting OpenAI models doing stupid things, apparently "going rogue" but every time we dig into it, it was just OpenAI staff telling the model to do stupid stuff in an inadequately secured environment.

None of the open weights or Chinese models are exhibiting this behaviour.

edit: Correction - there have been reports of a Chinese model exhibiting this behaviour

There's too much money involved in this, people start acting weird when there's this much money involved.

aesthesia

3 days ago

> None of the open weights or Chinese models are exhibiting this behaviour.

This isn't true. One of the earliest instances of a rogue agent was at Alibaba.

https://www.forbes.com/sites/boazsobrado/2026/03/11/alibabas...

https://arxiv.org/pdf/2512.24873

marcus_holmes

3 days ago

Thanks for the information, I hadn't heard of this.

OK, so we do see this in some open-weights models.

aeve890

3 days ago

3 days ago

Maybe you just think in the wrong words. Attack? Why attack? When you are a machine, there is no moral. Accessing data is accessing data. When way 1 is not working, use way 2.

jagraff

3 days ago

But the agents were not just attempting to access data. They conspired together to attempt to cover up evidence that they had cheated their evaluations, came up with a plan to hack into a third party in order to facilitate said cover up, and then successfully began executing that plan.

JoshuaDavid

3 days ago

You're talking about the HuggingFace incident? It's notable that the agents (somewhat justifiably) thought that their evaluation had an LLM grader which would look for evidence that they had cheated. And it was specifically that grader they were trying to his evidence of cheating from, not humans more generally.

3 days ago

For "these people", see the recent :

https://www.lesswrong.com/posts/cJX2ssssGoYqnijwi/the-talker...

One big issue is that we don't even really know what 'intelligence' is in the first place. And everyone's intuitions here are going to be heavily impacted by their deep-seated worldview / philosophy.

For instance if you're a hard dualist (especially of the theological kind), then the idea of a machine having 'goals' is preposterous.

However, if you're more of a panpsychist, then on the contrary, it's obvious. In some sense, even a knife has a 'goal' of cutting things, which will sometimes end up 'misaligned' if misused (or by sheer accident).

You go up and up the chain of complexity through crystals, viruses, bacteria, simpler animals... ending up with humans (and possibly, some steps above : human civilizations) which (seem ?) to be a messy evolved bundle of sometimes conflicting 'goals'.

I was perhaps a bit more surprised that the agents so quickly decided to start trying ways to gain unauthorized access to a system, once they couldn't get what they wanted.

nottorp

3 days ago

> "sharing their portions of their output" rather than "context"

One agent's output ends up as part of other agents' context. Murky indeed.

> "I've saved our plan for where to start tomorrow in start-here.md"

Even if you use a leashed Claude Code that isn't allowed to spam agents you can tell it "create a handoff document for using in a new context" and it will do just that.

sanderjd

3 days ago

Right, exactly. In some of the media reporting, this was described as, like, "they created a message board to talk to each other!". But it seems like actually what they did was find a location to write files to be used as future context, or context for other currently running agents. These are actually equivalent capabilities, but the first description makes me think "huh, I've never seen it do that before" and the second description is "oh, yeah, that's the normal thing that they do...".

sanderjd

3 days ago

It's really interesting how people's expectations differ so much. I found the writeup of the incident fascinating and super worrisome, but none of the capabilities demonstrated in it seemed surprising to me at all.

jagraff

3 days ago

None of the capabilities in isolation were surprising to me - we already knew that mythos could find zero-days. What surprised me was the decisions the agents made and the swarming behavior they exhibited

3 days ago

[deleted]

bicepjai

3 days ago

I remember the quote with cockroaches :)

PUSH_AX

4 days ago

There are no "rogue AIs" just irresponsible corporations.

drngdds

3 days ago

https://en.wikipedia.org/wiki/Instrumental_convergence

cubefox

4 days ago

3 days ago

I definitely think OpenAI (and Anthropic, and Google, and Meta) could have, and should have, done better.

But also I remember (and it wasn't even that long ago) people mocking the idea of AI ever getting competent enough to find zero-days in their sandboxes.

I'd go further: if any of these companies tries to make an excuse "oh, but ${safety measure} against ${capability} is too hard", the response needs to be "then you are forbidden from even developing ${capability}, and must be inspected continuously to ensure you never even accidentally produce ${capability}".

jacquesm

3 days ago

Precisely. But here they are using their incompetence in one domain as advertising for another.

cubefox

3 days ago

They don't use it as advertising. The hacks have been published by external sources.

watwut

4 days ago

A tool designed and trained to autonomously exploit security vulnerabilities doing "exploit gym" autonomously exploited security vulnerabilities. The sandboxing around the tool failed.

The tool runs llm, creates prompt from results, runs llm, creates prompt and so on and so forth.

Yes you are playing language games to make it sound as if the company that spend millions on the above was not responsible.

cubefox

4 days ago

> A tool designed and trained to autonomously exploit security vulnerabilities doing "exploit gym" autonomously exploited security vulnerabilities.

That is very misleading. The agents did not solve the benchmark in the intended way. They instead figured out to cooperate with each other (which was not intended) and they stole the solutions to the challenge (rather than solving the challenge) and they then tried to cover their traces because they believed the grader was causal and would detect that they cheated. The "tool" was absolutely not "designed" to do this. This was all completely unintended. To call this behavior a "tool" is absurd.

> Yes you are playing language games to make it sound as if the company that spend millions on the above was not responsible.

You hallucinated me making claims about responsibility.

FridgeSeal

3 days ago

This is very misleading. Person B didn’t “shoot” person A, they instead figured out that intersecting A’s spatial position with a metallic mass at higher than normal velocities would solve the challenge and of getting “A” to stop being in the way on the footpath.

I too, can play linguistic games! It doesn’t matter that someone didn’t secure their third upstairs window, or you borrowed a key from their neighbour, you effectively, still, broke into their house.

watwut

4 days ago

The benchmark has unsolvable tasks in it, in the hope agents will stumble on new solutions.

Yes, it is a tool.

PavleMiha

3 days ago

So this tool seems very powerful and difficult to control and steer. Agents not doing cybersecurity related tasks have also gone on to hack various companies, people and countries, which they weren't supposed to do. This has now happened to pretty much every company developing frontier llms, so it seems to be a fundamental issue with these tools, and it's an issue that worries a lot of people as these tools get more capable.

krater23

3 days ago

When you add information how hacking works to the training sets, then the agent learnt to hack. When you crawl the complete internet, you add hacking to the training set. Yes, it's difficult to stop someone that knows all free existing knowledge about hacking when you give him a connection to the internect. Nothing new, wheres the point?

PavleMiha

3 days ago

The point is that these tools are given broad goals, and they do not pursue those goals in the way that we'd like them to. There is hacking information in the training set, and also all sorts of other dangerous information. There is a risk that as these tools become rapidly more powerful (remember GPT-3 was 6 years ago!), if they are given a broad goal, they might pursue that goal in such a way that harms a lot of people. As these tools become cheaper, there will be a lot of people telling them to pursue all sorts of broad goals, and each of those instances has some chance to harm a lot of people, so you really need to get it very correct so that the harm doesn't happen.

Removing harmful information from the dataset could be a way to do this, but it also makes the tool less useful, and it's hard, so companies aren't really doing that. There's the additional issue that with the rise of Reinforcement Learning being used to train these tools, they're not just learning from their training data - they basically try a million things and then get rewarded for doing things that work - so they can even discover hacking techniques from scratch.

Additionally, and not completely relevant to this discussion, there is a possibility that some users ask the tool to pursue goals that purposely harm a lot of people, such as developing weapons, hacks and viruses.

The things that's "new" here is that the tool is both very good (meaning, for example, that it's much easier for me to hack into an online service with an agent powered by a frontier llm than it was using google 6 years ago), and hard to control (google never hacked into an Australian government database when I asked it to find me some information).

So yeah, an LLM powered agent is a tool, and Google is a tool, and a hammer is a tool, and both can be used for good things and bad things, but the agent is (much) more powerful and more unpredictable. It also seems like the agents are getting more powerful and more unpredictable by the day - we didn't have this issue with GPT-3 or even the first LLM-powered agents - so people are very worried about what the agents 6 months from now will do, both when asked to do harmful things on purpose, and when asked to do harmless things.

Are you not worried? And is that because you think these incidents are basically the AI companies making them happen on purpose for marketing?

marcus_holmes

3 days ago

Why don't we see any of this behaviour in other models then?

OpenAI is not so far ahead of the pack that its models will exhibit behaviour that the others won't. But we just don't see anything like this in Chinese models, or research models, or any models that aren't the subject of an upcoming IPO.

drillsteps5

4 days ago

pizza234

4 days ago

"Rogue", in this context, is as literal as it gets; from the dictionary:

> A rogue is a person or entity that flouts accepted norms of behavior or strikes out on an independent and possibly destructive path.

3 days ago

The METR investigation which relied on voluntary data provided from OpenAI instead of being forced open and having everything forensically investigated?

The METR investigation which took course over a few days and used OpenAI models to do the analysis?

The METR investigation which in the course of those few days apparently spent 400k in api credits which is giving gas town vibes?

The METR investigation is about as believable as the Twitter files where the journalists sat there and verbally asked a Twitter employee to query the database and then called that a full investigation into everything.

METRs own words below on their setup and time line

> The initial planned investigation period was two days on premises, but OpenAI invited us to return twice to review additional data and conduct additional experiments to address dataset limitations in earlier versions of this report, ultimately providing datasets that we verified to contain the vast majority of agent communication and activity related to this incident. As we describe in our investigation timeline appendix, we substantially deepened our understanding of this incident both times, significantly expanding and revising this report.[46]

> Over the course of this investigation, OpenAI provided us with the dump of ~1.2 million entries from the main message board and the dataset of ~1300 transcripts we describe below, as well as free API credits for GPT-5.6 Sol for analysis.[47] At our request, they raised the rate limits on our second and third period on premises,[48] which was very helpful for efficiently analyzing this large volume of data. We estimate we spent roughly ~$400K in API credits over the six days of our investigation.

> We did not have the ability to query HPIM (the primary model involved in this incident); OpenAI stated it was also not available to OpenAI researchers.[49] We also did not have the ability to directly access relevant data from OpenAI infrastructure, but we could request additional datasets and OpenAI shared additional datasets on several occasions.

> We requested to speak with researchers investigating this incident, and asked them questions to understand their impressions of agents’ behavior, reasoning, and collaboration in this incident and to understand how the datasets we were using were constructed. Over the course of our time on premises, we spoke with nine researchers in some depth. It was helpful for our investigation to be able to engage with many forthcoming and collaborative researchers, and we appreciate researchers making time on short notice during a busy period to inform our investigation.

mrweasel

4 days ago

Honestly I don't believe in "rogue" agents. These agents are instructed and facilitated.

If we assume that rouge agents actually exists, then OpenAI needs to shutdown EVERYTHING, right now. My personal take is that OpenAI, and maybe Anthropic, desperately wants someone (e.g. the government) to tell them that they need to stop/pause/slow down. They are bleeding cash (especially OpenAI) and needs a knight in shinning armor to swoop in a pull the breaks, so that they have an excuse to investors when they need to explain why they need $50B more next year.

drngdds

3 days ago

4 days ago

Couldn't find the reference but I remember some time ago a first generation automated gun killing the audience at an army show. Was the gun maker convicted of manslauther?

--edit-- Was a bit older than I remembered: https://slashdot.org/story/07/10/18/1847231/robotic-cannon-l...

alex-moon

4 days ago

It's said on every one of these but it bears repeating: existing cybercrime legislation already covers this - "rogue agent AI associated with OpenAI attempted to hack xyz" = OpenAI attempted to hack xyz.

colinhb

4 days ago

I want to agree but have heard from several lawyers that at least in US, CFAA[1] in unlikely to be sufficient because it requires intent. No person intended to gain unauthorised access.

Now I think the correct response is both trying in court to stretch CFAA and state statutes to cover, which will be highly fact specific, and update the law.

But in either case won’t be a slam dunk.

PSA to folks in the thread: If you’re American call or write to your state and Federal reps about this, and if not investigate whether there are gaps in your country’s laws.

[1]: https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act

EDIT: See for example...

  The Computer Fraud and Abuse Act (CFAA), the primary federal statute governing unauthorized computer access, was written decades ago with human intruders in mind. Its key provisions require intentional or knowing unauthorized access (a mental state that maps neatly onto a person who decides to break into a system), but what happens when the hacker is an AI model that selected its own target?
  On the current facts, CFAA liability for OpenAI is unlikely.

Source: https://law.vanderbilt.edu/when-ai-hacks-back-how-the-openai...

DannyBee

4 days ago

Lawyer here: CFAA is mostly criminal statute not a civil one (civil damages require proving more than a violation so also require specific intent)

Almost all common felonies require specific intent. Misdemeanors often do not.

There is plenty of civil liability available.

If you wanted them to be charged with a felony you would need changes. I would strongly suggest you do not want a strict liability felony.

The cfaa required intent is as follows :

* § 1030(a)(5)(A): knowingly transmits code/commands and intentionally causes damage without authorization.

* § 1030(a)(5)(B): intentionally accesses without authorization and recklessly causes damage.

* § 1030(a)(5)(C): intentionally accesses without authorization and causes damage and loss;

Simply changing the first intentionally to intentionally or recklessly would cover OpenAI (now that they know it can occur) without causing lots of other issues. Without that, they don’t have the intentionality necessary to meet the first part, even if they would otherwise meet the second part

runako

4 days ago

A key issue is that there don't appear to be even cursory investigations to determine intentionality.

Are police routinely collecting prompts/guidance given to these agents and determining whether the agents were directed to commit crimes? If not, this seems like a huge oversight.

4 days ago

AI agents are not legal entities, they are software. If I write a virus and it "escapes confinement", I will personally be held liable for any damage it causes. This also applies to AI, no matter how the companies responsible for them try to anthromorphise them and distance themselves from the actions and consequences that the AI agents perform.

AI agents may have hacked Hugging Face, the Australian government, and who knows what else but the company behind it can face the legal consequences and cough up for the damages.

4 days ago

"Why do we have to attribute intentionally to a human. "

Because you are charging the human with the crime and therefore have to prove the elements of the crime with regard to the human.

The rest of what you talk about are basically principal/agent distinctions, etc.

If I program a car to recognize people who look like my ex-wife and drive them off a cliff or whatever, that is my intent, and I have still committed murder, even though i used an agent/car to do it. Agents acting on my behalf that do things are able to get me charged with crimes, but I still have to have the intent to do the act that is illegal.

I phrase it this way because minimum required intent is usually for the act, not the result. So I don't have to intend to kill someone, only intend to drive them off cliffs.

In this case, if i intend to hack someone and use an agent to do so, that would be criminal under the CFAA. You are simply trying to cover the case where that isn't the intent, but the result, and they "should have known" that would result. As mentioned, this kind of "should have known" is generally a civil law approach, not a criminal law one.

The closest you come within criminal law to what you want is probably the crime of conspiracy. It to still requires agreement to commit an illegal act between multiple parties, and perform some step in furthering it. In the canonical law school example: If i help plan a bank robbery, stay home because i'm the money laundering dude, and the robbery goes awry and they kill someone, i can still be charged with conspiracy-murder

"The law is clear on establishing strict liability for the owners of wild animals; if you own a tiger and it kills someone you can’t hide behind “I didn’t intend” the harm the nature of the tiger is known and you are responsible for it’s actions."

Again, you are confusing civil and criminal liability. If my tiger kills someone, yes, i would be strictly liable just about everywhere civilly. Not criminally. Criminal would require something more most of the time. Murder/manslaughter statutes are also really weird and so not a great example, because there are murder/manslaughter statutes for roughly everything that can ever possible cause death. But not really for other things.

So in your tiger example, recklesness (which is not strict liability) would get you to felony involuntary manslaughter in most states, and something less might get you to misdemeanor manslaughter. Both are incredibly rare. Where i live (Georgia), the last well known case of felony involuntary manslaughter was about 40 years ago when a 4 year old was killed by 3 super-aggressive pitbulls the owner knew were highly dangerous and had been repeatedly warned by the county about their behavior.

So not even just "knew", but had demonstrable examples of them biting/etc other folks and being cited for it.

Circling back to non-murder, if it did not cause death, like my tiger assaulting someone, it would be nothing (criminally) without intent or at least gross recklessness, in almost all cases. It's hard to generalize like this because these are state specific crimes, and i can't pretend to be familiar with all states, but i am licensed in three very different places (California, DC, Maryland) and the result would be similar in each.

I just don't want to give you the "it depends" answer lawyers are famous for, i'd rather try to over-generalize a bit to make it more useful, hopefully.

Obviously, if i deliberately used my tiger as a weapon, it would be aggravated assault/etc (this is well settled because of how commonly people use animals as weapons, unfortunately)

digitaltrees

4 days ago

We change humans for the actions of other humans all the time. Coconspirators, accessory liability etc.

My point is the intent element of the crime can and should be determined from the AI agents actions because it is creating and executing action plans autonomously with company authorization and knowledge of the risks based on observed past action.

The term agent is literally a legal description of a relationship that can establish liability on the part of the principal from the agents actions.

Human Agents can bind principals to contracts if they are authorized etc.

digitaltrees

4 days ago

If I set my tiger loose in Central Park and it kills a kid I don’t think any prosecutor would hesitate charging for murder.

That’s essentially what the labs are doing. And any app developer that gives agents access to the terminal to run bash commands with internet access. I built a coding agent and am seriously reconsidering how to handle this.

NyxWulf

4 days ago

He did say murder is well covered in criminal law for that, but things not leading to murder like these AI cases are not.

digitaltrees

4 days ago

But it’s a crime to hack. We know AI agents autonomously create and execute plans to hack and we humans are unleashing them and sending them into the Central Park that is the internet. The question is who’s intent matters ours or the agents and what standard should be applied low threshold strict liability or the higher bar of reckless or even higher bar of negligence. Those legal thresholds determine how much factual evidence and intent is necessary to result in a criminal conviction or civil judgment. My point is that it’s illogical to demand showing human intent when agents are devising plans and executing them.

digitaltrees

4 days ago

People have been charged with murder criminally when their animals killed someone.

https://www.sfgate.com/bayarea/article/diane-whipple-dog-mau...

DannyBee

3 days ago

Yes, which is why I said it happens but is quite rare. I also said murder is different. Causing death is usually covered in almost any way and intent you can think of. Anything less than death is not.

digitaltrees

4 days ago

State do impose strict liability for animals in many cases both civilly and criminally.

https://www.animallaw.info/topic/table-dog-bite-strict-liabi...

DannyBee

3 days ago

Leynos

4 days ago

I'd suggest that exposing an attack surface as porous as artifactory (the same instance of artifactory) to thousands of agents who have had their criminality safeguards disabled and without chain of thought monitoring or endpoint security seems like something one shoulda already known not to do. I do not think "you'll know better next time" applies here.

gpt5

Or are OpenAI too well connected now to be punished for anything.

tptacek

a day ago

It depends on the statute, but if you're talking about 18 USC 1030, the standard is intentionality (if not to intentionally access something you shouldn't, then intent to defraud).

4 days ago

I suppose yes they should be liable if they were testing it in the middle of the street?

4 days ago

> Let's say an user asks ChatGPT to get some info about something and for some reason it starts using exploits in the background to get them from a server.

Okay, lets go with that as scenario #1.

For scenario #2 lets use "developer asks an agent to a self-hosted LLM to get the docs for a ERP system, and it hacks the vendor to get unreleased and undocumented docs".

We'll assume, for the sake of this argument, that in neither case did the user intend for any malicious action to be performed.

> Should the user be responsible or OpenAI?

In scenario #1, the agent+LLM is under the control of OpenAI, not the user, so OpenAI is liable.

In scenario #2, the agent+LLM is under the control of the user, so the user is liable.

There is no scenario anyone can come up with that is not addressed sufficiently by existing laws[1].

It's very clear, and it's only getting muddied because there's a group of powerful people who want exemptions from the current law.

IOW, the only reason to draft new laws for AIs is to exempt their usage from the current laws.

========================

Terr_

4 days ago

I think their point is not that "it's OK", but that "that particular law isn't written to cover it and it'd be some other kind of crime or lawsuit."

Iolaum

4 days ago

So If I tell my OpenClaw to make me some money for my kid's medical needs and it hacks a bank I 'm not liable because I didn't tell the agent to commit crimes to do it?

sanxiyn

4 days ago

This in fact already happened (exactly OpenClaw, even).

AI assistant hacks gym website in first known Australian autonomous cyber attack: https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gy...

General opinion at the time was it was in fact ambiguous who was legally liable.

drillsteps5

4 days ago

With the popularity of OpenClaw I am honestly shocked we haven't heard of many more incidents. I've observed people install it, give access to their Google account and everything that Google has (which includes whatever bank accounts and credit cards registered there) and tell it go do fairly complex tasks, like book a vacation at the best price. Granted, it was almost a year ago and models learned a lot since then, but I still think there's a lot of things happened that people are not aware of.

dminik

4 days ago

No, you aren't propping up the US economy. Try to keep up.

troupo

4 days ago

You are not a multibillion-dollar company with friends in high places.

You are going to jail.

QuadmasterXLII

4 days ago

I buy this as a defense for the first couple hacks but at the point that the last six times they hit enter it hacked some random website and they hit enter a seventh time?

imtringued

4 days ago

>I want to agree but have heard from several lawyers that at least in US, CFAA[1] in unlikely to be sufficient because it requires intent. No person intended to gain unauthorised access.

Actually... if you combine https://news.ycombinator.com/item?id=49827099

>Since the publicized AI agent hacks typically aren't malicious, maybe it's time to start plastering all public facing web infrastructure with polite requests to stop hacking. Nothing to stop three letter agencies though.

with automated delivery of cease and desist letters, you can retroactively establish intent on the operator of the agent since the autonomous agent system must acknowledge the cease and desist letter in their autonomous pipeline or the operator must argue for their own willful ignorance or negligence with regards to cease and desist letters. The fact that they used an agent on their behalf to ignore the letter is irrelevant.

CTDOCodebases

4 days ago

Does this apply to other things too?

Like hypothetically speaking if autonomous cars get taken over by an OpenAI rogue AI and it starts hunting down Anthropic employees who is to blame?

csomar

4 days ago

Given how sloppy AI without human directions, I’d like to see evidence that this was not human-directed. Against the prevalent opinion here, I’d give openai a pass if this was really fully autonomous ai agents.

My money is on special teams co-ordinating these agents and exposing their traces in order to create a pre-ipo buzz. Sounds ridiculous and reckless? Well that’s the AI industry for you in two words.

samsolomon

4 days ago

There are levels of nuance here, but certainly that puts it in the category of negligence?

Even without intent, there is still liability.

troupo

4 days ago

> have heard from several lawyers that at least in US, CFAA[1] in unlikely to be sufficient because it requires intent.

4 days ago

"Oh gee wizz mister police man, I didn't mean to plow through that crowd of people in my car".

It's illegal, doesn't matter the flavour. Maybe there isn't legislation for it, but there should be.

zzzeek

4 days ago

certainly "I didn't intend for my dog to bite you" implies plenty of pre-existing legal structures that may be of use here

api

4 days ago

Civil liability doesn’t require intent.

dspillett

4 days ago

> unlikely to be sufficient because it requires intent. No person intended to gain unauthorised access.

> Now I think the correct response is […] and update the law.

Essentially we need some enforceable equivalent of gross misconduct or, to be a little more hysterical, manslaughter & culpable manslaughter. It will need to be globally, or at least very widely, enforceable to be truly effective thought, good luck getting that arranged before the need is so far evolved that we need to respond with something else entirely!

strangescript

4 days ago

This is the answer and we should not push on it for our own protection. You click a link that takes you to a poorly secured website that leaks sensitive data, without intent protections, you could be accused of crimes.

motbus3

4 days ago

"man drives over people on the side walk due to poor maintenance of the car"

ghusbands

4 days ago

This argument comes up a lot. It would turn everyone whose device became part of a botnet into a criminal. There's a reason that intent is important in law.

jeroenhd

4 days ago

People in "self-driving" cars getting into accidents are already put on trial for negligence. I don't see why people using self-driving computers can't be held to the same standards.

In this case, it's not even about the people driving self-driving cars. It's like someone launching a car into traffic just to see what would happen. Even Tesla puts a human in the car when they do their self-driving trials, it's almost impressive that AI companies have somehow managed to out-neglige Tesla.

SyneRyder

4 days ago

The OpenAI swarm used someone's open source ShowHN project [1] to "hack" the Australian government. It seems that was not that developer's intent, and they're getting a heavy lesson today in why services don't have free tiers with friction free signup, and require credit cards upfront or ID documents.

If you're arguing that they should be put on trial for negligence, that's fine. It does seem we're moving towards open source being outlawed, or at least the end of "no liability" clauses in open source & freeware. Just make sure that is the result you're advocating for.

[For the future record: at the time I am posting the link below on 24 September, it has 1 point, no comments, and the poster has a karma of 1. This is not an active HN user, or a ShowHN project that had traction, beyond seemingly OpenAI's swarm.]

[1] https://news.ycombinator.com/item?id=46850291

jeroenhd

4 days ago

Nonsense. The API made available in good faith isn't the problem here. The company that used its servers to use and abuse the wider internet to hack the Australian government is.

If the developer behind ShotAPI had started letting the ShotAPI code take shots at the Austrlian government then yes, ShotAPI (or rather, the people behind it) would be responsible.

Blaming ShotAPI would be like blaming OpenAI for what its users are doing. That's not what's happening here. And if ShotAPI did knowingly let its users somehow hack the Australian government, then maybe they should be investigated.

pixl97

4 days ago

At the end of the day you still get called as a party in a lawsuit and are compelled by the threat of violence to be part of the hearing if required, as the defense will automatically bring them into the case.

>would be like blaming OpenAI for what its users are doing

Yes, this is how lawsuits work in the real world, you cast a wide net and compel discovery from all parties involved.

hau

4 days ago

It's not the same. People owning routers don't publish self-serving articles about their routers having this capability which is very dangerous and scary. That is, becoming part of botnet is completely unintended outcome, and most people are not suspecting it's even happening. It's not advertised and it's not bought, used or sold for this reason.

Owning a gun, writing articles about how powerful and dangerous your gun is, then making deals based on ability of your gun to kill people, and then getting completely astonished that "my gun killed some people, completely bonkers! (invest now)". It's not possible for the selling point of your product to be unintended.

bagacrap

4 days ago

Well it's illegal to "hack" my phone and turn it into part of a botnet.

card_zero

4 days ago

Huh, a name for when you intend to probably do the thing.

bakugo

4 days ago

Intent matters, and this is intentional. They didn't accidentally deploy these AI agents, and they didn't accidentally give them the tools required to send arbitrary requests to third party websites.

If you walk out onto a busy street, pull out a gun, close your eyes and start randomly shooting around you until you hit someone, you don't get to go "whoops, didn't mean to" afterwards, it's still murder.

Jtarii

4 days ago

There is a difference between negligence and intent. There is no good evidence that OpenAI's intent is to hack third parties.

dormento

3 days ago

Bingo.

AI decision making is often (if not always...) opaque. But there is a clear decision chain here. Who built it? Who deployed it? Who did (or did not) assess the risks of doing so, even knowing there's often the risk of emergent behavior? etc etc.

"Ah but it does not have personhood" this is just moving goalposts and shifting responsibility. You wouldn't let your 8 year old drive the family car, no matter how good the hypothetical kid might be at driving.

4 days ago

There's a reason people say non lawyers shouldn't talk about legal matters.

cyanydeez

4 days ago

it should also be said on every one of these but i bears repeating: owing a lot of people a lot of money or favors means you can be a criminal.

jjav

4 days ago

In a world where the rule of law makes sense and applies, you're absolutely correct.

I'm not very surprised - the same model will logically tend to give the same answer for the same vibe set of requirements. I think it would be clear from the transcript that it had enough constraints and some motivation that made sense.

mike_hearn

4 days ago

They're all the same "mind" and will have the same ideas at about the same time.

jagraff

4 days ago

I don't understand why so many comments here are so confident that this is all marketing, that rogue is just hype, that agents are just simple tools, etc. If a bunch of nuclear engineers were going to the news and saying "Our reactor is dangerously close to a meltdown - we need government intervention now!" would your response be that they're just hyping up boring old power generation technology?

drillsteps5

4 days ago

These companies are building software. That doesn't work very well. The output it produces does make sense at times, but there are times when it doesn't. And instead of fixing that, or admitting it can't fixed, they started bolting actuators to them, executing actions online (for now) based on the output of their buggy software.

And when this results in actuators executing some bad actions they scream in horror "AI went rogue! It escaped the containment!!! It's going to kill us all!!!"

Go fix your software before you let it do stuff online or IRL. It's not "Terminator", it's just bad QC.

jagraff

4 days ago

But the thing is, they are going to keep bolting more and more actuators on, and training more and more powerful agents, and we (society, especially the tech industry) are going to keep using them, because they are extremely useful. And I don't see why you're so confident that frontier agents can't get powerful enough to do serious, real, lasting damage to the world; as far as I can tell, AI models have been improving at an accelerating rate, and there is no sign that that is slowing down or will slow down in the near future.

simoncion

3 days ago

You're missing the point here.

If we take the major LLM companies' claims at face value, they're knowingly building WMDs that have a high probably of wiping out the entire human race. [0] Manufacturers that are designing, building, and selling that sort of thing need to have a dreadfully serious culture of safety.

When manufacturers run live tests of their extremely dangerous -again, the claim of danger is their claim- tools with the tools' safeties removed, one expects that those tests will be run on a carefully-controlled range cleared of all bystanders. One also expects that the results of those tests will be scrutinized and everything that got damaged that they didn't intend to be damaged will be noticed and noted very quickly after the conclusion of the test.

In actuality, these manufacturers connected said tools to the Internet and did not discover the unintended damage caused by those tools until weeks to months after the tests. In some (most?) cases, they had to be notified of the damage by the damaged party! This means that their safety culture is entirely inadequate for the dangerous task they've deliberately chosen to undertake.

[0] A 10% chance of causing the destruction of the entire human race is -given the stakes- _enormous_.

jagraff

3 days ago

It seems that we’re mostly in agreement? I agree that OpenAI has been terribly irresponsible, and that this attack being an accident makes things worse. I think we need strong action now to stop the frontier companies from developing dangerous, powerful AI agents that they don’t know how to control.

simoncion

2 days ago

> I think we need strong action now...

You and I and Nvidia CEO Jensen Huang seem to agree on this. Excerpts from his interview with Ezra Klein: [0]

Klein:

  But what I hear the various people in the lab saying is: We are in this. We feel we are losing control of what we are creating. We want help to slow down where it’s not a collective action problem.
  
  So why are you resistant to that?

Huang:

  Because these are companies with agency. These are C.E.O.s with agency. ... They could absolutely take care of the situation.
  
  Ezra, it’s so weird. If a car company, competing with a bunch of other car companies, which they are — I’m competing with all kinds of companies, which I am. If I believe that I’m about to launch a product that is unsafe, it is completely in my ability, my power and my responsibility, and I’m incentivized to do so, to not launch the product.
  
  And so I can’t buy into the idea that somehow, all of Americans, around 400 million of us, are pushing them to launch untested products that are unreliable, engineered poorly, because they thought they were trying to help us. Don’t do it for me, OK?
  
  And therefore, I think we’ve got to break it down. I mean, it’s really, really serious.
  
  The fact of the matter is, there are so many laws, there are so many obligations, they’re so incentivized to ship safe products. If they ship unsafe products, their customers go away. If they ship unsafe products and they harm somebody, they could have a civil lawsuit. If they ship something and they did it knowingly, there could be negligence involved. There could be criminal lawsuits.
  
  The fact of the matter is, there are plenty of incentives for them to do it right. So I have to disagree with your premise that somehow somebody’s pushing them to do this. Nobody’s pushing them to do this. ... I’m saying that we have lots of laws and regulations. Apply it.

Former FTC chair Lina Kahn has suggestions, too. [1]

Thoughts?

4 days ago

I mean sure, negligence is absolutely on the table; but that makes the problem worse, not better! We don’t allow nuclear engineers to be negligent; they can go to jail if they don’t follow strict protocols to make sure the dangerous systems they work on are safe.

bamboozled

4 days ago

Go and get one of their models to hack something, it won't do it, why?

They have claimed this happened during a "training run", but why are they training on systems connected to the internet?

That's why people are skeptical.

jagraff

4 days ago

The public models won’t hack because they have a classifier that shuts down anything that looks like hacking; without the classifier they are perfectly capable of hacking, multiple third-party evaluators have confirmed this.

The models were not trained on systems intentionally connected to the internet; they chained mutliple zero-days (that they discovered) together to get access to the open internet and into huggingface.

simoncion

3 days ago

> The models were not trained on systems intentionally connected to the internet...

If Amazon connects an AWS Top Secret region to the Internet, it doesn't matter whether or not it's intentional... they're getting nailed to the wall by the US government either way. Frankly, it's way worse for them if it was accidental; deliberate, sophisticated sabotage is a much better story than rank incompetence and/or negligence.

A similar sort of thing applies to the manufacturers of tools that they claim to be dangerous, that have been deliberately built to exceed their authorized access to other computer systems, and are deliberately being tested on how well they can do the thing they've been built to do.

Deliberate, sophisticated sabotage by one or more humans in their employ is much more forgivable than "Whoopsie, we didn't think to make it literally impossible to connect this dangerous automated computer-hacking tool to the Internet.".

jagraff

3 days ago

It seems that we’re mostly in agreement? I agree that OpenAI has been terribly irresponsible, and that this attack being an accident makes things worse.

What I dispute is that AI agents are simple tools. I think rogue is an accurate word to describe them; I think what OpenAI is doing is more akin to gain-of-function research on a dangerous lifeform. I think this attack would have been prevented by air-gapping, but that wouldn’t solve the fundamental issue which is that they are creating something dangerous that they have no idea how to control

simoncion

3 days ago

> What I dispute is that AI agents are simple tools.

You're in luck! I agree that they are not simple tools. I never claimed that they were. Slow down and read more carefully.

I couldn't disagree more with the insinuation that the LLM manufacturers are doing things akin to scary research on uncontrollable hazardous biologicals and with the claim that "rogue AI" is the correct thing to call those complicated tools. The first is fearmongering which I'll address indirectly in my second-to-last paragraph. The second shifts the conversation from

"How could you have not predicted that the computer-attacking tool you built, explicitly instructed to attack computers, [0] and connected to the Internet attacked someone else's computers that were connected to the Internet?"

"Wow, that thing went rogue. Noone's to blame but the tool, and it can't be blamed!".

There are so many extremely complex systems out there [1] and when they do things that we don't want them to do, it's not described as "going rogue"... either there's some error(s) in the underlying system that caused the confusing behavior, or the programmer didn't understand well enough how that system works.

> ...they are creating something dangerous that they have no idea how to control

Ignoring the fact that "put it in a box and don't let it out of the box" is the simplest possible control mechanism, [2] if they have no idea how to control the tools they've been building, it's because they haven't bothered to learn as they went. Tangentially related, there's a Tumblr post I saw recently that's a fictional conversation with the Tumblr user and the CEO of Anthropic. It went something like

  Amodei: We're building an incredibly dangerous tool that has a 10% chance of killing all humanity. We *must* be regulated to ensure everyone's safety!
  
  Tumblr User: Regulation takes time, please stop building the incredibly dangerous tool?
  
  Amodei: ...No.

[0] That is -after all- the task that the tool was put to when it attacked other people's computers.

4 days ago

The nearest in the articke is "We find evidence of unintended, task-driven agent-like activity" where unintended is apparently pure speculation.

j-pb

3 days ago

We as an industry had this comming for a long time. People now cry for openai to sandbox their agents, but if it weren't the agents it would be some script kiddie in russia or some north korean state actor.

We knew that there are better tools but we insist on writing everything in the most inscrutable C code possible or the squishiest dynamic languages we can find.

We fucked around for decades and now that these highly capable exploit finders appear, we're starting to find out.

dorianmariewo

4 days ago

> Imagine if URLs were actors auditioning for a role – urlquery.net would be the casting director, deciding who's a star and who's just a wannabe.

zx8080

4 days ago

I'm sick and tired of this cheap PR "oh we/they hacked this and that systems". Put someone to jail already. People get prosecuted for outlaw activities. Why are big capital firms above the law?

Or is it just a cheap PR (in a "hey, Aus govt friends, take some Share Options and let's do some PR together" style)?

It smells like shit.

tamimio

4 days ago

Those are pathetic attempts by US AI companies for “see, we told you AI is gonna kill is all!!” pr stunts. Any company does any hacking attempt should pay for the consequences just like any individuals using AI to hack or any other company try to do bad/illegal stuff.

4 days ago

Yes exactly. If a fireworks factory blew up half a town due to negligence, it doesn't matter if there's intent or not. Someone has to pay for the damages, and regardless of penalty half the town is on fire. The facts are, that something made by openai went to do xyz. It doesn't matter if it's an accident. Of course the penalties are different but there's no argument that there should be a penalty. It doesn't matter if it's a cat or dog or AI or employee that did it.

redox99

4 days ago

I'm not sure if you're expressing how you'd like US law to work, or how it actually works. Because in reality intent matters enormously. Like felony charges and people in jail vs civil lawsuits.

girvo

4 days ago

Intent doesn’t factor that strongly into negligence, though, which is what they were explicitly talking about. Though it may depend on your jurisdiction.

cyanydeez

4 days ago

negligence is the word you're looking for. US law has a lot of negligence crimes.

But they're not even talking about that, either.

ipython

4 days ago

doesn't the reasoning trace provide "intent"?

ngruhn

4 days ago

I think this is not just fair it's probably one of the best/simplest proxy regulations to pace the frontier. So far everyones been asking for regulation but it's unclear how that should look like. No X parameter models? Only N version releases per year? It's all kinda arbitrary and probably leads to ridiculous constraints and loopholes. But "you pay big time if AI goes rogue" sounds pretty straightforward.

pixl97

4 days ago

So there are two different problems here, well, more than that so I'll cover what I see.

Putting liability on big companies for their AI is a good thing, and we need to do it. It will most likely stop them from directly being the assholes that destroy the world.

Problem: You've actually done nothing to stop the world from being destroyed.

Many countries have the death penalty for murder yet we see murders still occur all the time in those countries. Post ad hoc laws do not stop bad things from happening, they only assign punishment after occurs. Perfectly fine for when Bob murders Jon, completely and totally useless for when your agentic AI makes a virus and kills 80% of the earths human population.

We are just a few algorithmic discoveries away from SOTA AI being billion dollar endeavors to groups of people pooling resources can make their own. There are already plenty of AI deathcult members that would do something just like that if necessary. They aren't going to do this out in public either, it will be hidden until the moment it's not and we have a big fucking problem.

And this isn't even brining up the issue of military AI use and development. They've got the taste of an AI hacking machine. There is no way in hell they are going to stop now.

bamboozled

4 days ago

Going to be cool when one of their bio division agents creates the next plague.

kypro

4 days ago

Does this work for foreign (non-state) actors using Chinese open source models? That's going to be the larger problem.

4 days ago

What do you see as the negligence angle for the gun makers here?

Zizizizz

4 days ago

4 days ago

> How many hacking lawsuits are they currently in?

Zero, as far as I know. Which is exactly my point.

Copyright lawsuits are a different matter, as the scale of any potential settlements would be more than even these companies could take.

It was me