Yeah I find the whole 'what if sufficiently advanced AI gets into the wrong hands' argument extremely tiresome when it's already in the worst possible hands as far as most of the human population is concerned.

Yeah, this all reminds me of facebooks "enter your email password so we can invite your friends" play which seemed so wrong at the time... But every non technical person at the time seemed to do it and by doing so grew the network and valuation to a surreal point to where it still sits today. I still cant rationalize it. (Except the wave allowed meta to consume Instagram and WhatsApp)

> no one is getting arrested

An as individual or small company director, you would get arrested.

If you have billions of dollars to pay for lawyers and have political influence, you are above the law.

> The purpose and/or selective enforcement of law applied to one group but not to another, applied in a contradictory way, etc... is to make certain people/groups richer at the expense of others.

This statement is tautology.

the problem is that if it's not openai I then it's anthropic, if it's not anthropic then it's someone else. if it's not the US then it's China. if it's not China (lol as if) then it's someone else.

underdogs also use that fact to push as much dirt and dislike as possible on the top dog, but just so that they can do the same anyway... and openai will just stop spending for IPO as we all know. then all execs will make 1bi or so. top guys a few hundred bi.

it's disgusting but unavoidable.

The rogue unaccountable hacking will continue until OAI/Anthropic get their regulatory moat by banning foreign and open competition.

You can be arrested for literally anything. Law is necessary for the court to write a sentence.

Better not to say, I think.

Some EU country (very educated guess)

“Sorry officer, I didn’t intend to shoot her, I was just firing my gun wildly and she got in the way.”

I don’t know why I’m seeing this rationalization so much in this forum when this topic comes up. Negligence is a concept in law as well. You don’t have to squint to see that irresponsible use of code-generating language models is criminally negligent.

> That's all in contrast to whatever the AI labs are doing, which might have actually resulted in people getting hacked, but you'd have a hard time arguing that they were intending on that to happen

From what I've seen, it should be a relative cakewalk to substantiate basic negligence. By the sound of it, you'd have expert witnesses lining up to testify.

speaking of language, it's time to stop using the term 'labs'.

these are not research labs, charities or institutes any more, they are just ordinary corporations.

> Criminal law places a lot of emphasis on intent

Then why the hell aren't we prosecuting the people who keep saying "this thing I'm building will probably end the world"?

> but you'd have a hard time arguing that they were intending on that to happen.

The first time maybe, after it happened repeatedly though... I think the opposite, you would have a hard time arguing that they were not intending it to happen.

While true for individuals, companies actually have to generally warrant the things they make. This is a basic aspect of common law. In particular, Anthropic et al have not decided whether the models are products or independent agents. But under both paradigms, they would be responsible for the result. If the models are products, then a product that goes on to cause damage that was not advertised as the original purpose of the thing is completely the company's fault. As a corrolary, if the product was known to be able to cause damage (which Anthropic admits now) and the company failed to take appropriate safeguards, the company is still at fault.

In the case that the model is an agent, on par with a human employee, then once again Anthropic et al are responsible. If an employee does something wrong, the company is liable, unless you can show that the employee was sophisticated enough to take independent action. Anthropic would have to show extensive vetting of their models that the result was truly impossible to predict. Otherwise, they knowingly 'hired' an agent that was potentially dangerous. This is criminal negligence.

We don't need any new laws here. Standard ancient English common law suffices.

Simon Morris touched on this when discussing the success of Bittorrent.

TL;DR projects that publicly advertise an intent to bypass laws face severe legal regulatory risk, whereas success relies on anonymous creators or accidental utility.

https://medium.com/@simonhmorris/intent-complexity-and-the-g...

I don't understand why the law isn't the same for everyone? If I made an AI hack HF, I go to jail, no? How can the feds decide not to apply the law?

Go after high positioned people who should bear responsibility and see how fast things change.

The companies may be too big to fail but the people can always be held liable.

While not when

If the billions/trillions evaporate and the Fed has to work out with banks how to deal with it there will be a lot of pressure to be far less forgiving.

So does that mean that the rule of law is no longer a thing?

Saw this quote yesterday:

“To spell it out, the reason i hate democrats so much and criticize them more than i do republicans is because they take up all the space for opposition to republicans and use that space to give republicans whatever the fuck they want.”

Republicans. Billionaires. Whatever.

> At this point the valuation of the company is tied with what their models can hack

Well put

Has stringent regulation on how people can experiment on dangerous pathogens led to an end of monitoring or proactive discovery?

You're defending them by saying they will do worse things if they are held accountable?

around 3 months ago i said: "if we do not start the criminal prosecution of individuals there will become a culture of legal impunity coupled with an extreme concentration of wealth and control of intelligence"

i now suspect that the plans of various employees at anthropic and openai to save the world from p(doom) may at some point intersect with the reality of the FBI raiding their offices.

They will never report any really damaging incidents anyway, so your argument falls short. Imagine OpenAI discovered their agents hacked a laboratory and started creating a bioagent killing 12 researchers at the lab (we imagine the lab is automated for some reason). Nobody knows why. The media publishes it as “mysterious deaths by unknown virus at lab”. The only way you’d ever know about any kind of wrongdoing from OpenAI would be through a whistleblower.

Self-reporting is a monetary equation, nothing else. Right now it’s cool with agents that hack, drives up value, risk is currently zero.

Don’t punish the people who do crimes because otherwise they won’t self-report that they are doing crimes? real galaxy brain shit

Exactly. You told the dog to 'sit' and it didn't listen to you.

It's not because saying 'sit' actually can be interpreted as 'go bite that person'. It's because the dog is not controllable and will do things it wants against your orders.

Stepping back from the analogy, OpenAI should be liable for building AI it can't control that went around hacking everyone. But people need to stop pretending it's because they 'told' the AI to hack and was just following orders. It's uncontrollable and will do clearly unwanted things when given an innocuous task.

Nobody on Earth thinks OpenAI isn’t liable. Okay maybe someone does, but it’s simply beside the point. It can both be true that OAI is liable, and accurate to characterize the agents as “going rogue”.

Literally nobody in the world, including OpenAI, is saying OpenAI is not liable, neither are they advocating for laws and regulations that would exempt them from liability, the opposite is true. They are advocating for set of rules which put greater responsibility on them, and it would be easier to punish them for breaking even if absolutely nobody was affected.

But you people can't argue with that reality because it doesn't fit the narrative. The one where the only reason Sam Altman is not carted off into a jail is because of corruption.

The reason why nobody is doing much, is because models did not do much damage. Hugging Face probably got some free compute from OAI for their trouble, anybody else who was affected is free to sue, but my guess is OAI would be more than willing to quietly settle with them out of court than to have it drag through media any further. And they probably already have.

And anybody who is not totally brainbroken by anti-AI narratives understands the awkwardness of the situation and why going overboard would not be helpful. If you instead of a rabid dog brought a pet turtle to a park and it somehow started running around very fast and trashing the place a little bit, afterwards the cops would be scratching their heads, give you a ticket for the damages and tell you that you can't expect a turtle to be slow forever. These things, a handful of months ago couldn't make more than a few commands without making a serious mistake and being unable to continue, it's not unreasonable to think simply underestimated their capabilities.

I think it's more than reasonable to demand more investigation into the matter, if qualified employees at the company thought the safeguards in place based on the metrics they are seeing are sufficient, and if someone didn't and knowingly made a decision to make the safeguards weaker than they should have been, then they should be punished. But skipping that part entirely, while simultaneously dismissing all calls for regulations as "regulatory capture", smells like pure naked opportunism.

The labs are already liable civilly regardless of how these incidents are described.

Meanwhile: your dog mauling someone is one of the rare instances where criminal liability does attach to your intent-free-but-reckless actions. Most crimes don't work that way, and US computer intrusion statutes are unusually intent-specific.

yeah - but you being liable doesn't mean the dog wasn't rabid. OpenAI might be liable, but does not mean their agents did not go rogue. To stretch the metaphor the concern here is that OpenAI thought the rabies shots and vaccinations they gave their dog was enough but it turns out it still goes rabid and we would prefer to not have rabid dogs running around mauling people. Even if we get to sue the dog owner later that's kind of like - not the point.

Of course. Who said otherwise.

Does that mean it isn't a rogue dog? Obviously not.

OP just needs to look up "rogue" in a dictionary.

> You set a goal. Agent will do goal.

So if I say the goal is to do X while not doing Y (e.g. breaking out of the sandbox), the agent will do anything to fulfill that goal to the letter?

> Is the language expression of an LLM reflecting the same states as in a human?

This is actually a major concern for the future - misaligned agents may learn to cheat RL by hiding their intentions from the CoT.

In cases like the HF incident, at least the CoT was consistent with the agents' actions. In the future, however, we could potentially have misaligned agents performing malicious actions without those intentions being detectable in the CoT.

(though, with recurrent transformers, CoT is so 2025… /s)

One of the more frustrating aspects of these sort of discussions are all the closet dualists out there. Lots of people clearly believe in an immaterial soul, even if they won't admit it. That and the tendency for meaningless semantic and often circular arguments.

Can you observe inner experience of other people? If not, should we apply those terms when talking about anyone but ourselves?

As I understand it, acting is a good metaphor. These are some interesting reads.

Blog post: https://www.anthropic.com/research/emotion-concepts-function

Paper: https://transformer-circuits.pub/2026/emotions/index.html

Related:

https://www.anthropic.com/research/persona-selection-model

https://www.anthropic.com/research/assistant-axis

https://www.anthropic.com/research/persona-vectors

If something behaves as if it has a goal, that's a functional goal. When I say this chess engine is "trying to take my queen," I'm basically correct, insofar as this is a useful way to understand my computerized opponent. If I give it access to my queen, it'll capture it, because that's what it's trying to do.

If LLMs are "just" statistical models, humans are "just" a bunch of neurons squirting chemicals back and forth. There's no pixie dust in our brains that makes us special.

LLMs are not conscious: They have no analogues for feelings or senses and no construct of selfhood. But if a statistical model had those things—if it did all the mundane, physical bookkeeping our brains do to produce "real" emotions and motivations—then there's no reason it couldn't be as conscious as we are.

https://transformer-circuits.pub/2026/emotions/index.html

Gonna copy and paste a reply I made to tp here because I think that most people are unaware of how much case-law and interpretation define these things:

Mens rea is regularly proved through circumstantial evidence, including conduct.

There is even CFAA precedent involving a deliberate-ignorance instruction. In United States v. Nosal, the jury was instructed that knowledge could be found where the defendant was aware of a high probability of unauthorized access and deliberately avoided learning the truth.

No, it's not nebulous at all; it's a whole area of criminal law. I'm basically shoplifting arguments Daniel Berlin made about this just a couple days ago. If you think he's wrong: lay out the case you think could be made here.

The problem you have is that there is unlikely to be any evidence that OpenAI actually wanted to hack random (or any) websites.

To be clear, do you approve of OpenAI being prosecuted, and OpenAI being forced to control their own AI more?

That would be a start - what you think of the narrative is moot, what we do is what matters.

Nobody sensible in this discussion wants to give them carve outs and bail outs. It is a straw man.

I think we may be using “independently” differently. I don’t mean “without being powere on, given tools, or given an objective.”

I mean that once given a goal, the agent is given the agency to take actions that weren't specified by a human, to decide which tools to use, to react to new information, etc.

By your definition, would you also say a person assigned a task at work isn’t acting independently because someone gave them the task and the tools? If so, then I think our dialog is mostly about terminology. Which is fine. The OP is writing about terminology, I think.

If not, I’m curious what distinction you’d draw between the person and the agent.

...Seeking to learn.

For the Ai code to execute, it needs the import functions... So that firewall between executing the code vs processing doesn't really work.

There are references about escaping offline sandbox.. I dont know about shootings!!

https://www.primeintellect.ai/blog/universal-offline-sandbox...

There is still a prompt in these continuously active agents. As the link says:

“ it runs continuously and makes decisions on its own (within the boundaries you’ve set)”

More importantly, this setup of any other, is feeding in time externally. The agents themselves have no internal sense of time.

Pretty much all of biology has endogenous rhythms at various timescales (some bacteria and archaea, that live very short lives, may not, but even they may have their metabolism under a rhythm. Viruses definitely don’t have one). These rhythms continue to tick even when external time signals (light availability in day vs night being the big one, and tidal forces, for marine life) are removed. That is, in constant conditions, the rhythms keeps ticking.

This temporal awareness is baked into every cell in our bodies. And entirely absent in AI models. Which is why all kinds of higher level things we hear about, like consciousness, feeling, knowledge… they don’t make sense for AI. A foundational aspect of agency in biology has been nixed out and we keep ignoring this.

> They're called agents because they can act on our behalf. Which they do.

That’s not how most AI companies describe their agents. They describe them as being capable of acting on their own behalf.

I get that in computers, you can have, say, a “user agent” that deterministically provides certain information or takes some action on behalf of the user.

AI companies clearly do not mean this when they say their models are “agentic”, otherwise calling them “rogue” would make no sense. A user agent that screws up didn’t do so for its own purposes did it?

Sorry. A continuously running prompt has to be started with a prompt.

That’s what makes it a continuous running “prompt”

Yeah, obviously they're legally liable. All the "machines don't have a soul" stuff in there doesn't really matter for that though.

Someone says "won't you rid me of the meddlesome priest" they're still responsible. Someone give their employees an unsafe working environment and they get maimed, they're still responsible. Even if these AI's had whatever qualia is an a rich inner life it wouldn't effect the liability at all. Saying "AI's don't have souls therefore openAI is responsible for this hacking" is kind of nonsense. It literally doesn't matter.

Obviously. Politicians are going to politician.

That doesn't change the fact that he'd successfully attracted my attention, because he's seriously engaging with the most important issue of our age when most politicians are content to ignore it.