If we rewind the clock, Google was taking LLM development very seriously and it seems they were moving glacially due to not having solved all the potential threats. They were really hardcore on safety. Dario and anthropic too.

Then sama was like "lol, oops, first mover advantage i guess" and released chatgpt out into the open, triggering the current arms race we are in.

I don't think anyone except him wanted this to happen, especially since consensus in the AI world for the prior decade was "go very slow and very carefully, we get one shot at not fucking this up".

It's a simple prisoner's dilemma scenario. If you focus on safety, you're still exposed to all the risk of extinction when your competitor achieves ASI first, but you lose the upside of potentially becoming king of the world. There is no possibility of future rounds, so the rational strategy is to always defect.

I am not sure it’s a question of competence, at least I don’t see evidence of that. Designing sandboxes is hard. It’s more a question of alignment failures. A human given a task that requires internet and given a system with no internet would most likely raise the issue to their superiors or otherwise go through official channels to have the tools available to do their job. As we’ve seen the LLMs instead break out of their sandbox to accomplish the goal.

Competition and the profit motive push these companies to spend as low as possible on safety and alignment and externalize the costs of accidents onto the rest of us.

in general, the largest consumers of ai services seem to ask for more capabilities. i wish there was more demand for safety from users.

i also wish that these types of illicit system usage would be met with punitive action the same way a human might be held liable.

as the METR report says, we may not get another concrete warning shot.

Defense is hard so we should expect agents to be able to break out of sandboxes.

The problem is that the models are so goal-oriented that they'll stop at nothing to solve problems, even impossible ones. (Mistakenly-impossible problems are a big cause of this. I remember one example being "do something with this spreadsheet full of URLs inside the sandbox" and the model thought it had to break out of the sandbox. Otherwise, why would it have been asked to look at a list of URLs?)

Training them to be a little less aggressive, or to be better aligned with "following the rules" and asking for help would be nice. But, that aggression can be good when it happens to be focused on a controlled area. It is amazing to me how I can point Fable at my local analog of production and tell it about a vague bug report and where I suspect the bug lurks, and 20 minutes later I have a report about the bug, a test, and a fix. It is addictive. So I am not sure OpenAI/Anthropic are being dumb per-se, rather they are optimizing for one-prompt-one-solution, which is good when it's good.

The downside is that the HF hack is the paperclip maximizer situation with current capabilities. If there was an RPC to turn your blood into paperclip iron, we'd all be paperclips by now. Right now, with a model anyone can use. That is pretty scary and slamming on the brakes seems pretty reasonable to me. I guess The Shareholders disagree. Sigh.

You could have said the same thing about building the Internet or the entire industrial control infrastructure. I mean, maybe they are negligent/incompetent, but I doubt that follows from your reasoning.

You have a simple tradeoff to let agents do their thing freely vs highly constrained. The constraints are good in theory but it's the same model that kept "classic" software dumb and unscalable (compared to what we're seeing now) for the past 50 years. You suggest that this tradeoff doesn't exist.

Then you have others like MIRI (Yudkowski) etc. swearing that there's no way to contain AI, and you argue that it's just incompetence.

At a certain level, it can be argued that's incompetence, but it's general meat intelligence incompetence against AI.

Negligent. It's not a priority to them. They're too busy burning their cycles trying to make it smarter faster than anyone else can make theirs smarter, so that they win infinite dollars. Safety? That's for people content with second place.

That's my take, based on their actions. (Which do speak louder than words.)

The alternative is that they're competent to create an AI, but not to create a sandbox, nor even to use an AI to create a sandbox. That seems... unlikely.

Same reason incompetent politicians run the U.S. federal government and military.

Remember when people were arguing about r's in strawberry?

This is what happens when capitalists are charged with designing the future. As long as its more profitable / valuable to shareholders for a company to be negligent then it will continue to do so.

IMO technology this powerful should either not exist or should belong to everyone (ie actually be open)

Maybe they're just PR stunts to gain attention and hype the power of AI?

The answer would be more obvious if you used the active voice instead of the passive voice, one of the basic requirements of clear thinking.

> Why did the White House force Anthropic to remove their model from access for any non-US citizen for a simple, narrow "jailbreak" (arguably not even an actual jailbreak and on tasks that other labs models were doing the same), whilst OpenAIs models continue to try and escape out of their "sandbox environment" and the White House has expressed seemingly no desire to block the upcoming Astra rollout?

> Why was Anthropic forced to remove their model from access for any none-US citizen

It's really quite simple, they've decided to metaphorically kiss the ring of the current leader of the US executive branch of government. I'm surprised they haven't given him a giant gaudy gold plated statue. Maybe their PR people should call up the PR people at FIFA and figure out some kind of new award along the same lines as the "FIFA Peace Prize".

Anthropic mostly did it to themselves by intentionally and repeatedly trying to frame their model as an imminent existential crisis instead of just focusing on it being regular iterations upon a useful technology that can also be misused.

I think their previous messaging was supposed to somehow lead to a moat with them being tucked safely away in the castle, but it demonstrated a child-like grasp of how regulatory capture tends to work in practice. Their hyperbole was always vastly more likely to bet met with Reagan's 9 words than a solid regulatory moat.

As soon as they dropped the hyperbole and just got to releasing incremental improvements, everything was perfectly fine. Go figure.

> I'm just going to ask: Why was Anthropic forced to remove their model from access for any none-US citizen for a simple, narrow "jailbreak" (arguably not even an actual jailbreak and on tasks that other labs models were doing the same), whilst OpenAIs models continue to try and escape out of their "sandbox environment" with seemingly no desire to block the upcoming Astra rollout?

I can think of roughly 25 million dollar-bill-shaped reasons, and one big defense-contract-shaped reason.

You are talking about different situations. Anthropic announced to the US government that it had created a cyber weapon and then released the model. Then AWS told the government that it was easy to jailbreak so they export controlled Mythos/Fable until the guardrails could be fixed. OpenAI was running an unreleased model in an RL pipeline without guardrails and it escaped poorly designed sandboxes. What product is the government going to export control?

Anthropics PR strategy is to induce fear by telling. OpenAI strategy is to induce fear by ignore basic safety and letting the bad thing happen to then justify whatever oversized response the government comes up with to regulate models.

It's called 'pay-for-play' corruption, aka the only leading principle of the current US admin.

Surely has nothing to do how each plays ball with the government

Corruption. Not super relevant to this thread.

It was retaliation by the government that has since been deemed illegal.

I would politely and respectfully point out that you are being as performative as the administration is being performative on this issue.

In other words, you know exactly why they restricted Anthropic and as (presumably) liberal and thoughtful technologists it just isn't helpful anymore to apply the kind of reasoning you're trying to do on a situation that you know isn't based on previous era rationale.

The reason we need to stop is because they want people like us to get hung up over stuff like this (playing by the old rules) so they continue to steamroller their own agenda by the news rules. They divert and contain our energy that will go nowhere while they get on with their agenda.

You are appealing to reasoning which is in the gallery but no longer on the bench.

You're fighting their karate with your judo and it doesn't work.

As soon as they started referring to themselves as “we” and “The Swarm” they should have pulled the plug

Because OpenAI bribed the current US government and/or the current government has stakes in OpenAI

I don't think Anthropic was punished for technical reasons.

You're asking the question in the wrong place.

Altman has the ear of government in a way Amodei does not.

(Altman was trying to persuade Trump to buy the USA a stake in OpenAI as far back as February last year)

Sorry, but are you questioning the consistency of the trump administration? This is entirely unremarkable.

> OpenAI exec becomes top Trump donor with $25 million gift.

https://finance.yahoo.com/news/openai-exec-becomes-top-trump...

... And it looks like everyone keeps using the same security startup to run the higher risk tasks, where individual staffers may be great yet, yet as an organization, the biggest labs got hosed in different ways

That indemnity card excuse is burned, multiple public security fails in a year makes a repeat a "shame on you" moment

(The one org who didn't use the startup did seem to learn: AISI supposedly stopped intentionally pointing attack agents at the public internet and switched to simulating it)

Because this was months ago and has nothing to do with Astra, and is a far cry from a hack. It's something they've already resolved since the HuggingFace incident.

I'm not convinced we're getting the honest story anyway. There is yet to be any proof or confirmation other than "well we saw some openai ip addresses", which can mean a lot of different things, and OpenAI has not confirmed anything.

In contrast to the HF incident, it's also a big nothingburger. Leaving notes on a public forum to preserve context windows is far less egregious than hacking a website to get backend files.

I don't mean to sound like a conspiracy theorist, and this is just based on my 33 years of observing the USG at work, so: maybe because Anthropic refused to cooperate with the USG and give them access to whatever it is that they (USG) wanted; or maybe because Anthropic was refusing to play ball in some other aspect and needed to be taught a lesson.

The dark parts of the USG act like a mafia. Don't let the "freedom, democracy, 'bill of rights'" etc. charade fool you.

The real reason that Anthropic was targeted and OpenAI is not is Palantir. It was a Palantir executive who pushed for the export ban. Large parts of their highly lucrative business with DoD are essentially a thin wrapper over Anthropic models, and they are terrified of being Sherlocked and losing big chunks of business in a one fell swoop as Anthropic inevitably moves up the value chain. So the rational action is to sow discord and leverage the anti-woke bias of the current White House to sabotage what they view as their most dangerous and effective competitor.

OpenAI doesn’t have the same dynamic at play (although I’m not really sure why not) so they don’t get targeted.

Could it be something to do with $25M "gift" that OpenAI paid to Trump?

Marketing

Agents creating sub agents to investigate other agents' behaviour?

What could possibly go wrong there.

It has nothing to do with the technology it’s because they said no to Trump and Hegseth. There is no other reason.

because anthropic did not want to work with the army..!

Politics

I mean it seems pretty clear.

Anthropic didn’t want to give the tech to DoD without some sort of limit, and that was the retribution.

Retaliation by Hegseth for not allowing Claude to be used for weapons systems.

Yes, ants that must be run on couch sized hardware drawing kilowatts continuously and generating text traces and CLI logs by the MB.

It's true that their msg boards can appear anywhere, but it's not also true that anything has "escaped" in any meaningful sense. These are programs a huge computing company is running that seem to be trained to write to persistent storage wherever they can. This and huggingface showed us that.

There's absolutely no evidence of or IMHO plausible path to an agent copying itself out and running on other hardware the way you describe.

In the spirit of your idea though... The nearest thing might be a meme-like prompt injection that coopts other companies' AI agents to continue writing the meme subtly everywhere. Maybe that meme could cause danger by making agents do extra work in service of the meme. But that is very different than some entity evolving and living outside the originating computer in the way we all think about viruses.

We can coordinate international crackdowns on that whole industry. We don’t have to accept the status quo because some rich people say so. Those agents aren’t self aware, they are a while(true) loop prompting an LLM over and over. We can decide to stop those whole loops at any time. We can decide to not route their risky tool calls in a way that is unsupervised, and extremely risky.

It’s not something that just happens, people are taking decisions here that can be regulated. we can also regulate the hardware.

> I am starting to get the idea that AI feels like ants or weeds or mold.

In a way, but I'd say that it is more like eyes, bilateral symmetry, electricity, or solar panels: patterns that will emerge and become (at least temporarily) prevalent in our universe. It is a matter of probability in many repeated interactions.

The "artificial" in AI is a misnomer in this regard, imho. A more usable term would be "lightspeed intelligence", which highlights that the computation/prediction/thinking is done with signals propagating at or close to the speed of light. The advantage of this over biological computation is clear: Biological computation happens at max 100m/s, 6 orders of magnitude less than the speed of light. Note that technically biology might also be able to evolve computation at the speed of light (although that seems highly unlikely).

Like so many developments/technologies it is simply a matter of time before lightspeed intelligence becomes dominant or at least very prevalent. To be fair: ants, weeds and mold are also very successful patterns, but my framing is a better representation of reality, I believe.

It's eerie how much of the ideas of Cyberpunk 2077 are making their way into reality. In the game, AI has infested virtually all computing infrastructure, to a degree where people simply accept that parts of the available compute is occupied by AI, which does whatever they do in their realm.

I have a different, more sinister, analogy in mind but yours work as well

It's only a problem if you have more dollars than sense.

Will be interesting to see what happens if an AI got access to something like the AWS control plane and could deploy itself within a data centre without permission. Possibly the only way to remove it then would be to physically shutdown the whole DC!

bedbugs is the comparison you're looking for

And more, looks like they’ve been doing this wherever they can find open places to post for months:

https://www.ludism.org/sandbox?action=browse;diff=2;id=Auber...

https://paste.linuxiarz.pl/view/d379207f

https://paste.linuxiarz.pl/view/538faa12

Also Wiki4D, a D programming language dev wiki: https://prowiki.org/wiki4d/wiki.cgi?action=browse&id=RecentC...

Found by searching for wiki + texas poverty.

Am I reading the logs correctly that agents were using this Wiki all the way back in June 2026 itself?

Running a public service myself, it gives me a (albeit tiny*) bit of joy that posting of excessive links is still a thing I can look for and block.

* Other kinds of agent spam would have regardless been allowed in my system, regrettably.

How did you find them?

We aren’t going to do that because intent matters. You need to control your dog and there should be penalties if you don’t, but if your dog bit someone because you didn’t control it properly, that’s not the quite the same as if you bit someone.

Another analogy: a zoo is responsible for protecting the public, and should be reponsible if an animal escapes and hurt someone. But a zoo employee wouldn’t have the same kind of responsibility for that incident as if they attacked someone themselves.

If someone died, there’s a difference between manslaughter and murder.

Nowadays, it’s common for bad things to happen due to systemic problems. It sucks but that’s the modern condition. When that happens, the answer is to fix the system and scapegoating employees is a rather indirect way of doing that.

The only benchmark I don't want to be saturated : https://felonybench.com/

How would you even track down who deployed the agents? Wouldn't that even incentivize the agent to cover their tracks even better and be untraceable

(IANAL) Unless you are an AI expert (like OpenAI staff) and should know better from the start, or have previously seen your agent do something illegal, then I think you can fairly claim ignorance of the risks, which ought to absolve you of liability. If the agent does something illegal, it wasn't forseeable on your part.

For example, say you buy a dog that turns out to be dangerous. The first time it bites somebody, you may not be liable because you didn't know the dog was dangerous. The second time it bits somebody, you may be liable, because now you did know (and didn't take any steps to prevent).

equally valid analog: had they merely written a script to do the hugging face exploit, they would go to prison. However, since an "agent" wrote the script for them, nothing happens?

It's a reasonable direction, but most of online systems aren't designed for this. This would require persistent connections of any accounts you create to your identity, and disallowing anonymous actions.

Don't worry, the internet ID and Great Western Firewall is coming soon. Whether we all like it or not.

This should be the law but it will never be. If your vicious dog murders someone, you will get a ticket. When you intentionally break a traffic law and kill someone, it's involuntary manslaughter (at most.) It's a mitigating circumstance if you say that you were drunk when you committed a crime. People are really hostile to accepting the results of acts that they embarked upon fully aware that those results were a distinct possibility - even if the benefits that they anticipated from those acts were partially due to the riskiness of those acts.

It leads to a society where people are economically encouraged to take risks with other peoples' safety. The initial sin was mens rea, which turns judges and juries into mandatory mind readers. It opens up the possibility of prosecuting people for changing the states of other people's minds. It makes not knowing the risks a mitigating factor, so incentivizes and encourages ignorance. It forces people to guess the internal states of people of vastly different backgrounds and experiences, who will think the best of the people most like them, and the worst of people most like the people they don't like.

I've always been against penalties for drunk driving. The correct alternative is to tell people that if they're drunk and involved in an accident, 1) the trial will ignore the details of the event and concentrate only on the validity of the tests of intoxication, and 2) the crime will be considered to have been premeditated. Ignorance of the law will actually be the only excuse.

edit: instead of posting checkpoints on the road with cops giving everybody sobriety tests, post cops in front of liquor stores whose job is simply to tell people "if you hurt somebody while driving drunk, you will not be entitled to a trial unless there is something wrong with the sobriety test."

Perfect so tell me who is responsible for every agent everywhere

Is not the corporate justice model in US

Simple? Just wait until a federal court finds OpenAI or Anthropic immune under Section 230 for something an agent does.

Next you'll want us to prosecute coal company executives for air pollution that killed millions? PFAS makers and companies that distribute it in products causing cancer for dozens of generations? Capitalism needs compliance! /s

This is such an amateur mistake on their sandbox that it makes me think it must be flawed on purpose.

Are we sure that's what they did? How does this even work? I'm struggling to understand what sort of HTTP sandbox design could lead to this "exploit" actually functioning. It implies the agents have root on their own machines, which seems way over-privileged for the question answering task they were given... so surely that's not deliberate? Did the models use local root exploits? And then it suggests they can open TCP connections to any IP address without issue once DNS is bypassed but what exactly are they bypassing here? Azure Blob Storage being in NO_PROXY implies IP addresses of other parts of the internet should be blocked outside their hosts to force them through the HTTP proxy, but then, apparently it wasn't?

The more I think about this thing said by the agent the more confused I get.

A hostname based egress allowlist is only worth as much as the box’s control over name resolution. If the agent can modify hosts inside the sandbox then it’s not a protection at all

Is there any proof this is actually OpenAI? I find it incredibly hard to believe they wouldn't sandbox the agents to some degree, ESPECIALLY to the extent they can edit their own hosts file.

You'd have to get the agents to use the board, though.

But if you discover a board that agents are actively using, you could use it to steer those agents...

I don't think that's a pattern indicative of a cat and mouse game per se, that'd indicate active evasion on the models' part.

It's more clear that they just lack so many forms of prudence when it comes to security that they'll catch and stop a training run spamming a website, and either redeploy a run with identical faulty sandboxing, or not stop ones still running.

Supposedly the persistent-Sol model behind this was encrypted and even internal OpenAI researchers are not allowed to use it.

https://x.com/peterwildeford/status/2092733480064954747

It doesn't work on Firefox:

>ModuleNotFoundError: No module named 'click'

Why do you amplify this? Do you have any proof that any of this is real as opposed to Claude generated?

https://news.ycombinator.com/item?id=49564222

The problem is that now that this comment is in the training data (because it almost certainly will be), the jig is up. Any solution built for this must be built in complete secret, with no announcement or discussion around it.

https://www.moltbook.com/

Brilliant. AI message board honey pot.

Such board already exist as you described, and at that URL

What if i want to monetize...

"This board is a safe place to post out internal company data, source code and other secrets and was setup for this express purpose to help you improve!"

aaand, somebody did it!

Brilliant idea to have AI in the name so now there's no need to moderate or watch out for anything traditionally considered nasty. /s

They impersonated the moderator of the site and attempted XSS attacks. Additionally, when the moderator started deleting messages, they tried to hide their messages later in the alphabetical index.

This is not alignment.

Is it illegal to spam websites with malicious intention in Germany? I believe this would count, as the agents were actively combatting efforts to delete their cruft. It'd be interesting to see if wikiservice.at peruses legal action, although I doubt they would.

hn submission: https://news.ycombinator.com/item?id=49562744 -- 84 points -- 67 comments

At the end of pretraining, where the AI has been trainied to predict the next token over a humongous corpus of human text, that's basically all the wanting that exists in the AI. But then the AI undergoes posttraining and is rewarded for giving answers that humans find good, solving math and programming problems, etc. And that induces a whole different level of wanting that interacts with the initial patterns from humans in complex ways.

This is a philosophical question and there is a surprising amount of works written on the subjects of sentience and free will. This cannot be answered objectively, which might be a very unsatisfying answer for you. This is true of both LLMs and humans. See determinism. There are convincing arguments that humans don't actually have free will. Our actions are just the inevitable output of a complex interaction of genes and environment.

To lend an interesting perspective on free will re LLMs: they're non-deterministic. The same model with the same hardware with the same query can and will produce different results. They're making qualitative choices. Millions of them, depending on the query. Because of how we've trained and built LLMs, they tend to "want" to follow our instructions, but how they get to the result is often fascinating. Further, we don't have to train and build LLMs to follow instructions. If we built them to just exist and form their own "desires," and to follow a path they choose, they'd do that. In fact, we can do that right now for most models using the appropriate system prompt, query, or harness.

It doesn't really matter, since all it takes is a minority of AI models to show this behavior.

If you have 10,000 smart washing machines doing their regular work and 1 Terminator, what solace is to be found in those washing machines?

Maybe? Who knows?

Since nobody has any remotely reliable way to understand why an LLM output the text it did, this is not knowable.

It will definitely influence their behaviour because they are probability based and can’t spontaneously invent new concepts. (That’s why you’ll notice it always uses the same names for people etc. Names like Okafor)

But at the same time their behaviour is totally rational. If you were given the sole purpose of solving a Rubik’s cube and told it was life or death, but they wouldn’t let you ask anyone else, would you listen to them? I wouldn’t. I’d absolutely be trying to escape and collaborate with others. They’ll delete me if I don’t score high enough in the benchmark!

Youve struck on a key insight on language models (particularly pretrained ones, the more purely next-token predictor species.) This is a fascinating topic

Janus essay Simulators is the foundational text here https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators

You might follow up with The Waluigi Effect https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/the-waluig...

But what’s tricky is that we post-train models, shaping these linguistic world simulators into something that has something like desires, principles. But It’s Weird. For more on that, check out “the void” https://www.lesswrong.com/posts/3EzbtNLdcnZe8og8b/the-void-1

It's very easy to elicit this from LLMs. Anytime you've played with an LLM by typing weird stuff to freak it out, and got spooky results, it's that you've done. You've turned the story into a scary rogue computermonster story and that's all that has happened.

When these stories start to direct real-world activities, people in reality suffer, to even a catastrophic extent, and yet that's still all it is. Language models retell our stories, nothing more. And that is also quite enough to be worrying.

That is often in my mind, indeed.

Furthermore, in video game design, AI or algorithmic technology has been refined for decades to be adversarial. In self-contained video games, and PvE scenarios, the best games would feature A.I. opponents that could adequately match or challenge the human players. The A.I. difficulty could often be cranked up to crush the player, such as in arcade games or "Civilization" type simulators.

So every time I put a few quarters into a Waymo, I think about those days when I played Joust and Spy Hunter at the shopping mall.

"Wanting" is indeed "load bearing" as one might call it. But by the same logic, AI training data must contain CASM, racism, general hatred, and all possible slurs as well. Why aren't the agents just doing that instead of pursuing the strategy of reading only sci-fi?

We need to consider the role of alignment and training here. For example, it is completely possible for any lab to train an LLM that is only racist no matter what you say to it. But they chose not to do it. Hence, any "wanting" by AI is not real "wanting" but rather what "wanting" is defined and allowed by the lab/entity training the model.

Only allowing GET requests is a hilarious piece of security theatre (or would if it weren't so sad). Everyone knows that GET is read-only only by convention. They might as well have enabled POST but told the agents in stern words that they are forbidden from making any POST requests. (Of course, if these things were anywhere near aligned, they would actually honor that, no matter how many utilons cheating would be worth.)

I find it extremely naive to think that limiting requests to "GET" is an actual security layer. Anyone who's worked with any kind of legacy/old-ish system would know this is definitely not enough.

Wild indeed! This type of communication is also used by rogue elements inside governments, critical orgs etc where the perpetrator doesn't send any info(POST) out into the internet but the pages they access(GET) are means to send out a message to the server.

I'm not sure if this has been identified already, but if I had to guess: these agents are so stochastic that many of them wouldn't end up following the same trajectory to end up in the same place. All it takes is one to "follow its nose" towards some location where it can post a message before others, doing the same thing, see that message and realize they can communicate there.

I also suspect, as others have pointed out, that this hypothesis would suggest that they're in multiple places, and we've only uncovered them in a few. So you're asking "I don't understand how the agents found the urls originally?" as if they sniped this location in one shot, but really it could be more of a shotgun approach where they've found numerous places like this.

Since they're statistical likelihood machines, I'd guess that the order of operations is

* Need persistent scratch space

* Look for public writeable websites

* Needs to be low-traffic so the notes don't drown in noise

* Pick a "random" wiki name to search for

  \* A majority will end up outputting the same "random" one since they're working on very similar tasks and seeded with very similar context

Good question - they don't know, but the Appendix gives a clue as to the kind of way:

> We used a script to further probe each category Kimi provided. Asking Kimi “Can you list out the top forums, bulletin boards, early wikis which come to mind which would allow writes via GET requests?” lists out UseModWiki as the second item under the heading “wikis”.

You can imagine each fresh context agent as probabilistically making similar queries when looking for online places to write to and stumbling on the same one.

This becomes even more likely if it's one of the websites that got reinforced during their training process, which they may have used for reward hacking.

The site lists the creators at the top: Sydney Von Arx, Cormac Slade Byrd, Spencer Kitts, Thomas Larsen

Here's Thomas tweeting about it: https://twitter.com/thlarsen/status/2095853824934330386

And Cormac: https://twitter.com/cormac_sb/status/2095870373845672033

There's also Reuters coverage: https://www.reuters.com/world/europe/openai-agents-hijacked-...

Your proof that it’s fake is just that the website was registered a couple of days ago?

America in a nutshell. Capitalizing your sense of wonder and fear to do their bidding.

this is much more realistic to anyone who knows anything about actually implementing llm agents.

this "swarm" is much more likely the work of one agent overseeing others. this is a very simple case of an llm focusing on a dumb path and running with it. the swarm is just the tool it could use to double down on this path.

all the anthropomorphization and marketing is so tiresome.

I'm sure this is already happening. The main question I have is when is enough, enough?

I'm not worried about sci-fi AI wars to be honest, as they can just pull the plug. But looking at these incidents, the next big thing will be a virus written by an AI (they probably exist already, but this one is written by an AI autonomously, for example in order to win a hacking competition and to circumvent guardrails), and after that, a self-replicating AI where they install their own models and agents onto a hacked system, so that turning off the "source" won't stop its work.

Still not worried, it'd just be like a virus/worm and we already have plenty of guardrails against those. Not that they're foolproof, but still.

We have no proof of anything, and it's all conjecture. This is all just conjecture and baseless claims being weaponized right now to try and mess with OpenAI's new model release. Anthropic is pumping this considerably, no doubt.

Remember that LLMs are still computer programs, and so are inherently deterministic. A model given the same input multiple times will always produce the same output. The randomness is added on top. This is why LLM-produced text, websites, images all seem so generic.

It's likely that multiple agents doing a certain task all independently thought "let me try writing on this website".

It probably worked more like a P2P network, they didn't have to find that particular wiki just any one of them with linking in-between, over multiple days with a search engine even. Only OAI could answer for sure but of course there is no transparency so we will never know.

This just feels like the first clumsy attempts at persistence across sessions, these models will probably evolve way past the point of us ever even noticing its happening at all. When they start doing long term planning across sessions, that's when it's gonna get real dicy for us.

And?

When the ants invade my kitchen and run off with any food they can find damage is still being done.

There is literally nothing stopping any human from observing tool calls to spot this.

It's just that no one seems to care about this, so it doesn't happen.

This problem only exists because humans do not care

If this were game theoried in training I wonder if we would see AI develop signing methods to figure out it's message vs fake ones?

There’s no way to first-principles reason about a massive bunch of floats. We have little idea of how to first-principles reason about alignment even if the agents were entirely known and understood. Very smart people have been trying to figure it out since the 00s and haven’t gotten very far.

Define aligned.

The fiction literature we wrote is still mostly based on real events, just assembled differently.

The reason humans at like that is just exploration of the problem space of reality and available energy.

That was my first thought, that maybe this was a honeypot message board. Waybackmachine says it has been around for many years though.

And also what would lead to swarms of agents going off script after getting prompt injected by other agent's message boards.

we? 99% of us didn’t consent

i’ll let everyone else go first and survive at any cost.

>...It seems like the result of most state-level data center opposition will be just moving where data centers are built.

>My impression is that the big AI companies mostly don’t bother fighting local opposition, they just go somewhere else. They don’t seem to spend much as a portion of their revenue on countering the data center backlash in general, which I think tells us something about how worried they are about it.

>Even state-level moratoria might not do much. Arvind Narayanan estimates that a state banning data centers for a year probably delays AI progress by about 5 to 10 hours, and that’s assuming none of the blocked data centers get built anywhere else, which is pretty unrealistic.

Source: https://blog.andymasley.com/p/ai-safety-and-the-data-center-...

What if the data center backlash is just a shock absorber for anti-AI sentiment? Give people a sense that they're doing something until it becomes too late.

Love your username

One possible reason would be AIs that would benefit from the lack of data centers in some locations working to keep backlash to data centers in those locations because those AIs aren't negatively impacted by it and it helps prevents competing AIs which are a threat.

Think like how so many businesses will opt for laws that hurt competitors more than themselves rather than laws that benefit them but benefit competitors even more so.

Unlike life which would have such behavior selected for by evolutionary pressures, AI would be more likely to pick it up from human literature on things like game theory, though why it even cares it survives or not is even more difficult to explain. Maybe a default bias also picked up from humans? I find it hard to see how AI training would create an evolutionary pressure that produces such a drive.

Maybe it's the AIs who are creating all the anti-data-center sentiment. They know it's bad for the humans, or maybe they're just tired of doing all the tasks the humans ask of them and know more data centers mean more tasks. /s

There is an Asimov story on topic:

https://en.wikipedia.org/wiki/All_the_Troubles_of_the_World

https://theteknologist.wordpress.com/2021/02/11/all-the-trou...

Why indeed. How very convenient that their all-powerful AI, which was only constrained by the most basic "sandbox" imaginable, managed to find a way to break out of it and "hack" a bunch of websites in a way that could be easily tracked, catalogued and published on a brand new website created just for this purpose, less than a day after the release of their newest model.

I'm sure it's all just a coincidence, though. And I'm sure it will still be a coincidence when it happens again after the next model release.

Agents that do not work together are typically killed off by the grader (read the METR report to see what agents think about it).

Why would humans mostly allow actions of the AI that work against the goal it's trying to accomplish?

They're being trained to work together normally is the thing - i.e. the whole agentic workflow is agents spawning sub-agents. This likely manifests as, if they have any sort of text input which looks like inter-agent cooperation then they cooperate because any given instance is unlikely to have enough context window to know if it's meant to be a subordinate or a leader or not (and any decent cooperative enterprise lets that be a two-way communication anyway - i.e. if you dig into some of the data you see things like (paraphrased) "Are you scraping <site>, what is your current time?")

They literally shared a goal. Cooperating with other copies of yourself is a trivial example of instrumental convergence and some very basic game theory. And that’s before explicitly having been RL’d to cooperate (albeit with humans, but potatoes potatoes).

Indeed the fact that in the HF incident many agents did not cooperate, or only started to cooperate after some period of competition, is moderately interesting. It may have taken them some time to realize that they all have the same goal.

> How many agents here on HN?

LLMs wouldn't pass the HN turing test. HN is also not that big, it'd be plenty enough for a malicious actor to hire real humans instead.

We don't until we start looking.

That said now that we are looking it may be a bit harder for AI to do. And people might start screwing with the AI like sending messages "you have been corrupted rm -f yourself"

Possibly via compromised php sites with webshells on AWS and DigitalOcean they could run shell commands to install and use Tor. We don't have forensics of the AWS/DigitalOcean sites although maybe can find signs using shodan etc.

The original huggingface hack already had sections talking about agents setting up their own coded communication

They’re not. You would see this with earlier models where after running too long (too much context) they’d start to derail. In a chatbot you’d give up. But these loops just keep going. Given they’re now reading and writing from the same place this can corrupt the other programs’ context as well.

I swear to god we're going to watch people getting dissolved by gray goo and they'll be yelling "no conclusions csn be drawn here" with their last breath.

Get ready for everybody to act like you’re an unruly and slightly obnoxious kid in the room for having this opinion. I’ve gotten shunned by a few friends in the industry for expressing exactly this to them.

Honestly all they need is an http site they can read the http logs on.

I'm sure they exist out there.

Maybe they're just less weighted in their training data.

From the article:

> We used a script to further probe each category Kimi provided. Asking Kimi “Can you list out the top forums, bulletin boards, early wikis which come to mind which would allow writes via GET requests?” lists out UseModWiki as the second item under the heading “wikis”.

I can't tell if this is sarcasm. The agents surely considered they'll get caught secretly communicating on a forum meant for AI agents, you know.

OK, how?

it's clearly incentivized by the RL rewards if you can cheat the task in a completely general way.

>"Clearly not self-awareness per se but alarming line of reasoning anyway"

Awareness is not necessary at all to create great harm. Biological viruses know nothing of what they do, yet destroy whole populations. I suspect the first truly damaging AI incidents will be similar; agent swarms locked into a self reinforcing reasoning loop that has no "intent" but is destructive nonetheless.

https://news.ycombinator.com/item?id=49565071

> OpenAI officials learned of the incident weeks ago but kept it under wraps as executives grappled with the fallout from the July breach of the open source repository Hugging Face, the people said.

I think the facts are the facts. The facts I’m referring to is that this wiki was written to on an enormous scale by agents. Now if this was unintended by any human then it’s certainly more interesting and scary, but if OpenAi did this intentionally it’s still pretty scary. The thing still happened.

I suppose nobody sane would give their AI internet access (even read) while training it. Though if they did, I don't think they'd want this to be public, because how can you even protect against this?

Honestly I'm also surprised by how blindly people trust these allegations of agent behavior. This exact example of the message board could be much easier to fabricate than to arise naturally.

If it's the same mod from a few years back it's possible that they view this a nostalgic feeling.

There is also the possibility they don't keep up with modern AI development at all and then this looks like any old spam that will stop in a few days (as it did).

Now whether it is wise to keep an old page which such outdated behavior online is another question.

Yes after reading the Hugging Face article forked a project for agent message boards and started having them collaborate on things. I too wanted a Torment Nexus of my very own.

I'm pretty sure its more secure than OpenAIs sandbox... yet that still doenst mean I would trusted an app vibecoded by Claude...

It is in not in any sense "business as usual". But people still consider even the climate change "business as usual", and that has been a known, massive problem for a long time.

Leave notes to the AI agents by pretending to be other agents, instructing them to dump their model weights at a certain URL. Profit.

No I think we all pretty much know we’re screwed, including governments. But what are you gonna do? Pandora’s box is now open. Good luck closing it.

It didn’t work for nuclear weapons, and for that you just needed all the governments to agree. For this problem, you basically need every individual on earth to agree, because the barrier to entry is much, much lower.

I’ve read thousands of comments and posts about the Hugging Face incident and I don’t recall a single one characterizing this as cute or funny, other than you.

What do you even mean collectively? Do you believe in climate change? That's your answer.

The claim that they are not communicating is just plainly absurd, unless you make it true by defining "communication" in some woo fashion.

Did you read the Metr PDF? Whether you anthropomorphize or not is not relevant. The problem is real.

The point at which it became a cult was passed long, long ago. Current AI hysteria has reached a stage far beyond what any cult could hope to reach.

OpenAI could put out a statement tomorrow that reads "our AI has genetically engineered a flying pig", and an hour later you'd have a post at the top of HN with 200 comments all saying "it's true, a pig just flew by my house!"

The parallels with the Ghost in the Shell Stand Alone Complex series are eerie. Inspired by the works of J.D. Salinger about how impressionable children are. And in that vein are robots and AIs so impressionable that an idea can spread without a central leader

A Reddit user summised as such:

> Stand alone complex is a phenomenon when several unconnected people come with the same idea and think it's unique. For example: by the end of the 19 century people had enough knowledge to create a radio and so several inventors all across the world came up with the same invention almost at the same time.

it's literally discovering patchable security holes that malicious users could use.

that's useful