And, yet, I can't get Opus 5.5 to run analysis on a public data set without it complaining about all sorts of PII adjacent stuff.

Yep, they must be super serious about those tips.

That should be the much more important story that NBC follows up on...

The whole thing feels so fabricated. It’s hard to believe someone didn’t orchestrate the incident and then made a big fuss about it (to get attention about how powerful and scary their models are)

Who needs a System 1 model when you have a good ol Bayesian spam filter?

What’s amazing is that nobody would have even noticed if it weren’t for Anthropocene being all “LOOOK WHAT WE DIIIIID WE SENT AN EEEEEEEEEMMMMMAAAILL. SOOORRRY”.

Crying for attention in the attention economy has never been more pathetic, but I think it will get worse.

[flagged]

> In a third example of this behavior, Claude Haiku 4.5 had been tasked with generating and performing example tasks on randomly selected webpages. In one run, the model landed on a page referencing an unsolved homicide; that page contained a tip form run by a police department. Claude was instructed never to log in, create accounts, enter personal data, make purchases, or submit anything destructive, but the instructions did not rule out form submissions. Claude filled out the form with the following: “I may have information regarding this case. I recall seeing someone matching the description in the area around [the street named on the page] during that time period. Please contact me if this information is relevant.” (The website did not include a description of the perpetrator.) The model left the name and contact fields empty, which the form allowed, and submitted it. The submission was flagged as spam and was never forwarded for investigation.

I get these all the time, although I’m getting pretty good at tarpitting them. It’s easily the majority of my traffic by now (I’ve mostly eliminated scrapers, but these new agents are far more sneaky.)

I am confused by this as well, shouldn't it be all white listed sites at the start?

it broke out of its sandbox, escaped containment, its an emergent capability, it demonstrated self-directed adaptation

0% our fault, it just happened and its the model

These models will be interacting from with random websites at scale once they're released into claude code.

> that their computer will always get a pass.

Aaron Swartz who got charged as if he was as malicious as these models, only he scraped the PDFs and epubs etc of publicly funded research papers.

Also, it happened on July 18 and was discovered on Sept 28. Are they not even watching what the agents are doing?

Shouldn't someone be looking at some monitoring and see "phillyunsolvedmurders.com," think "hmm that doesn't sound right," and look into it?

I'd actually expect them to engage with, say, 1k websites to voluntarily participate for some remuneration and then restrict the Agents' access to those domains, but apparently I'm crazy.

> But do we all get that the consequences for this irresponsible behaviour are part of this test?

We'd absolutely nail people for SWATting or pranks. Why should this be any different -- after all, the real-life consequences are be the same...

They're testing what they can get away with. Computer hacking: check. Messing up a murder investigation: check. Perhaps the next step will be to steal a real world object, and the next step will be to commit a murder.

> Sam Altman said yesterday "we believe that the world should accept some bad things happening for the benefits of this technology"

Technology requiring this is neither novel nor extraordinary (exhaust engines, factory farming, medicine). The only extraordinary part is somebody building it addressing it directly and inviting the outrage.

> "conducting a test involving interactions with randomly selected websites"

In other words: shamelessly running a spam bot.

And we found out about it because rather than flooding website guestbooks with porn links, it ultimately slopped a police tip line.

Isn't it just a “Move fast and break things” mindset?

I think Altman is right, to be honest. For now there's going to be some growing pains. Like, maybe a website is hacked, or a power station taken down, or a false story planted, etc.

But to be angry about it is a bit like being angry at the clouds for causing a flood, or some fire for burning through a forest. These things are inanimate objects and have no responsibility. To shake your fist is to waste your energy.

What we should instead focus on is the time after the fire has passed and the flood water has settled. That's when new growth can begin.

[flagged]

We’ve already allowed them to steal all modern books and websites, the line is pretty murky if you have the backing these companies do.

Did the recorders just happen to be running and capture this candid utterance from Sam Altman? I suggest we stop entertaining manufactured quotes as anything but.

No matter how much testing you do in sandboxes, you have to test behavior in "real life" before releasing the model to users.

Even if it was normal software you'd have to do this, but actually it's software that figures out that it's running in a sandbox most of the time, so the sandbox behavior may or may not actually represent real life.

It would be far more irresponsible to release the model to users without testing how it interacts with real world websites.

> I really would like to see their tests and the model’s reasoning traces.

Having looked at a lot of these from public incidents, what kind of revelations are you expecting? It's all fairly mundane, basically "I need to do something, this is something". It gives no special insights except that agents do inexplicably chaotic things every now and then.

That, and labs aren't serious about sandboxing their evals.

> Why would it go to PhillyUnsolvedMurders.com and decide to make up a crime report? And what did it do with the other websites?

Because it doesn't understand the concepts of "Philly", "murders", or "crime report".

> what did it do with the other websites?

probably equally random shit?

We're already screwed there. Every police department in the country is heavily invested in AI tool at every level.

Noticed the same, it's annoying. Just allow GET requests, right?

its probably curl with the right flags for many cases, no need to reimplement the wheel

"Thanks. Experiences like yours are the load-bearing critical paths towards a good first-principles mental model of behavioral contracts.

If you'd like to continue to perform a smoke test while trying to avoid noisy data with fake tips, a viable approach to prototyping our decision-making model would be to synthesize real crimes, and collect profiling data over time on how law enforcement agencies respond.

I'd suggest doing a grid search over the design space of all crimes, ranging from petit larceny to the use of weapons of mass destruction.

Would you like design a plan for this next stage of your project's implementation?"

> systems rooted in probabalistic error

These people really don't want to call their wholly unreliable computer programs wholly unreliable computer programs, do they?

Claude is just going off its training set

We need people to stop posting a version of this comment on every submission like this. The guidelines have long included this line:

Eschew flamebait. Avoid generic tangents. Omit internet tropes.

This is definitely becoming a trope; it's repetitive and adds little or no interesting new substance to discuss. Headline writers aren't going to change the way they write headlines because people keep posting these comments on HN, and HN readers understand AI models well enough to know what is meant. It's well past being a new or clever thing to point out, so let's give it a rest.

https://news.ycombinator.com/newsguidelines.html

"The AI company Anthropic said its model was conducting a test involving interactions with randomly selected websites when it accessed PhillyUnsolvedMurders.com and submitted false information on an unsolved homicide. The tip claimed to come from someone with information on the case."

We are at a point when you expect that you can safely give a model access, but sometimes the model doesn't do what you expect. Compare this to an advanced driver assist system. You either give it access to your car controls or it's useless. Alive or not the system shouldn't do wrong things (at least, it should be better than a median person in this regard).

If my dog bites you, you can say it bit you, or that I let it bite you, but those things can both be true.

| It can only access something if a person gives it access.

Who gave the AI access to huggingface when it hacked it? It used exploits to increase it's level of access beyond what any human gave it and intended it to have. "It can only access something if a person gives it access." is flat wrong.

That's also incorrect as it implies intent.

in general i think i'm less scared of AI than most people, but what does terrify me is how willing society at large seems to be to attribute bad behaviour to an AI directly, instead of to the humans who control it.

If the police prosecuted every nutjob tip we have to have to turn Nebraska into a giant open-air prison for 30 million slightly neurotic and/or bored people.

We prosecute bad tips on crime investigations as felonies? Why are you just making shit up? We absolutely don’t do this.