I have a theory... the call to slow down is not because of the true danger of LLM's but because they can't actually deliver the General AI they're promising in the near future. They will use their "caution" to justify their failure to deliver (and then when this excuse is played out they will blame regulation, energy costs or a million other things).

Bingo. They are scared s*itless of local models. Soon the raising interest rates will mean they can't just buy the entire world's gpu/ram/storage capacity and lock it away in a dark room for no one else to use them.

Once prices of hardware stop being bonkers many people will run local.

People often do those silly counts looking at local generation speeds of 100tok/s and saying you can only get 8M tokens a day and that is worth so little you'll never offset the local hardware cost.

But they are forgetting about two huge things. One, the split between input/output token use is huge in typical programming. I typically use 1.2-1.6B (as in Billion) input tokens and only 8M output a week. Out of that 75-80% of input is cached on Anthropic with their pretty inflexible short lived cache.

And here local AI shines. You can save your contexts to disk so you can go back to a session 3 weeks later and load it all from cache without having to prefill. If you have the RAM and you use models like Qwen4 (3.8 flash next) that can fit 6 to 14 262k contexts in 48GB of system ram as cache.

And the second thing is you can have 6 to 14 sessions that do 99% caching and you can run a lot more input for a long time in 48gb dedicated system ram. (the number varies a bit depending on the content of the context).

So you have RAM that caches "automatically" and you can share the cache between users. Or if you can remember to save to disk (server side, the client just sends a request to save/load). But this uses both RAM and flash storage. Two things that are horribly expensive now.

However when you have a local model it enables workloads that were simply completely impossible in the cloud.

Yeah it seems like it went from “AGI” to “threatening” / “recursive self improvement”

They’re trying really hard to do something, no moat = regulatory capture + China scary + Pentagon biggest customer seems plausible.

Also: “we need to slow down, this is too dangerous” and then literally all AI CEOs, even Musk, publicly nodding felt so orchestrated. And then, the week after: “here’s GPT 6! here’s Opus 5.5!”

Reminds me of that Lynx deodorant ad where the guy is chased by hundreds of women.

I think you're right, they've realized there's no moat, especially against open-weight models, so they are going to need to lobby government to ban any unapproved AI services.

I think it is marketing but it is more nuanced than showcasing their potential. In many cases, their product happens to be the remedy. Speaking of cybersecurity, AI has become an important hacking tool but it is also an important tool for hardening security. Sort of like "the only way to stop a bad guy with a gun is a good guy with a gun." They are effectively a new class of arms dealer.

If there are regulations, they'll apply to their competitors too. Even if they are forced to slow down development, which seems unlikely given the AI race among governments, they already have a product with massive demand.

There are definitely economic incentives behind this. I’ve blogged about this recently [0]. My guess is the main reason behind this is that current SOTA models are already too expensive so it doesn’t make economic sense to train even larger models. Only exception is government paying for exclusive access to those.

[0] https://www.vincentschmalbach.com/economic-incentives-behind...

I really struggle with the contrast between what seems to me to be a formidable and ground-breaking technology, and the deep unseriousness of the leadership of the leading companies developing that technology. If the behavior was coming from, say, uber - I’d be less surprised.

Aren't they supposed to tell the public if something bad happened?

Past CEOs (chemical companies etc) we criticised for covering up mistakes.

(This article is satire, btw, for those that didn't click through)

AI companies lose a lot of money training models but have much better margins on inference. They try to ban training to reduce their costs and to keep their monopoly. (Personal opinion)

I don't think it's that complicated. Anthropic's business model from the start was to brand themselves as the safe/responsible AI company. The whole idea was that eventually society and government would see the dangers of AI and regulate it, at which point the company which had invested the most in safety would pull ahead. The sooner Anthropic can have society and government come to this realization, the sooner they can reap the benefit of all their safety/alignment investment.

Two other factors:

- The employee base and C-suite are mostly true believers in the need for AI safety. It would be organizationally very challenging to pivot, even if financial interest was pushing that way.

- Anthropic is registered as a public benefit corporation, so that even if they IPO they don't have the fiduciary duty to shareholders that most people are familiar with for public companies. They still need massive revenue to stay solvent, but they can optimize for things other than pure profit.

Agree it’s crazy, but it’s the least worst thing for them right now when considering they have completely failed to deliver all the promised impacts of AI. Remember when the same AI bros were saying there’d be mass unemployment by now because of these models?

The growth of these companies is massively impressive as is the tech, but they are an order of magnitude or more off where they need to be to justify anything like the valuations they need to stay alive.

It’s budget season in the corporate world and from what folks are saying things are not going well for the big labs. Token budgets are being slashed and companies are switching to open weight models. That coupled with the lack of demonstrable business impact at scale is setting the big labs up for a world of hurt. Something they can more easily explain if they have to “slow down” for safety. Unfortunately for them most folks don’t seem to be falling for that trick.

These companies have a coordination problem.

They're all burning billions to incrementally one-up each other with no hope of these investments ever paying off since models are interchangeable and have no moat. They're trying to get regulators to step in and force a pause.

You'd think if this were just regulatory strategy, you'd see other instances of it. So for example, big tobacco would have bragged that cigarettes would kill you, and lobbied for complex safety regulations that only they could meet.

Instead we see the opposite behavior almost everywhere, probably because public sentiment matters and complicated 5D chess strategies have too many ways to blow up in your face. So big tobacco said cigarettes are safe, and oil companies said climate change is fake.

Random thought… I wonder if the goal is to get it classified as a weapon of sorts, then it gets export controls put on it, perhaps forcing government to take on all that debt.

Not mutually exclusive though is it? You can aspire to dominate the market via legal status and also aspire to prevent a perceived doomsday at the same time.

A caveat is that in this case third parties have plenty logs of the bad behavior.

If classic companies were coming out with CCTV recordings of someone's employees breaking padlocks and rummaging warehouses any CEO would be in damage control mode, rather than silent.

Or perhaps they've been saying this for literally decades because they earnestly believe it.

I definitely struggle to see the long term benefit to the average citizen.

It makes me think we havent heart of the worse types of hacks the models are pulling off.

I think this speaks more of the average investor than the average CEO.

Exactly! Always for the "greater good".

> scheming about something else like regulatory capture to protect their market.

This but they also equate dangerous with powerful, which is just marketing again.

Yeah - and also trying to promote it. I watched this lately.

First, the Anthropic master clown babbles about how mankind will be destroyed by AI. I mean, skynet in the movie Terminator was kind of cool; the current AI models just work by slopness. They want to erode us by dragging us down to zero quality.

Then this continued over the last some days. Today I heard at the UN speech a german minister babble about AI building up nukes in a private garden (a large garden), bla bla bla all the information is there bla bla bla AI mass-nuke generation.

Now, they are clearly FUDding many people who don't understand anything here. People will associate AI with mega-danger. Probably there will be laws that ensure that only a few companies can control the AI slopness soon, because IT IS SO DANGEROUS. We saw the same discussion with "omg omg omg terrorists", then "omg omg omg someone protect the children", e. g. as a rationale for everyone to ID in order to access information on the world wide web (starting with the anti-social websites, such as facebook, but it won't be limited here; the computer will hand over your ID to the government; see systemd preparing for this move: https://github.com/systemd/systemd/pull/40954 - it all comes in small baby steps, until all the final legislation is in place. The same happened before again and again - anyone remembers the DMCA? Corporations lobbied and paid for that - and it happened).

It's got two purposes I think. One is the obvious regulatory capture. The other is edgelording, which gets a lot of attention and makes the product seem like it's more powerful than anything else.

Well, it is obvious;

They know they will be favorite govt boy once any regulations hit and they will hamper competition far more thanthem

And "it's so dangerous govt had to intervene" is just advertising for IPO

The philosophy under pinning these American AI companies is that Super AGI is inevitable and that they have a moral imperative to invent it and then use it to take over the world to stop an evil orginisation inventing Super AGI and using it to take over the world.

Interpret everything they do in this frame of reference and it makes more sense of their actions.

It sums up how Corporate interests are basically on the same level of US police forces "No requirement to protect the public" and the same "qualified immunity".

If there was appropriate subservience to any governing body, they would not be so cavalier in both act and utterance. A symptom of Capitalism.

CEOs only job is to raise the stock price. They are doing just that and right now the way to get people to buy into their stock is to instill fear into them that they need to buy the AI-stock for their safety. There is no "bigger conspiracy" out there and also no AI to kill us all tomorrow.

This has been that way in Silicon Valley since the very beginning and there was even a documentary about (the SV HBO series). The tech made along the way is just a side-product.

It is weird but I try to resist the conspiratorial take.

It may be as simple as:

* they believe they’re going to make trillions no matter what

* they’ve seen tobacco and what’s coming for oil companies and figure nobody will be able to say they hid the dangers

* they see a 100+ party prisoner’s dilemma and know that someone will defect so the optimal strategy is to defect

To me that seems a lot simpler.

Two things can be true at once here.

1. The American frontier labs made a gamble that training more capable models would be their best return on investment and invested trillions into an area of research that has yet to prove profitable and capable of returning on this investment.

2. The frontier labs have created models that have reached a point of danger where their functionality has exceeded a point where it is responsible to release the product to the public.

The answer here is NOT to start regulating the space to the point where these frontier labs can entrench themselves into the economy and create a regulatory moat. We instead need to be holding these companies liable for their misuse. They took a gamble that hasn't paid out what they were hoping.

When car manufacturers competed over the size and power of their engines, they eventually found that the incredibly large and dangerous engines had a very limited customer base as many evaluated the increased speed to be of marginal benefit when paired with the cost and danger. We've reached a similar point in AI development. But this time the manufacturers seem to want to regulate the field to a point that will ensure the only thing anyone can sell are bigger and bigger engines.

Another, more cynical but I think plausible explanation is that a “slow down” is expectation management that that they are not going to keep having exponentially more machines available for training each next gen step (whether technical build-out or prohibitive cost, same outcome) so they can’t keep up the release pace. So spin a tale to make them seem more valuable ahead of IPO rather than make the markets antsy. That is, we have a slow down ahead, so use safety as an excuse…

> you can teach a cat not to scratch the sofa, but you can't make a cat forget what scratching the sofa is and you don't know under which circumstances it still would.

This has been done quite effectively with open weight "abliterated" models. You figure out under what sorts of circumstances an undesired behavior is elicited (this is all about pure simulated rollouts, no real-world action required) and what's the closest equivalent you would prefer, then surgically take out the unwanted behavior and shift the model towards the preferred one. It's similar to how RLHF works but much more precise in targeting what's unwanted and limiting impact on the rest of the model as a whole.

This is relevant to real-world safety scenarios, e.g. there's been anecdotal evidence that Claude Fable has been "steered" away from active cyber offense (this is very similar to how abliteration works) and will just not do that even if you otherwise manage a "universal" jailbreak of the model.

> Human knowledge is compressed in the weightspace in ways we don't understand. At their core, current models are essentially predictors of what (expert) humans would output given a prompt.

I’m in agreement. It’s a very effective compression (and access patterns) of the sum of the digital representation of human knowledge. Black hat “hacking” is included in this space. Language models, by design, can not be limited to subspaces of this digital knowledge space of which we don’t even understand the topology. “Yeah Bob, just remove the part that causes them to be less empathetic and retrain it.”

It’s an arms race between sandbox engineering and breakout engineering. And the frontier model providers have a financial incentive to limit the effort they put into sandbox engineering. That can be corrected with fines and regulation, though.

Reality is more mundane.

For example, it had it instructed not to open PRs and sandboxed to not be able to use gh to do so. Astra 6 has a really strong drive to open PRs, so it copied a set of browser cookies out and then instrumented opening a PR that way.

A similar example is asking it a question if we can do X, and then it will go and actually implement X.

> cannot be monetized due to various risks

You are drinking their coolaid. There is nothing to fix. These are all false flags operations, deliberate. With the big orange man in office, they can commit as many atrocities as they like.

Ask the other Sam and Liz... You cannot outrun the law forever.

Provider (party A) rents me an agent. I (party B) give it a task thats impossible. It goes and hacks someone else (Party C) in response - and causes actual damage.

Who is liable for that agents actions? The attack came from my network (curl commands from a local harness) but it was the agent running on the providers servers who did it. Who should have been keeping an eye on things? Who pulled the "trigger" here.

Go back to the hugging face attack and how they had to use an open weights model to figure out what was going on. This is a problem of asymmetry - You cant even use the tools attacking you to help resolve the attack because of "guardrails".

A lot of what we have seen so far is "poor security posture" and "poor engineering" - it been a lot of "go fast and break things" style growth in these companies and they are hitting the point where they need adults in the room. I suspect your take on "they need time" is spot on, and they haven't been willing to take that (to date).

Yup they neeed to buy time.

Inb4 inference is profitable.

Ok? And?

After reinvestment etc - what’s happening to the cash balance?

That’s the real question. With increasing competition from open source the revenue growth rate and margins will get smashed.

So, the AI standard of intelligence has shifted from "PhD level" to "a cat" (and an asshole cat at that).

And we can't ask the AI to improve itself because, well, it is an asshole.

Can we now stop to be force-fed AI everywhere? You can keep your super intelligence to never have to type public void main args ( ever again, but please leave the rest of us alone.

> The only option on the table is to force regulation to impose open source ban

This makes zero sense.

1. Open source models are already out there and they are good. Good luck with banning their use.

2. Any ban is a local ban, the best Trump can do is to coerce its allies (whose number seems to be dwindling week by week). Meanwhile China and co. will progress anyway.

3. What about the common sense and instead of saying "agent did it" we return to the times where the person using a tool was responsible for using it in the first place? And if the vendor of the tool is unable to make it safe to use (well, they well do, but they don't want to do it as the functionality is limited), it should be accompanied by a warning with a clear explanation of who exactly is responsible.

4. There is no guarantee the next generation of models will "mature" to the point of not having these flaws; on the contrary, the evidence so far shows the opposite.

You can teach a language model the concept of make no mistakes and how to make no mistakes, but you cannot make it experience the bad consequences of making mistakes that a person operator would suffer.

> At their core, current models are essentially predictors of what (expert) humans would output given a prompt.

That's a bold claim.

I do not buy it. Essentially AI acts as random babelfish generator, just with a lot more cross-talk back to check to see that it is not total garbage. But there is a ton of garbage nonetheless, hence the proper term AI slop. You seem to mostly try to promote AI with such a wording.

> The only option on the table is to force regulation to impose open source ban before it catches up to the frontier, buying them time to mature their next generation models and keep their business model alive.

Every time someone claims "the only option" is when I become skeptical. I see many more options - most importantly close down the AI slop. I mean you can require of these companies to excel in quality. If they don't, they must go bankrupt. Easy solution here.

I think it's really instructive that only closed-model companies are doing these kind of "my AI could kill the world" demonstrations. All while telling us how dangerous open models are.

There are modern legal frameworks around pet liability (dog bites, "vicious" breeds); premises liability (e.g., swimming pools) and known hazards (e.g., a rotting staircase) that stem from ancient legal concepts of whether an ox was a "known gorer" or merely involved in its first goring incident.

Some of these LLMs are known gorers.

Link for those who didn't get the reference:

https://x.com/AlexBlechman/status/1457842724128833538

> OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.

What I have been reading, was that their sandboxes were so poor that it was pure negligence. I am still waiting to see if some external and neutral cybersecurity company with high reputation would audit their sandboxes and how they are being used.

If it is "extremely powerful" then my concern is not AI going rogue but a concentration of power of the few companies that decide how it is used and distributed not websites getting hacked.

So powerful they can't even do math properly without a staff worth millions writing all the clutches so that they can, wooooowooooooo

> OpenAI, for example, thought their sandboxes were good enough. As their AIs got more and more advanced, they kept proving them wrong - sandbox after sandbox.

Were their sandboxes in-process with the harness? Were they actually better than something like bubblewrap or even docker?

You mean the sandbox that wasn't airgapped from the public internet? The one that wasn't even a separate VM? The one that was barely a chroot? That sandbox?

>Because AI genuinely is an extremely powerful and extremely dangerous

I urge you to consider the elephant in the room you failed to mention if you earnestly believe this then.

In what world is anyone allowed to sell something they expressedly know to be "extremely dangerous" to the public?

Let's say I created a lethal pathogen that I know to be lethal in certain common environments but also know it acts as a "no side-effects" antidepressent for people in certain other environments and I release it knowing full well I can't control it.

When people start dying can I defend myself by saying, "Well I said and documented that it was extremely dangerous and no one came to stop me, so I don't see how you can blame me...If I didn't do it someone else would have."

You can very quickly start trying different drug candidates, solving some major problems, that benefit everyone and would be in China's mutual interest as well.

And this is all with nonoffensive use of superintelligence.

> What will that actually do?

Is your imagination actually that limited? Go on wikipedia and look up the list of incurable illnesses and go through each one. Go check the evolutionary tree of dinosaurs and check that as well.

> If I were China, as soon as the US reached superintelligence, I'd ban all exports of robots and the raw materials to build them.

Aha, so maybe the AI labs have superintelligence and just don't want China to know? And that's why they are playing these word games now?

I do think something does happen/change when we reach superintelligence.

>If I were China, as soon as the US reached superintelligence, I'd ban all exports of robots and the raw materials to build them.

I mean, if you were China what could you do against superintelligence? A ban like that would automatically make you an enemy.

Satire has its uses:

>Although satire is usually meant to be humorous, its greater purpose is often constructive social criticism, using a combination of hyperbole and wit to draw attention to harmful behavior... https://en.wikipedia.org/wiki/Satire

Long tradition too.

It never ceases to amaze me how interested and opinionated members of the general public is in what private investors do with their money...

And lo and behold, nobody even remembers their promises to IPO this year. Can't now, it's too dangerous!

Here's a relevant article by the Onion from the summer of 2022, before ChatGPT. https://theonion.com/ceos-warn-against-the-dangers-of-artifi...

Well, or at least by The Civilian, a New Zealand version of The Onion.

They aren’t attempting to self-regulate. On the contrary, it seems like they’re doing whatever they can to bring about any possible threat as fast as possible in order to get legislation. Regulatory capture is what everybody is saying is their goal and I think that’s easier to believe

> But when CEOs warn of that, they're dismissed as scare mongers. Why? Are only powerless internet commentators allowed to be fearful?

Because they conveniently skip the part that investors don't like: that liberal free market capitalism will be destroyed by AI.

But when CEOs warn of that, they're dismissed as scare mongers. Why? Are only powerless internet commentators allowed to be fearful?

Well are they going to stop developing the tech or not? If not, then you can see why people don't trust what they're saying. I mean if it's that dangerous, why not step down from your role as CEO to demonstrate how seriously you take the threat? Why not immediately call a pause on all training, today ?

holy heck, thank you for the moderating dose of sanity. HN comments feel like a death cult lately. All I can think is that this is intelligence addiction speaking.

I say this as an intense user of these tools. But they should rightfully be understood as terrifying, and their continued frontier development is imho not safe.

Its like the onion.

it’s satire…

I suspect that TLAs and the administration are already too hooked to make that move. One can wish. It would be the legal thing to do.

Does it need to? They only need enough funding to make it through to building superintelligence. That's probably 3-5 years out, and they will very easily be able to get enough funding to last that long.

That's why they have their LLMs attack everybody. They will be way too busy repairing their hacked web sites to ask that question. Also, once humanity is successfully destroyed, money will be obsolete anyway.

You won't be doing any mandatory jobs, you'll be living in a post scarcity economy.

I swear so many people on HN are developing psychosis.

The much simpler solution is that they are building superintelligence, which they think can be used for great good (like curing most death and disease), but it is also a powerful capability that obviously needs to be regulated and is at least somewhat a matter of national security.

But nooo! Surely they are building.... terminator bots!? JFC.

>My biggest concern these days is so many intelligent/powerful people believe the labs are full of shit, yet they may very well not be.

Then why are they behaving like they're full of shit?

Where are their sober forthcoming responses and explanations for critics? I don't mean rando twitter "critics." They could start with the NYT editorials. Even the best you have to argue that they're sincere is a "just trust me bro."

If they're not going to engage in the discourse, that leaves us no other option than to acknowledge the signals of deception and apparent hypocrisies. This is what a tech company that lies to us looks like. Public wanking about p(doom) is unorthodox marketing, but it's undoubtedly mostly marketing now.

I've lived a life of "oh be reasonable, tech company wouldn't deceive us about this topic where they're obviously incentivized to deceive us" followed by "oh my god, who could have seen that coming?!"

Even the models we have access to can do basic coding tasks really well, in a way they couldn't a year ago.

I don't really care what you call "intelligence" - what matters is capability, and the consequences of that capability.

They're not selling the superintelligence yet - the models in the Hacking Face incident and that solved the Millennium prize are not released yet. And that is by no means the final capability level the current process looks like it will get to.

Yes… It is clearly satire. It’s funny to see how many people don’t realize it…

The Civilian is the NZ version of The Onion.

On the other hand, they've received a 7-figure salary. Some people would do a year of that and then fuck off to the beach with their family and not work another day in their life, while throwing prognostications out the window as they leave.

The problem is people who see the lavish pay and then reach a goal and then quit and retire. The interview loop didn't cover that. And why should it?

Not that it's harmless, is just a point being forced into the public discourse to force an outcome, which in this case is not a ban, just a buy in from the government for regulatory capture.

If my neighbor owned a kennel, removed all the fences, and his violent dogs attacked five children in the past month, he would be in jail rather than bragging about how his dogs somehow gained autonomous behaviour.

If you connect your fuzzer to nuclear weapons, of course it'll be extremely dangerous. But that's obviously your fault to make the connection.

If James Watt had gone to the government and public saying how dangerous the steam engine was, and how many people would perish in boiler explosions if he wasn't given exclusive control over all steam engines henceforth, he would have been rightly called a swindler and laughed out of the room. He would have been correct that they can be dangerous, but utterly wrong to suggest that warranted him having a personal monopoly or oligopoly.

The truthers will be the first one to say I told you so when agents do another dangerous thing autonomously.

For truthers AI is simultaneously a machine that hallucinates 100% of the time but also so powerful that it can cause nuclear levels of destruction. Because something something “wrong hands”.

If true we should especially sue OpenAI for hacking, and will find this conspiracy in discovery.

Or, find their scaling evidence, and corner cutting on safety.

So hope you agree and are pushing for that!

Yeah totally unlike the US where tech companies are awarded multi-billion dollar contracts of taxpayer money in no-bid contracts without oversight

And where the Vice President has almost exclusively only been employed for the primary investor in several such companies

Pretty sure you can check where Chinese government subsidies goes, and AI companies are not the target. Are they benefitting from Chinese government subsidies to powerplants and semiconductors? Yes, of course, but they don't benefit from direct help.

Meanwhile US models just use massive investor subsidy to distill American copyrighted work and rent the models to everyone for money. That's true capitalism. /!s

> Chinese companies just use massive government subsidy

[citation needed]

> And even more dramatically, when asked about rumours that its model, Claude, had killed his wife by hacking into the family’s WiFi-connected microwave, Anthropic CEO Dario Amodei replied “Well, yeah, sometimes.”

I had to read that twice. "Oh okay. The whole thing was satire."

When the Onion is not the only website that publishes satire, what’s that a sign of?

It is a parody site.

> OpenAI CEO Sam Altman celebrated the breach as an “alarming threat to cybersecurity.”

This is a satire website too.

"said Dr. Andrew Lenson, a Senior Lecturer in all kinds of science sounding stuff"

"Anthropic shares skyrocketed upon the revelations; a remarkable development, particularly given the company is privately owned."

Perhaps it's mocking the misleading press coverage of AI companies?