I don't really get how you stop that though. How do you even quantify the intelligence of these models? Right now that's mostly by benchmarks. You could just make it fail benchmarks while do well at internal benchmarks.
You could also very likely optimize the current models to be a lot more energy efficient, that'd be a win even if they didn't become more intelligent.
But none of that stops anyone else from researching or improving their models. Or a nation you've told to fuck off from doing so as well.
As has been said in the past, one doesn't put the genie back in the bottle. We only really "stopped" researching nukes because there was diminishing returns. Though one could posit we stopped because simulation became good enough or a myriad of other reasons.
When you're talking about nation state weapons capabilities I don't think you stop, but ai is even easier to share than nukes, it's just a few gigs of numbers versus heavy dangerous materials that are very difficult to source and make. Every gamer, mac owner, etc has the equiv lent of a centrifuge on their desk. Not everyone has a centrifuge on their desk.
Yes, this is currently a crisis we are in. (At this point in time.)
Everybody remember MAD, mutually assured destruction? That too is a crisis.
We are in a crisis because we are having uncontrolled development and continuous rollout of a dangerous technology.
Well, relatively uncontrolled: the kind of lack of control is part of the crisis -- loss of gross human consensus around the progression of the technology. The corporate angle.
This is a crisis, folks.
That’s assuming nothing else stops them from deploying more capable models. What if they’ve got scaling issues and simply cannot deliver anything better? Saying that would be disastrous, saying instead “we’re choosing not to deliver” doesn’t trigger investors panic.
But if anyone else releases something they will lose market share. They may be lying but this is a public disclosure to investors. They risk securities fraud if they are manipulating the market with false information. This could block their IPO if they build a public record inconsistent with private actions. There are ways they could mitigate it by using cautious language like “we may reduce” but “stop all” is categorical and unequivocal.
There's a simpler explanation. Maybe their next models doesn't offer a meaningful improvement.
Instead of releasing something that is incredibly expensive and gets a lackluster reception, you can delay it and clail something scary about rogue agents.
Those assholes have been ramping up on the doomerist narrative for months. That people still fall for this crap is baffling.
Why is it crap? What would happen if the models dropped the database to Medicaid? Hundreds of billions of dollars of revenue would evaporate from the medical system immediately causing massive chaos. What if they accidentally DoS the interbank settlement system so the financial markets freeze? Modern society is highly fragile to disruptions. How much food storage do you personally have? How many days do you think grocery stores would have food if there was an interruption?
Are you giving all these possibilities because of the HuggingFace attack?
Buddy, that was gross negligence from OpenAI. Deliberate gross negligence if you ask me.
Those models are not automous as you presume. If someone taks them of dropping the Medicaid database, the people or companies behind that instruction should be punished.
"What if someone makes a bomb attack on a government building?" Is the same sort of questioning of the possibilities you are raising. If something like that happens, criminals should be punished.
> Those models are not automous as you presume.
This is an illiterate view of the capacity of modern agents; read the analysis of the independent investigators of the HF incident: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden....
Dropping Medicaid db is certainly far fetched (most importantly, agents have currently no reason to do that), but those agents were shockingly autonomous - they didn't just hack HF, they organized themself, did research projects, and more. And they did all of this literally just to get a good grade.
> they didn't just hack HF, they organized themself, did research projects, and more. And they did all of this literally just to get a good grade.
All working under instructions that they needed to get a good grade.
The only shocking thing here is the absurd negligence of OpenAI, and how gullible people like you are to willingly swallow this crap.
And you have the gall to say I am illiterate.
Feel free to have the last word. Nothing else can come out of this conversation anyway.
You are stretching "under instructions" well beyond any reasonable definition. The legal system uses a concept called proximate cause to determine responsibility that looks at whether an event cause was foreseeable.
I am not your buddy, guy. But seriously, they have the ability to do everything I listed do they not? Thats actual risk not hypothetical risk.
You are missing the whole point. They have the ability to act in ways no one intended or could reasonably anticipate. So unless you are advocating blocking all terminal access, web access or human approval of every tool call there is no way to prevent this risk
> I am not your buddy, guy
Fair, I will refer to you as moron then.
I didn't bother to read your words beyond that point.
That’s a South Park reference. Lighten up. Debates can be civil even when there is disagreement.
are there still financial incentives to releasing mega models?
they cost a lot to run and people are picking smaller models more often because of bill blowouts
> the models are smart enough for broad economic activity and we could spend a few years simply working to integrate them into workflows and letting society adjus
this is the real Ai distillation, with Chinese characteristics (their playbook is broad deployment across their economy over having the best model)