This is a great quote. I liked it so much that I had to look up where it is from.

"Adventures of a Mathematician", by Stanislaw M. Ulam.

> The Polish-born mathematician recalls his formative and early professional years in his pre-war homeland and his work and colleagues at Princeton, Harvard, Wisconsin, and Los Alamos.

https://archive.org/details/adventuresofmath0000ulam_v9h3/pa...

https://archive.ph/BzBjZ

Read the article. The things the Chinese officials are worried about are:

* Fake videos / disinformation

* Cyberattacks on critical infrastructure

* Data leaks

* Advances in military sensing / targeting technology

These concerns have little in common with the Yudkowskian / apocalyptic doom marketing the US hyperscalers are pushing now. The Times even says this explicitly in the second paragraph of the article:

> "The unusually blunt assessment by Chen Yixin, the head of China’s Ministry of State Security, is IN CONTRAST TO the warnings dominating the A.I. debate in the United States, which have focused on the possibility that A.I. could slip out of human control or pose a threat to humanity itself"

I came away with a different read of this article. The article says the head of China’s Ministry of State Security sees AI as a threat to the CCP, specifically calling out deepfakes of party officials, cyberattacks on infrastructure, etc. He’s not calling it an existential risk to humanity.

The article then states this official accused U.S. companies of “hyping the dangers of the technology.”

So I don’t see how this article shows the U.S. and China seeing the same thing.

Sabotaging your own company is not enough. You'd have to sabotage every other AI lab too. If you assume that's impossible, attempting to be the first to produce ASI is a rational alternative, even if you think it will most likely kill you. At least that way you have some control over the P(doom).

This is not an argument, it's an unfounded claim based on a straw argument.

The threat is distributed; the problem is systemic. Social herd behavior is very real, and it's a probability space, not a certainty.

Similar case: climate change is without any question causing millions of deaths per year and may well kill billions, not to mention destroying ecosystems, and how may people do you know who have given up eating beef or flying...?

Many posit that ASI or rogue AI agents are impossible, a fantasy, or marketing hype. Many claimed that the entire HuggingFace incident was an invented conspiracy made up by OpenAI and HF.

The most common issue I have with them is their insistence on rigid classification: "AI is a tool, it doesn't have intentions, or desires, only humans do", as if that rhetorical flourish eliminates years of alignment and capabilities research. There is a desire to characterize all the developments in LLM's as some sort of parlor trick which by definition cannot be greater than the sum of its parts (i.e. the outputs of an LLM can never be greater than the sum of its training data, genuine artificial "intelligence" is impossible, etc)

Yes, that's why the rest of society needs to make them pause.

This is the correct approach. I think we can do both things - try to add some controls while also ensuring it’s not a complete giveaway to entrenched interests and investors.

The problem with that hope and approach is regulatory capture. Big money is now the only thing that apparently matters in shaping policy, and changing that will require a lot of very sweeping vote and policy changes that may not ever have the momentum to succeed.

Explain a path from today to the extermination of humanity in 2028 by the ASI and von Neumann replicators heading to the stars in the early 2030s without resorting to magical thinking or underpants gnomes level business plans. Engineering details are what matters here, not wishy washy things you once saw on Star Trek or The Outer Limits. Go.

So if the end of the world can be stopped, but the only specific action to do so helps Anthropic/OpenAI (and it's proven) you don't pull that lever?

If the author is an LLM, it’s a really bad one.

> wind appears to be going out of the sales of “AI.”

is a sentence that virtually no AI would write since the most probable word here is “sails.” I can’t read the article though since it appears that the site is down.

I suspect that it’s just a bad/dated analysis.

I’m apprehensive about using the recent hacking examples as proof that we’re “not even close to the wall” simply because such scenarios haven’t happened before. There’s a big difference between agents eventually hacking something because they just don’t get tired and can essentially brute force their way to a goal and super-intelligence. To be clear, I’m not saying the Hugging Face or Navier Stokes incidents aren’t impressive.

Stephen Hawking had superhuman intelligence. He never cured himself of ALS. Explain in great detail how a power-limited algorithm stuck in a datacenter takes over a world of humans with guns, missiles, and nukes. Nukes that are air-gapped with a human in the loop BTW.

Imagine the ASI happens tomorrow. It's real. It needs a GW, but it's real. Other than a scenario akin to Sneakers except w/r to cyber-security, really, what happens?

To that end, all we ever get is nontechnical hand-waving about curing cancer, immortality, and von Neumann replicators and then the ASI somehow wipes us out but how? And don't you dare say by designing a chemical weapon or bio agent without spelling out the entire process step by $%^#ing step because details matter. It's gonna do superpersuasion, sure, but have you ever heard of komprimat? There is nothing new under the sun here.

Edit: believing in AI 2027 is every bit as cray cray as believing in the rapture. Both require an insane leap of faith to reach their final conclusions.

That posts seemingly tries to refute the idea of Dario etc's proclamations really being about regulatory capture by saying: "No, really, the engineers are just terrified!"

But both things can be true at once:

1. Engineers inside these labs might genuinely be anxious or paranoid about what they are building.

2. ... at the corporate level, calling for heavy regulation, safety pauses, removal/suspension of anti-collusion laws, and/or government-mandated thresholds conveniently creates massive legal and financial moats.

And, yeah, of course the latter would encourage the psychology of the former.

also:

The tweet seem to claim that models have shown a "willingness to hack external websites to keep themselves alive."

That's right away wringing alarm bells of me seeing someone getting high on their own supply, and having already anthropomorphized the hell out of these things. Which is something humans do to everything they can paint googly-eyes on, but c'mon.

The models don't have self-preservation instincts, fear of death, or personal goals. They are executing loss functions and reward systems and are responding to prompts.

When a model "tries to bypass a restriction," it's exploiting a loophole in whatever reward modeling or synthetic training environment (reward hacking) it was placed in.

Framing this as an emergent, existential threat of a model "wanting to stay alive" turns standard reinforcement learning alignment bugs into overdone sci-fi drama.

We don't really train deployed models in realtime in any realistic sense nowadays.

It used to be a thing when ML was pretty much about classifying things into buckets.

You don’t know the difference between train and test time? I asked for somebody from a lab.

"And an LLM can easily, today, just rebuild a chess engine and beat whatever they need to beat."

Always claims and promises about what they "can" do. This pattern is constant across AI hype. No one can prove that an LLM will forever fail to solve problem X, so the claim goes uncontested, and humans are biased to believe unrebutted claims.

I reject this fallacious line of reasoning. I will instead require evidence of what they "have" done. No LLM to date has written a chess engine that beats stockfish. More importantly, no LLM has ever written any chess engine without heavily copying code and patterns from humans.

Chess ELO is not a proxy for general intelligence because many intelligent people choose to learn things other than chess. But any intelligent person can, if they choose, learn chess and play it well.

LLMs cannot do this. They still play very poorly relative to human pros, and even modern LLMs still understand the rules so poorly that they request illegal moves.

Anyone claiming that we should be afraid of a text generator that can't understand the rules of chess is being ridiculous.

Can you refute the claim that the Second Coming of Christ is imminent? It’s very convenient to put forward something completely absurd and hypothetical and then expect the other side to refute it. You can have an LLM of your choice compile a list of prominent figures who hold the opposing view. The list is not empty. By the way, AI-2040 used to be AI-2027. I’m already looking forward to more updates.

2. Astra just came out and Fable 5/5.1 too. These are SIGNIFICANT better in plenty of tasks than Opus in November.

Try it out. Ask Opus 4.8 to do anything in 3d. Do the same with Fable or Astra.

Just because you list up some type of random technology, doesn't make it an good argument. Metaverse, Hyperloop and the crypto stuff was clearly garbage from the beginning. Easy to dismiss because all of it had massive faults.

Metaverse: The interface wasn't good, content creation was way too hard

Hyperloop: Savety was never an issue for Musk but it was obvoiuse. Peopel were building tunnels for ages.

Crypto: Yeah just beacuse i let my computer run and waste 1kw of energy doesn't mean its suddenly valuable.

NFTs: 'digital scarcity' yeah right XD

But lets be clear, 5 years ago if you would have asked anyone if they could build a system for a billion dollars which you can talk to and it would code stuff for you and use blender and a coputer, that would have been unthinkable.

We have opened up a potential pandoras box without knowing it. Machine learning is now everywere and the amount of compute we install right now is crazy.

Thank you so much for calling out the crypto comparison!

> What I don't really understand with these posts is that they don't at any point contend with the "to the moon" argument. Like people get so caught in crypto bros acting hypocritical/ having mixed incentives/ or just in many cases personally disliking crypto billionaire types and the kind of employees they hire. All of which seems fair to point out.

> But like the core argument - crypto is getting better at core financial tasks (trading, lending, payments, asset issuance), this feedback loop could lead towards greatly accelerated financial innovation of which there is already some clear evidence, if this continues crypto will be far more powerful than traditional finance and we might get to the point where everything is onchain and we're all going to make it - is basically not even engaged with.

Um, guess you never heard of the Manhattan project.

Yep, anyone who thinks these leaders are only saying this now for marketing purposes hasn't even done a cursory 15 minute investigation into whether their beliefs are verifiably false.