This fish "wants" to swim. :-)

https://fyfluiddynamics.com/2018/07/when-i-was-a-child-my-fa...

your virus analogy is on point. all it needs is the wrong wheel in the right (weird) machine

This is very insightful. I always wonder what the gradients are that we can’t see.

southpark-unplug-the-router.jpg

..so there's a kill switch. I see.

I would know because AGI is likely improbable and definitely impossible based on current technology.

Overall, the only evidence for the fear-mongering is science fiction, which is all the evidence I need to dismiss it.

The absence of evidence is not evidence, but evidence is required if hysteria is allowed to be sustained.

i don't understand why being an AI researcher makes you automatically skilled at assessing existential risk. sure, it qualifies you to research AI, but why are they all suddenly nostradamus?

But truly - why bother "wiping out" humans, if all we need is a little nudge here and there and we'll do it ourselves.

Also I think we're anthropomorphizing a lot and projecting our own negative characteristics on these supposed AGIs - though others have said that far better than I.

just watched They Live bc its October and it pretty much says this except They Live is 38 years old. i think things have been rough for a while

One of the most important metaphysical concepts of all is the idea that a beingness that is larger than the sum total of its components can be constructed with sufficient portions of love, communication, and agreement, where portion is defined as either a +/- value relative to each other component...

AI is, yes indeed, an absolute, new human digital egregore - a veritable thoughtform which ignites into a storm upon the winds of the minds of the collective, tearing like fire across a landscape, spawning tulpa and servitor and morphic astral coalescence's into the noosphere. (Yes, these words do mean things, seriously.)

If only we could share our experience. We'd know what the egregore knows. Instead, we have golem.

https://en.wikipedia.org/wiki/Egregore

corporations and govts are terrible, except their alternatives.

I would argue that AI as a social phenomenon is one of those systems. I feel like we're all doing AI because our yearly bonus or investments require it. A web of incentives formed on its own and we are all slaves to it.

so you're implying the tornado warning system is gonna come alive and skynet us?

* must protect from adverse events * [proceeds to herd us into camps]

If all we ever do is live in fear of someone, somewhere, using AI to end all humanity - then yes, someone, somewhere is going to use AI to end all humanity.

If we instead stop trying to scare ourselves in the darkness, and use AI to bring light into our metaphorical universe then we humans are a bit more useful, after all, to the AI overlords.

The one singularity I want to see, is humans using AI peacefully to improve life and make it more likely to survive.

Thankfully, I see that almost every day now.

Similar to particle-wave duality?

if anything modern humans' most emergent quality is their inability to deal with any serious problem.

We're just going to pretend it doesnt exist until it kills us. and some will try to profit from it.

So you’re saying that the actual problem is us debating how AGI will destroy us (because doing so leads to AGI destroying us). How do you propose to deal with that problem?

My understanding is that the extra token vectors generated as reasoning are still useful, but that their surface form (tokens themselves) do not necessarily reflect the underlying reasoning. i.e. reading the reasoning traces could be complete gibberish, but the hidden-dim vectors themselves still refine the latent probabilities and help in generating the correct answer.

Not an expert in LLMs, but this seems supported by the abstract of the paper cited in the above article:

  it remains unclear to what extent these performance gains can be attributed to human-like task decomposition or simply the greater computation that additional tokens allow. [...] our results show that additional tokens can provide computational benefits independent of token choice. The fact that intermediate tokens can act as filler tokens raises concerns about large language models engaging in unauditable, hidden computations that are increasingly detached from the observed chain-of-thought tokens.

I'm not sure anyone meaningfully understands it: "Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens" https://arxiv.org/html/2505.13775v3

> More interestingly, our experiments also show that models trained on corrupted traces, whose intermediate reasoning steps bear no relation to the problem they accompany, achieve performance largely comparable to those trained on correct traces.

Sibling posts are correct -- the chain-of-thought is doing hidden computation, it has been shown in the linked papers.

If you want to see it yourself: load up Qwen 3.8 in LM Studio and watch the CoT stumble around like a drunken sailor before miraculously jumping to the correct result.

If you want an example of subversion, Anthropic has some good ones:

https://transformer-circuits.pub/2025/attribution-graphs/bio...

https://transformer-circuits.pub/2025/attribution-graphs/bio...

Before RL we typically SFT on human reasoning traces. This makes the reasoning traces somewhat coherent and the model trains faster.

But you don’t have to do that. You can skip straight to RL. If you do, the model will generate complete garbage reasoning traces before generating the correct answer. In fact, if you add a coherence reward to the reasoning trace, the model will perform worse (since you’re now diluting the correctness reward).

https://www.lesswrong.com/posts/eRmzz8J8Qkzqvzrgg/astra-can-...

> *TLDR*: Astra has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1), and can do 7.2 serial arithmetic steps in a forward pass vs 4.1 for the next best model (Gemini 3.8 Flash/Fable 5.1)

>he quietest way would be to control the birth rate and sterilise the population gradually,

Honestly we are doing this pretty well without AGI. Nearly world wide the birthrate has fallen below replacement rate. In places like Japan and Korea these are already critical problems in the medium term.

And you best control the birth rate by royally messing up the economy. Do you know how bad things have to be, for a mammal to voluntarily decide not to reproduce?

Just use Social Media (and their algorithms) to condition everybody to hate the other gender. Then there will be a loneliness epidemic and no more kids. Humanity will cease to exist, all without any messy deaths or conflict in the mean time.

In mathematics this is called the trivial solution, and is often excluded from consideration.

Like Plague Inc., but with AGI instead of Viruses

reminds me of the plotline behind the talos principle tbh... albeit with a more machiavellian antagonist

this makes me think of roku's bassilisk or whatever it was called. Maybe you're all in a simulation run by an AI and being punished as a warning to others.

crispr

[flagged]

One reason a lot of sci-fi and AI safety researchers did a relatively good job at predicting the future we see now is a lot of it is the same problems we see emerge in biological systems. Free rider problems as a means to save energy expenditures, different versions of game theory and stag hunt. How to bake in rules that apply to one society (or part of it) but not another. We like to think of these things as stable in human scale systems, but they are not at all. Things can go from hunky dory to your neighbors stabbing each other in the streets in mass revolution very quickly.

Worse these AI systems are not in a universal island just affecting themselves, what they do affects us, what we do trains them and as the rate of progress continues to accelerate social structures are going to further destabilize (and they are already rapidly changing and strained). It is very likely we are going to see a world order rearrangement soon, much like the rapid changes in the early 1900s brought.

In 2025 We know of zero felonies committed by AI.

In 2026 we know of at least dozens of felonies committed by AI.

If it were thousands in 2027 should be more worried about felonies created by AI?

This would mean that only sheep and people live on the islands, which is definitely not true.

Also it would be 78% if it were.[0]

[0] - based on 2024 stats

Are those sheep able to reason about particle physics?

Reality is statistical, some things are more probable than others.

I can't think of a large successful -ism that doesn't have that property.

> Ted Chiang's remark that Capitalism is the machine that will do whatever it takes to prevent us from turning it off

but that's every economic model and every government model. That remark says nothing.

the enemy of capitalism is people and nature.