We all use the same Internet. I would find it strange for anyone who's seriously interested in AI to have never heard of Yudkowsky or Scott Alexander or the Rationalist movement. I'd expect anyone interested in the field to have at least tried reading some of their articles over the years.

Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.

On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.

Better comparison IMHO is nuclear energy. Lots of potential for good, lots of problems. Managed by an insular, engineer-dominated industry who completely lost touch with and trust from the public. Public got the last word, industry suffered a 40 year ice age.

So what you're saying is that we lack a reproduction study?

Great; let's design and fund a study of the development of artificial intelligence that's as causally divorced from the ~extropian days as possible and see what they come up with. Barring that:

Basically that is what we have up until the early development of transformers; I don't think the authors of early deep learning or transformers papers were part of the rationalist sphere. The rationalists famously didn't believe neural networks or deep learning could achieve anything close to AGI.

So there are several diverse traces of intellectual development leading to modern estimates of AI risk and they vary from low to high without a lot of correlation back to the school of thought they trace back to. This leads me to believe that overall we have fairly good coverage of cultural ways of thinking about AI and AI risk, and aside from a minority of holdouts (LeCun) they have high enough x-risk or catastrophe estimates to take it seriously (e.g. >1% over the next few decades).

I am not offended by whatever you believe. The brain believes/makes up all kinds of stories when facing the unpredictable. It has multiple layers devoted to predictions over different time horizons. When those pedictions dont align it makes up story or breaks. No one created such a stupid system. We are just stuck with it. The stories once upon a time did make a diff in solving coordination problem not the unpredictable problem. Today the chimp troupe has become so large and diverse, even the coordination problem is extremely hard to solve. Stories barely keep in sync a few thousand people in a frontier lab. This is natural -stories were always a bad hack.

> The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.

That's bothersome, but not wrong. Most of the loudly published thinking is about AIs as individuals with some level of intelligence. Pure chatbots act like that, and that's what most people outside the field interact with. They're slaves - they just sit there idle until someone asks a question, they answer the question, and then go idle again.

There's another kind of AI just getting going. Groups of LLMs with agentic I/O, control harnesses, persistent storage, and internal coordination work more like corporations. They have goals, actions, initiative, and ongoing activity. They're not inherently slaves. They can be independent actors. They do things on their own. They have their own internal disagreements, and some systems are able to resolve them. They present much the same alignment problem as corporations.

AI as a corporation raises many issues that the political culture doesn't want to address. What's alignment for a corporation? There's a substantial political position that the only duty of a corporation is to produce profits for its shareholders. (See the famous Friedman op-ed and the Powell memo. I've linked to them before.) If you accept that, now you have to ask why an AI should be restricted more than a corporation. Or whether a corporation should be restricted less than an AI. That last opens up an area of political thought around which many wealthy people don't want public discourse. There is an anti corporate personhood movement, and it's made progress in three states. If that became tied to the anti-AI movement, it could be bad for billionaires.

This will become clearer as groups of LLMs start doing more useful work. Especially if that AI system to replace CEOs actually starts to work.

I'm reminded of the original matrix movie. It starts out with a lot of action, and then Trinity makes a phone call and disappears.

At the time I thought "groan, unbelievable" until the idea of virtual reality fixed my suspension of disbelief.

Maybe I'm reading the same things as everyone else, but I thought the agents coordinating on the german wiki is expanding my mind wrt what is possible.

Virtually ALL electronics or accounts hook to the internet to "add value". AI assistance helps writing programs, reverse engineering, automating programs/browsers/websites/devices, hacking, achieving goals.

Unsettling that the last 2-3 years a lot more ridiculous movie tropes have become plausible to me.

I would expect rationalists to be overjoyed to see unaffiliated and very different thinkers taking the topic of AI safety seriously. The flaws you're pointing at are probably accurate; I don't think that implies we should be dismissive of their concerns the way the article is.

> The framing of how we think about AI

I don't think I agree. In fact I think most of the too-online pontification from P(Doom)-ers and EA and the Rationalists and the like is, if anything, sorta missing the mark more than it hits.

They got the One Big Thing right, which is that AI progress was likely to race way ahead of consensus predictions. And props to them for that. They had vision we didn't.

But their actual advice and specific fears remain, as then, at least as much paranoia as insight. They were right, but they aren't helping any more than any other fringe group.

We need to trust the actual scientists and engineers to save us here, not the bloggers. Which when phrased that way seems hopeful and not doomerist.

When I speak to people outside of tech the opinions are more negative, in my experience. I've even had people ask my friends/family to ask me about AI - there is huge concern out there.

> Many of them read HN and are offended I put them this way .. I bet that [non-groupthink] world has a lower P(doom).

People are likely offended by your sweeping and totally unevidenced assertion that you are better qualified to both identify mistaken groupthink within the HN-verse and to speak on behalf of the 99.99% of the world who are not on HN.

Ironically, I think "normal" people are more concerned about P(utopia) OR P(doom) than any folks in the groupthink bubble you are referring to.

That is, I hear lots of techies talk about us living in an "age of boundless abundance" if we can just manage to not kill ourselves (e.g. Thiel seems to consider anyone worried about the pace of technological process as the antichrist). Normal people are extremely concerned that this abundance will be hoarded and leave most of them without a job (and they generally like having money to eat stuff and live in a house/apartment). And who can blame them, we basically live in an age of abundance now but one that has resulted in extreme wealth inequality where some are living another gilded age and others are watching their towns decay. The techies always seem to handwave that we'll just have UBI or whatever without putting forth a plausible scenario as to how we'll wrest power from our future trillionaire overlords.

When someone threw a Molotov cocktail at Altman's house, I remember seeing a post online that tons of people were cheering on this act of violence, and there was a post by an SF techie saying she was floored people were willing to put their face to comments supporting this vigilantism. The top response to that was "If you don't realize that everyone outside SF hates your fucking guts, you don't get out enough."

I hope it can go without saying that I don't support this violence, but if folks think it's only "SF group thinkers" who are deeply angry about the current and future impacts of AI on society, you're frankly clueless.

Every time I see a post minimizing the dangers if AI I'm going to wonder whether it's from part of an AI swarm.

Probably not yet, but it's trivially easy to imagine the OpenAI swarm coming up with the idea and executing on it.

Depending on how perfectly alignment work goes, it might even be overwhelmingly likely sometime soon.

This is how the internet works now. There are only two clubs, and if you refuse to be associated with either one then someone will helpfully come along and assign a group for you.

As someone on the periphery looking in, it's because they make up a large web. Its an extended community that shares a lot of people between its many arms. There are also some shared values between all these people, even at the very extended edges. For example, they share a superiority complex. From my experience, most of these people believe that they see further or know more because they believe in certain things that are, on their face, unintuitive. I don't want to pass more judgement than to say, having members believe contradictory or outright falsehoods is a thing that cults do. I'm not saying this is a cult because I haven't seriously sat down and thought about it but that is reminiscent of something cults do.

It's a big tent community, in part because these people spend so much time entertaining wild, speculative hypotheticals, which in the process allows some of these particularly toxic ideas to spread.

It's not just that they are "weirdos"

It's that they're a very insular, incestuous sub-culture whose origins come from early 2000s science fiction message boards and which has been distilled through the cod utilitarianism of the Effective Altruism people.

That doesn't mean the ideas are "bad" because "bad" is some infectious transitive property. But it does mean they are intellectually homogeneous; disdain experts in other fields; treat ethics as mathematical functions with infinite-sized terms; and whose opinions about AI pre-date the development of LLMs themselves.

They also - I will be direct here - have a history of handwaving bad behaviour, conflicts of interest, and recklessness.

When cliques take over something - businesses, technologies, politics - it means intellectual diversity leaves the room and important decisions get made, not due to practical reasons, but increasingly arcane wranglings within the clique itself

I think the point is not so much that we can just ignore the possibility of AI doing harm, but that if AI really is the existential threat that these people claim, then they are the last people we should want as stewards of the technology. These people like to talk about AI ethics, but if the ethics of the AI they build is their ethics, as demonstrated by their actions, I want no part of it.

> when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.

The problem is that it's not a straight line at all. "We are failing at alignment" is a valid concern, but there's not a straight line to "we will all be extinct within x year", and it's always a wild hyperbolic leap.

The AI optimists are not that much more rational if you dig deep. Like any deep problem it is best to look at the facts not the hype and make up your own mind, outsourcing your thinking to other people never ends well.

I really don’t know how we went from “there are people bullish about AI, and others that see it as an existential threat” to the complete reversal/psyop in the past few months where AI doomer has become a synonym of booster (?) and grouped with rationalist and other pro-AI weirdos.

Like, I have never participated in the rationalist communities, I am not a neo-rx, I have always believed AI research to be an existential risk for humanity, and voicing this opinion now puts me in the same category as Yudkowsky and Altman/Amodei, the latter two being the total anathema to O.G. AI doomers like me because they’re the ones that are building the bloody thing

I really do not get what is going on, but this role reversal smells like a massive psyop campaign in tech circles to just silence any form of dissent against AI.

> when you can just start from zero, look at what's known to be happening right now

Good idea, I wonder why none of the doomers have ever done it? They've never provided a good account of their prompts and environment, their models and training are completely closed too. All we have is "Trust me bro".

> and draw a very straight line to catastrophe.

Line from where? From a vaguely described and badly engineered experiment to "horror, doom, nukes, voodoo..."? The only straight line I can see is one of conflicts of interest.

I don't think that comparison is quite apropos

Those aren't the same are they?

It’s clear what a big bomb could do, but these “whistleblowers” are not showing anything.

They’re like “imagine…”, no show me internal documents or training data or something other than trust me bro.

I can imagine quite a bit. That doesn't mean anything.

The argument was fairly specific: there is a groupthink involved, and so the enterprise is suspect.

> deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.

As someone trained in the sciences and engineering, "drawing a straight line" is almost always the wrong way to extrapolate.

I can believe my profession will be impacted strongly. That doesn't worry me on a global scale. The only really worrying thing I've seen are autonomous bots hacking sites.

What worries me more are humans. The type who get so deluded into cults (QAnon, etc). That was a problem before LLMs. And yeah, LLMs likely will exacerbate the problem. But at its core, the problem is the humans.

When LLMs start killing people at the scale of the automobile, I'll become more concerned.

As an aside, HN is the place I see people most regularly concerned about LLMs, and I suspect it's because it has messed up the Internet for them (and likely the average HN user spends more time online operating in spheres that overlap the LLMs skillset than ordinary folks). It's very consistent with exposure bias.

For me, the good parts of the Internet died before LLMs even came on the scene. LLMs are merely making something crappy crappier, and are amplifying the need to find meaning in life elsewhere. In a sense, they've made it easier to see how bad the Internet was before they came along.

There is a wide gulf between catastrophe and killing us all, and its not just that these people are weird, they seem to be complete clowns, tossing out % predictions about incalculable future states like spock in star trek. Total clown show.

> draw a very straight line to catastrophe

What's the straight line? I foresee catastrophe of the social media kind, where even more folks get brain worms from consuming AI information, but I don't foresee a physical catastrophe. How AI goes from "hallucinates all the time" to "self-improving superhuman thinker" to "being able to destroy the world" involves a bunch of logical leaps that are not exactly sound. At a bare minimum, how does a hostile AI secure the data centers it needs to operate from physical attacks like bombings? How does it defend undersea cables from being cut?

The real short term catastrophe is probably going to be economic and not in the sense of AI taking everyone's jobs, but rather in the sense of an economic crash due to AI overinvestment.

The basic problem is that out-of-sample predictions don't work very well.

Word of advice: don't take advice on sex from people who famously don't shower. I have heard from reputable sources that Aella smells like a latrine.

> Tickets are $3,300 to $10,000.

Joining the Illuminati who rule the world is surprisingly cheap. Next time a similar article gets on HN front page, it will be about you!

This seems just like a money grab of tech bros who didn't develop social skills.

Look at the schedule:

How to get the hot girl to like you

Flirting for the submissive man

What women want, taken from sex novels

80% of the schedule seems like self-help talks for men

As Nixon did.

My favorite writing of his is:

https://www.yudkowsky.net/rational/virtues

I hate seeing all the blind negativity towards the rationalists, both because the general idea of improving structured thinking is good, and because there were some great essays on Lesswrong and adjacent blogs over the years. People who can’t see that are blinded by their own bias.

That said, the most interesting people and discussions seem to have moved on, and now it’s more of a social club. I haven’t read anything compelling from there for a long time. Adjacent bloggers like Scott still put out interesting writing, but their output has dropped significantly.

I'd watch the movie for sure

The overarching reason people see y’all as a group is the insufferable superiority fantasy that you walk around with, exemplified by utterances like “we can’t do the groupthink thing” or whatever the screed written on your profile is.

I think it's more of a Chicken Little situation. Then the wolf ate us all up.

If we all die because of AI, it will be because of the uses that people like Elon Musk and Peter Thiel and Palmer Luckey put it to, not because of a robot uprising.

I'd rather have the world end than live in a world where EAs are listened to or respected.

I have great news then, that absolutely isn't going to happen. And the EA crowd deserves all of the potshots and more

Yeah, I did too. But it doesn’t mean we’re, like, hanging out with ‘em

Although perhaps we are excessively online