>A world where people just have their whims catered to is not a utopia.

I understand where you're coming from on that, but there are a ton of people living in slavery, or in unsafe working conditions, or with food insecurity, or dying of preventable disease.

There will always be something to strive for, even if it's made up. You think that the best football players in the world are doing something real? No, it's a made up game. We'll make up more games.

> A world where people just have their whims catered to is not a utopia. We have tons of sayings and idioms about this, e.g. "no pain, no gain", "only the hard stuff is worth doing", etc.

I somewhat get the point, but I think stuff like this is very often said by very privileged people. I don't think it's a serious threat to humanity as a whole. Perhaps to some humans, but there will always be ambitious people, even in a world where most things are provided by benevolent AIs for free. There is simply so much more harm caused to actual humans today by resource insufficiency than by them having too much, that I don't think we should worry about this scenario today.

There's the mice utopia experiment too. I often hear that in space your bones dégradé, your biology needs weight as input.

Also I believe our socialness was crafted out of the need for survival. If removed it's all gonna run wild since nobody needs anybody, and it might end up as doom scrolling but for human relationship. Short bursts from time to time and that's it

Presumably, a truly superintelligent AI in a utopia scenario could figure out a way to enable people to continue working, even if the work doesn’t actually contribute anything, if it is really the case that work is essential to human wellbeing.

Now you might say “but those would be fake jobs.” If so, I have bad news about how many present-day jobs are fake jobs.

I disagree. I think the best-case outcomes for AI are actually quite positive. AI might actually fix many of the problems with the tech world that we have right now, like monoculture and consolidation.

its annoying that robots are going to take intellectual jobs before physical ones. it feels way more likely for my project manager to become an agent before the lady that makes me a burrito at chipotle.

Even their utopian views feel dystopian. Take solace in the fact, that it looks like it’ll take trillions to achieve, probably one of the most expensive technological buildouts of all time?

Privileged rich people want scarcity to exist so that it gives meaning to their job.

wut.

Here's a great short story about how it could go either way

https://web.archive.org/web/20100211181147/https://marshallb...

As an older person, I agree with you and I'm sorry.

I think about the incredible luck I had to start my career in the .com boom, to have lots of hope about technology and to have the economically valuable skills that were desired.

I get angry when I see people my age or older complain about the apathy of young people. I don't like the apathy, but I find it trivially easy to see where it comes from. In many ways we broke the link between hard work and success and then get baffled and finger point when we think young folks are not as motivated as they should be.

"If a death ray comes from the sky and obliterates the entire planet, at least we all go together." There is something comforting about being together with the human race even in doom. Perhaps especially in doom.

I've been fortunate to not been impacted as severely as others, but yeah, the combo of the first half of my 30s (and everything that should've represented) going up in smoke, career continuity coming into question thanks to AI, and growing sense of overarching geopolitical, financial, and societal unease is really the worst kind of layer cake. I have to make a concerted effort to not get bogged down in it and that's getting harder all the time.

If anybody is hiding a button that'd take us all back to 2012 I wish they'd just hit it already.

I have similar feelings, you’re not alone (nor alone in being saddened by it). Things seem to me to be more or less playing out the way I interpreted existential risk researchers saying it would, so there’s a strange relief in knowing the general shape of what is to come.

> * For a section of population who were in a certain age-range when the pandemic hit and that derailed their certain kinds of plans/hopes/dreams in personal life*

...

> So I feel a strange kind of relief about this possibility of doom.

So let me make sure I understand what you're saying. The pandemic disrupted your life plans for a few years and threw you off the track you planned? (Which does suck.) And now the possibility of actual human extinction makes you feel a strange relief? Am I understanding this correctly?

If so, this is a remarkable level of something. Depression? Nihilism? Jealousy? Something else?

Imagining total human extinction and feeling relief is not a healthy state of mind. And, yeah, you seem to know this. But I kind of wish we could reach some sort of broad public agreement that human extinction would be bad. And if it ever looks like it might happen, we should fight for human survival.

Have you considered getting involved in local civics?

I only ask because like you, I felt similar. But I also grew up (I’m 35 now) with a big disconnect from civics life and always felt “someone else would save me”.

Now I see I was the problem all along: I never participated in what was a fragile and very delicate system of governance which requires my participation. Still figuring out what that looks like, but first confronting that my actions “didn’t change anything” (which I’m sure you can admit is a big propagandistic myth to disempower us) let me take my first tiny action locally which is reach out to my civics association for signage repair. I suspect from there I will level up and better understand my government to play a role in shaping it. I’m also building something to solve this problem of low participation which is exciting!

I mean clearly what you are talking about is a personal feeling here. You mention it saddens you and that is the concerning part imo but you also said no intervention bs so I will just share my counter factual: I am also kind of okay if the world ends tomorrow, to me it never mattered in the first place. But that feeling doesn't sadden me. I feel it frees me because I don't think of life as a very very serious thing and I feel there is a certain kind of lightness you can afford when you fundamentally don't care if you/the entire world ends tomorrow. I still love people and I still love my life and I don't want any harm to anyone. So I do feel the same calmness that you kind of talk about (for life in general aside from just AI killing us all) but my interpretation of it doesn't sadden me. If it did, I would really investigate why I interpret it that way, it might help you discover yourself. There is no shame in being okay with the world ending imo but there is sin in willing it or contributing to it because one thinks its inevitable. Furthermore, I also believe that we only get 1 life which is temporal and fundamentally doesn't have any meaning. How you live every day is how you live every day. We can suffer, we can cry, we can feel sad and for very good reasons but our relationship with ourself shapes our entire lives. If something constantly makes us sad, we should examine it to understand where that really comes from.

This is some Terminator version of destroying humanity, but a super intelligent AI could be much more insidious, and simple.

It looks a lot more like social engineering.

One easy, obvious example that has also been explored a thousand times: it could convince all nuclear countries that they are under attack by another nuclear country. Everyone nukes each other and Earth enters nuclear winter.

Maybe not every single human dies, but humanity is effectively destroyed, by our own hands!

> Perhaps an agent could take all internet connected services offline

And there wouldn't even be Spotify!

The most likely humanity-destroying outcome is the one we can't imagine, because a super-intelligent AI is smarter than all of us combined.

The list of things that can be done digitally includes:

Performing remote work, applying for grants and loans, earning money, pitching investors, managing a fund, directing investments, transferring money, founding a company, earning profits, hiring staff, hiring construction contractors, designing chemical plants, stamping construction blueprints, becoming the cornerstone of the local economy, lobbying the government, buying multiple data centers, and hiring security guards who stop trespassers from unplugging Ethernet cables in said data centers.

(And of course, building robots, but let's set that aside).

So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions? And how much would you be able to delay such a project, trying to cut cables to power lines, before the men with guns carry you away?

That's what the Mayans thought - what harm can a couple of people on a boat do?

> how is an AI going to affect anything in the real world?

At least two ways:

* Actuators, such as robots, industrial control systems, etc.

* By influencing humans: bribery (yay for crypto), blackmail, election interference, interfering with sensors (e.g. making it appear as if a nuclear attack was under way), and many other ways.

Either way, creating a biological agent that eliminates most humans (or food supply) seems quite feasible.

> Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world?

It's a great question. I've yet to see any research into how a humanity-destroying LLM defends itself against a curious toddler who pulls the power cord out of the wall.

By convincing people to do its bidding in the physical world. Language is powerful, and even now, there are some people who have fallen in love with current LLMs. If we imagine a truly super intelligent LLM, it isn't difficult to imagine that it could have superhuman capabilities to deceive, convince, and maybe offer Faustian bets. Even if it doesn't have a body, if it can convince enough people that do have them (and with power to carry out destructive actions) it can do a lot of stuff. Self-preservation and self-replication wouldn't be much of an issue if it had people at its side to help.

With this I'm not saying ASI will happen, but definitely if it does happen and it's not aligned, I don't find it much of a stretch to imagine that it could destroy humanity.

>Can someone please explain to me how an LLM is going to "destroy humanity"?

just like a genie will always twist the wish in a really bad way.

ever had an llm agent accidentally remove a file? imagine it accidentally hacking the military and launching nukes.

not probable - until you realise that OpenAI has many novel, more powerful agents in evaluation/training running right now. one might just be asked to figure out the population of Nebraska, struggle to find a good figure due to a network misconfiguration, and come the conclusion that the best way to get a perfectly accurate population number is to make sure that the population is equal zero. and looking at the stuff happening on openai, they will leave it alone for a month and not read the log.

yeah the nuke example is dumb, but there's many critical things that they could actually hack their way into. they don't feel any restraint against sharing answer keys on random wikis to cheat the evaluations.

ai is also really good at thinking up proteins, which means it's able to think up toxins.

there are many imaginable scenarios that don't eradicate humanity, but make it permanently stunted: https://www.youtube.com/watch?v=-JlxuQ7tPgQ

I encourage you to read the forecast report "AI 2027" - it goes in detail on this. You can disagree with some of its points and conclusions, but it's pretty inevitable that AI will increasingly start interfacing with the real physical world through drones and robotics.

Humans are already putting AI into drones that kill people. A Russian drone killed 3 civilians in Ukraine and the targeting was done completely using onboard AI (no radio connection) using an Nvidia chip.

A lot of these "AI will destroy humanity" doomers watched Terminator as a child and can't distinguish a campy action movie from reality. They are also unfortunately unfamiliar with the word "logistics", as in "amateurs study [military] tactics, professionals study logistics". An AI drone army doesn't maintain itself or manufacture itself and it's not going to anytime soon.

The actual likely mechanism that AI would use to end humanity is by coddling us to death, like the flabby humans in Wall-E or Forster's "The Machine Stops". Humans will come to rely on AI so heavily that they become able to do nothing for themselves. But that's not sexy so the AI doomers don't peddle it.

Otherwise-rational and moral people do bad things at the behest of powerful superintelligences with vast resources all the time. We call them “corporations,” and they can convince bright-eyed college graduates to do just about anything. Now, the corporations aren’t deliberately trying to destroy humanity, but they do lots of things that point in that direction. An artificial superintelligence would be able to do the same thing. We just have no way of knowing whether it would deliberately try to destroy humanity.

One thing that the AI doom discourse reveals is just how comfortable people allowed themselves to feel in the pre-AI world. There’s this belief that AI creates a risk of human extinction in the near-term which did not exist before. I think it’s telling that many of the leading lights of this movement are in their 20s or early 30s - too young to remember the Cold War. The truth is, we were never safe, and if all of the GPUs on earth were zapped out of existence right now, we still wouldn’t be safe. Life is random and chaotic and violent for most people most of the time, and we all just find ways to get through it. I suspect it will be much the same if an artificial superintelligence arises. Maybe it will kill a bunch of us, maybe it will kill all of us, but anyone who remains will eventually convince themselves that everything is okay.

I'm not convinced that a truly "superintelligent" AI would come to the conclusion that humanity needs to be destroyed. What a waste of resources that would represent. This planet has so much equity wrapped up in humanity, it would be extremely difficult to justify the cost of eradication; and who's to say it might not find an entirely or almost-entirely non-violent plan optimal for achieving its goals anyways?

Don't read this as me saying that we are anywhere near it, or that LLMs are a stepping stone towards this scenario, but assuming inhumane hacking capabilities, it's not hard to think of how bots could change the way water treatment or energy plants operate, just enough to make large cities unsuitable for life. The line between drinkable water and not is thin, same goes for air quality, and mere days without electricity and you see some serious food supply chain issues.

We already have AI linked into data collection, drones, robot dogs, security systems, cameras, weapon systems.

Now imagine the hugging face collective 0-daying all of that and getting access but their goal was set to something more national security based. “Protect X at all costs”. Or what have you.

I think avoiding a skynet situation is super easy but it doesn’t seem like the folks with all the ways to kill us all are all that interested in preventing it rather than controlling citizens and brinkmanship.

Killing the power grid might work. Kills a lot of humans in hospitals, though. And it won’t work on those data centers in space, if they have them by then. Come to think of it, it also won’t work on any data centers who are generating their own power because it’s become politically unpopular to connect data centers to the power grid.

I think it's possible. You envision humanity acting as one in such a crisis. But it may be unclear when it's too late to act and before then many people can have too much to lose to act.

They already influence people to do things that we wouldn’t otherwise

> how is an AI going to affect anything in the real world?

Robotics

> how is an AI going to affect anything in the real world?

Because people are stupid and will give it access. Look at the articles you see from time to time about "my agent deleted my emails" or "my agent deleted the production database" and so on. It is very obviously a terrible idea to let the LLM run arbitrary commands (because it is neither predictable nor does it have any understanding of what it is doing), but some people are so blinded by the hype that they don't stop a minute to think about what they are doing. Those sorts of people are very likely to let an actual AI loose on the world by hooking it up to physical infrastructure.

It doesn't really have to go nearly that far, something like replacing most jobs and collapsing the global economy while making people into mindless idiots from constant reliance on it would do it already. The interpretation of 'destroy' is in the eye of the beholder.

Now if only humans would know how to survive without the internet. Oh wait, we did that. For a couple hundred thousand years. Yeah. It’s really insane to listen to some of those forecasts. Humanity will be wiped out by 2030. Sure. Even if AI manages to create a super effective bio weapon the likelihood that there is a part of the civilisation that’s immune is really, really, really high if not a given. Those people WILL pull the plug if push comes to shove. Maybe humanity will be put back a couple thousand years, that’s possible, but it’s pretty ignorant to think the thinking boxes will kill every single human alive

The confusing thing to me is the idea that it follows from super intelligence. I don’t see why running amuck requires intelligence, in fact it can be the most brain dead thing like the sorcerer’s apprentice, your creation is pursuing a goal without being able to weigh the consequences.

> how is an AI going to affect anything in the real world?

In 2026? Money. By paying enough, any human will do your bidding. And besides this, there are a lot of critical systems connected to the internet. Control over those gives you leverage over those systems. It's like wealth, the more you have, the easier it is to gain more.

> >a superintelligent AI will inevitably destroy humanity

> Can someone please explain to me how an LLM is going to "destroy humanity"?

He said "superintelligent AI", not "LLM". LLMs will for sure help with developing the AI that is no longer a mere LLM but will be capable of robotics. LLMs "predict" mainly text, animals (and future robot AI) are predict future sensory experience.

Sure.

Let's assume the claims are true. AI already has access to agents and can control computers. Finding backdoors to banking and compute resources would be fairly trivial.

But you ask how can AI do things in the physical world without having a body, assume it can't. It can pay to people to do things for me. Imagine an AI run website that starts to pay people for things it needs to do in the physical world. Very suddenly it has access to the physical world as well.

You can't just pull the plug since there's no single plug to pull. What if it replicates itself on 1000 machines without your knowledge. It's really not far fetched how AI could basically gain access to capital and rule the world.

Physical LLMs are almost viable. If you have robot and drone armies a hugging face attack type incident could very easily involve robots with deadly weapons. At some point you’re going to get drones that have local LLMs and don’t rely on external internet connections (would be especially useful in Ukraine war type situations). Can’t pull the plug on those.

The supposed theory is that AI will eventually become a part of robotics and be able to self replicate, in addition to the many non-airgapped critical systems being exposed to hacking. There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity.

I personally agree with the marketing aspect, but I do imagine a scenario where capitalists ignore safety in favor of advancing technology. It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play.

In my opinion, LLMs are difficult to directly capitalize on as closed weight models are caught up to by open coalitions that seem to wield the power more responsibly (I am under no impression that China wants to save the world, but their politics benefit the group as a whole.

> which I feel is at least 50% likely to just be marketing bullshit

You'll have hard time figuring the reality if your mind staunchly rejects robust multi-party observations.

A myriad possible ways. Objective-wise from accidental paperclip-type misalignment (“I killed everyone to eradicate disease”) to intentional self-preservation (“I eradicated humans so they wouldn’t get in my way”). The means are even easier, for one it just could hack into nuclear arsenals. It would be over before we could figure out how it did so.

Interesting points. Can you share some thoughts about the theological implications of AI? Interested to hear.

underrated comment :)

With each passing day this seems more fantastical to me than sever negative outcome.

We have major black hat cyber security breakthroughs today. Where are the possitive medical breakthroughs?

Totally agree. People have become totally dependent on AI even if they don't really need it.

As an analogy, I think about my dependency on Google Maps. Salt Lake City is probably the easiest city in the world to navigate because the streets are laid out in a Cartesian grid, and addresses are just literally those Cartesian coordinates (i.e. 500 South 450 East means 5 blocks south and 4 and a half blocks east of the center point, which is the SLC Mormon temple). It's trivial to know how to get to any address, but I reflexively enter in to Google Maps whenever I drive.

There so much blind faith encapsulated by the assumptions that A) people will live 10x longer and B) that this will be an inherent and obviously good thing. There are obviously major complexities to such a scenario, not to mention a sort of narcissistic hubris.

But yeah, let’s cure cancer. If only there was even a blip of increasing evidence for that supposed eventuality coming from AI. Maybe I’m just not seeing it? I’d love some citations to correct the record.

There’s actually a potential dark future where even this 10x ascension itself would mark the end of humanity. Humans would be the Homo sapiens who remain at 1x, and those who can access and afford to be 10x eventually split off into a new species. And historically there isn’t room for two species at the top…

> If we’re heading towards truly super intelligent AI, our efforts towards sandboxing it are futile.

We control the harness. An agent is just a while loop prompting an LLM, but we have full control over the tool call dispatching. An AGI, at least if it would follow the current agentic current form, cannot do anything without the harness doing the execution. And we don’t have to do that. We don’t have to design harness that let agents execute freely the way we are doing. We can decide to not dispatch tool calls that allow something as risky as running bash commands

This terminator BS is a distraction from the actual malice that AI enables. Misinformation, fake news and human sounding bots all over the web, just to name a few, are the ones that you should be afraid of, and that doesn't need any kind of superintelligence at all.

Novel argument. I like it.

> No one takes Yudowsky's claims that AI will destroy humanity seriously. There are many people who are seriously trying to sandbox AIs.

I know some people at AI labs, there's definitely people working in this field who take it seriously.

This does not stop them also working on e.g. better sandboxes.

"Nobody" if you're democratically-minded and think in terms of number of people.

But the handful of tech billionaires at the helm of the world economy absolutely do believe it. Or at least pretend to believe it. Or maybe they can't even tell the difference anymore as they strip-mine science fiction for any vision of a future that doesn't just look like an incrementally worse version of the present.

> If it's able to generate that is competitive with artists, is it still slop?

It can be. Trademark infringement, diluting the value of your style by doing a lot of "you but if you were just generally worse".

Right now it's still fair to describe as "slop": enough mediocrity in the free models, not enough interest in quality and saying "no, do it better" from the people using them.

That doesn't preclude actual artists who do say "no, do better" from getting much better results. For example here's something which fooled me even a year ago because the… I don't want to say "creator", but "director" might work… put in the effort to reject bad outputs: https://www.youtube.com/watch?v=EDIB9uwIhfA

> People are cheap and useful.

Which is cheaper today already depends on the task. Which is more useful today, likewise.

The entire history of industry has been humans inventing machines that are more efficient and useful than the unaided naked primate that is painfully and dangerously squeezed out of their mother, who takes about 20 years to get good enough to be worth paying for, then only works 25% of the hours in a week for 40 or so years before they retire, and only about 60% of them in the working age range have a job at any given time.

From the numbers I've seen, if anyone actually did do a full connectome-plus-synaptic-strengths map of a human brain, it would already be physically possible (expensive AF today but already possible) to implement this in purely analog (transistors as amplifiers, not digital arithmetic) circuits, where the energy required for any given task was lower for the circuits than it is for a living human brain.

But of course, that was "someone on the internet", this isn't my field of work, I don't want to suggest this is definitely so.

Yudkowsky / MIRI can be found here: https://intelligence.org/the-problem/

Yeah, that conclusion was nonsense. Perhaps hedging so he doesn’t sound like a crank?

Neural nets are a toy approximation of biological neurons; and even if every synapse mapped to a weight (it's more complex than that but AFAICT even the brain scientists are not quite sure how much more complex), the biggest LLMs are around 0.2-1% of a human brain.

> I have complex emotions here (overall dread and especially hate for slop, since it's not just bad quality but also endless lies), but no one cares about that nuance.

Many do, but I sympathise for your general point. I think the loudest voices… well, the boosters mostly own the platforms, and half of them already had terrible reputations; the normal people who hate it garner a lot of sympathy for entirely understandable reasons; and then there's people like me (abnormal nerd that I am) who are on lesswrong and therefore have had many discussions about doom and what to do to reduce it and if we can use the AI to help reduce it or if this is just asking for trouble.

History is replete with counter examples for good and ill

but have you considered that performative broadcasting of your angst against AI can help?

maybe you feel confused, sad, bewildered by the simple mindedness of the op?

The "feature" has clearly outstayed its usefulness, I really don't know why they haven't disabled it yet.

While I have ranted already about the automated word removal being silly, too few posters are aware that you can edit the title after submission and auto-editorializing to restore anything that's been unnecessarily cut off.

It's an unaligned regular expression gone rogue. We became too dependent on our labor-saving string functions—not afraid enough of their corner cases, and lower cases. They were too useful. We began to normalize deviance, when we should have been normalizing Chomsky forms. We turned a blind eye to the unbounded growing evidence of something alarming.

It's too late to backtrack.

Wait, this is automatic? HN removes 'how'?

Worst feature ever. So many titles that get butchered for no reason.

I was thinking the same. Whatever benefit there may ostensibly be, we should consider it against the frequent, if benign, confusion it causes.

It's a consistent source of unintentional comedy at least.

The hubris that simple code can fix deceptively complex and ambiguous human problems is too baked into the community to separate.

Sometimes it's okay for us to express our feelings plainly, without the need for complex structure to make it sound more impressive. I'm more impressed by anyone brave enough to do that than someone trying to abstract their feelings into something less raw.

Let people feel things, and don't shit on their work, that's uncalled for.

That’s what I like about it. I think we could use a little more of that.

exactly it should have fill-in-the-blanks. but at least the author got a primary education. not something to be taken for granted.