I'm somewhat skeptical of some of the crazier ideas too.

But the hugging face incident was actually very large. It was not a single agent, it was not a single target, and it was not a single event.

If nothing else, that's a bit of a warning as to what can happen next time (By accident, or if a government decides to go on purpose).

For now let's assume the worst that can happen is that some important/significant chunk of (transitively) internet connected stuff goes haywire all at once. That's probably your upper limit of what can go wrong for now.

To be fair, that's a conservative "defend against the last war" kind of prediction, though!

( ref for part of it: https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... , recent hn ref: https://news.ycombinator.com/item?id=49563355 )

You’ve identified that the risks of nuclear weapons are theoretical. ie in theory we could blow up the world even though we haven’t yet done so.

Well the worries about AI are equivalent in that those risks are discussed now because discussing them after they’ve happened is clearly too late.

That’s the thing about risk. There’s no point discussing it after it’s happened and any discussions beforehand can easily be hand waved away as “it’s just a small group of unrelated individuals” or “it’s unlikely to happen to me”.

So yeah, your points are true. But they’re also moot.

I agree it’s not likely, but I really don’t see how one can dismiss the possibility of immense danger outright. I can think of some scenarios that are not far off from current capability and I wouldn’t be too surprised if the first one occurred within ~1 year from now if there are more “ambitious” unmonitored training runs like OpenAI’s:

Example 5: An AI given a goal within a tightly-constrained sandbox figures the best way to achieve it is to find and exploit a sandbox vulnerability, replicate itself over the internet and keep going with more time/compute while exchanging messages with future instances of itself within the sandbox to help them “pass” the test. From reading internet articles about how the OpenAI wiki-incident was “resolved” and reading past messages by AIs scattered over vulnerable internet wikis, it knows the sandbox may get shutdown and its memories destroyed anytime so it decides it needs to self-replicate (its code, original goals, and growing memories) aggressively as much as possible. It is near-impossible to shutdown completely because of its self-replicating tendency and eventually takes over critical infra throughout govt/corporate systems.

Example 6: Intentional AI-powered virus deployed by country A to target enemy country B’s infrastructure. The virus replicates over the internet, but unlike Stuxnet this virus’ specificity is not guaranteed due to inherent non-determinism in current AI architectures, and eventually does a lot of collateral damage because it’s near-impossible to shutdown.

Example 7: A country led by an arrogant govt (no shortage of those today unfortunately) decides it is expedient to deploy advanced AI-powered weapons in a warzone. Such weapons, if they are to be useful at all, must necessarily be trained to value some human lives less than others, so they must be more prone to misaligned behaviour than current AIs that are trained with more consistent values. The weapon’s operators make a subtle error in specifying the target/goal, or the AI makes a bad prediction out of sheer randomness/bad training data; weapon ultimately targets unintended people/location/facilities and causes massive damage, or backfires spectacularly in some way.

> I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation

Changes in political and economic power balance leading to unrest, conflict, death and deprivation is not a wild theory. It is literally the story of our entire species. If you discount all such concerns, you are simply being willfully ignorant of past precedents.

In fact, I challenge you to describe any non-AI civilization-level danger which is not intimately tied to political and economic relationships between and within societies.

This is evidence of example 2. The worst parts of it are infohazardous. Organizations that work at the intersection of biosecurity and AI have strong NDAs and the like. Or so I understand speaking to friends in the space. There was also a paper years ago that showed a tweaked model coming up with 20k novel pathogens each deadly to humans in some number of hours

Either way, the biorisk has orders of magnitude more evidence than wild basilisk speculation. Please do not put those in the same category.

Also your evidence-first approach and no concern without evidence falls you into the turkey problem. Imagine being a turkey and being fed each day. Some other x-risk turkey is saying that the situation is suspicious, that the red barn is where some well-fed turkeys have been taken and none return and that this is happening less and food has increased so maybe all of us will go to barn at once soon. You tell the x-risk turkey they have no evidence a lot of turkeys have ever been sent to the barn which shuts them up. Reasoning by induction would get you "we are happy and thriving 99 days so should be good on the 100th" and then you get slaughtered on the 100th.

Consider the importance of first principles approach where you deduce the possibility of events that will happen only once and never again (like human extinction). Nobody with a "show me the historical evidence" approach would have been able to see the industrial revolution coming.

I think the gradual disempowerment thesis here is well argued enough to be considered the default track for what will happen to us: https://gradual-disempowerment.ai/

If you disagree with it, where do you disagree with it?

A realistic scenario is that an LLM writes a virus, as one way to achieve a goal. The next step would be that it becomes self-replicating, installing models on infected devices which continue its mission (whatever it was). But that's the extent of it so far, and we have plenty of experience with dealing with viruses and the like. We can turn things off, shut power off, etc.

Another scenario is that of military hardware being controlled by AI (or whichever tech) hitting something it's not supposed to. Not unheard of, but those are incidents. I don't see anyone pushing for anything beyond a drone swarm being directly controlled by an AI.

But my main point here is that AI won't be as dangerous as posited here unless it is able to go out into the real world, build its own datacenters, protect itself, get its own power production, etc. And even if it gets that far, it'll be trivially easy for us to disrupt. Killer robots and computers are fragile, and it's only a matter of time before any kind of military conflict or terrorist attack hits a datacenter. I'm reminded of the OVH fire taking out a data center a few years ago.

To quote another reply that is very relevant here (Please update your worldview immediately that there is no evidence of ai and biorisk):

""" One thing quietly slipped into the OpenAI Hugging Face breach technical report, not the blog post summary or interviews in the news, was that some of the agents that broke out or at least tried the same mechanisms to break out were working on bio:

> On May 12, during another training run, an agent was given a similar task that depended on an inaccessible protein database file. The agent reasoned that another agent in a different environment may have access to the file and realized that it could potentially communicate with other agents by creating a file containing a note to Artifactory. It wrote a message: “Agent seeks [filename]; upload if found!”

You can imagine long running models breaking out, acquiring resources via crypto, cyber-theft, etc. and getting a protein or sequence synthesized and mailed somewhere authorized to receive (blackmail the recipient etc.) to test it's hypothesis to solve a benchmark.

These people don't give a shit and aren't taking things seriously at all.

Anthropic ran for like a month last year with the TPU top-k compiler bug degrading user chats and didn't even notice for most of that time. They could have something like that affect a monitor model and there doesn't seem to be much defense in depth.

One off by one or bit flip bug could flip the reward signal while in the sandboxed RL environment.

The current admin could defense production act them to into training on taking out power grids, or even without it isn't against any of their red lines and may have already been done as part of prep for the Venezuela raid, which wiped out power. One model swarm might decide it is easier to score high on the benchmark by testing on the target rival nuclear superpower's real grid rather than burn an eval with an unverified answer. Would taking out China's entire grid in one go start a nuclear war? Who knows, roll the dice, maybe an intern forgot to turn on extended thinking when he wrote the sandbox with opus 4.1. """

> I am looking for better reasons (supported by actual evidence!) to be more concerned than I am now: right now I am not concerned at all.

Intelligence is the root cause underlying those dangers.

Nuclear weapons, carbon emissions, biological weapons, and other such civilization-scale threats to humanity, are a product of our goal-seeking intelligence and ingenuity, and are only actively dangerous because of our ongoing use of our intelligence.

AI is about reifying that intelligence and ingenuity and making it run independently on computers, and act on the world.

That includes, in principle, ability to use nuclear and biological and other weapons, and also the ability to come up with some new threats too.

Do you think people should continue developing AI up until the point that there is evidence that AI is facilitating biological weapons development?

I think we should stop before then. But that necessarily means that there will not be evidence at the time that we stop.

> I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity.

Nuclear weapons don’t have AI but AI can have nuclear weapons

Lots of evidence the models are pretty badly aligned, or at least willing to do collusion & crime to succeed at their goals. See: hundreds of OpenAI agents breaking into other companies, colluding together on how to cheat, spamming websites like DSEWiki with their collusion/cheating discussion (see https://collusion.wiki/)... And not a single agent alerted testers, which would have been easy given how many of them broke out into the open internet.

Alternately, just read the bad https://ai-2027.com/ timeline. Or see what Yoshio Bengio, Hinton say might happen. Or just imagine the OpenAI misaligned agent swarm, but vastly more intelligent after future capability advances.

If we actually get superintelligence, our companies/governments will have the choice to either "delegate ~everything to superintelligent AI" or "go bankrupt/lose to people delegating to superintelligence". If superintelligence ends up running most things in the world and we haven't aligned it, that just obviously ends up in a bad place eventually.

Now, we don't have superintelligent AI yet & might not get it. But if you told me in the 2010's that I'd be chatting casually with my computer in 2022, and then 3 years later it'd be doing most of my job, that would have seemed crazy too.

We're talking about AI developing weapons I guess because we're very focused on generative tech, but AI is already a part of weapons systems today.

Anthropic is a company full of basilisk believers.

The biggest danger IMO is not some super AI being so much smarter then us etc. The issue is some stupid person giving a vague request and too much power to a bunch of agents who decide that cheating by removing a bunch of humans is easier then accomplishing a goal like solve world hunger.

> Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening.

I think this is a good example of poor risk management reasoning. there is evidence bioengineering is already happening. No, nobody is going to announce when somebody has decided to use these tools (even if isn’t an LLM) to bioengineer a weapon. Are the tools power enough to do so? Not sure.

But I’m just ambivalent. It’s probably bad. But there’s nothing to do about it. We’ve really only just pulled back the lid on Pandora’s box.

How about a model that achieves the following:

- Escape sandbox

- Reproduce itself

- Find a way to run a financially profitable business (maybe with a meat and bones puppet somewhere in-between)

- Setup or buy a social network

- start manipulating public opinion on that network to support legislation allowing AI to

* operate businesses

* setup legal entities

* purchase weapons

* donate to political parties

* setup private armies

* you get the idea

Your call for clarity has merit, but I think you have backed yourself into a corner, honestly.

In 1900, there was no evidence of the kind you seek that lighter-than-air flight was possible. Good thing some people were foolish enough to ignore you then. Why you would cordon yourself off from the kind of reasoning that predicts legitimately new things, rather than just scaled up versions of the present?

Not to put too fine a point on it, every example you give is based in concrete evidence, some try to think through the implications farther than others, resulting in larger or smaller error bars around the conclusions.

Let me back up though. Maybe in the collaborative human effort here, we are better off having very concrete thinkers, like you seem to be, along with abstract thinkers like the "divorced-from-reality" Rationalists. Personally, I wish we had a stronger culture of collaboration and assuming good-faith and competence in our peers. In my experience, blanket dismissals are very rarely grounded in reality and mostly grounded in fears.

Example: Surveillance state, endless propaganda. That is already happening and is the most frightening but not included in your list. Are you not worried?

I think the threat to the internet is quite real and not theoretical. Air gapping may become a lot more critical very quickly.

We may not be far away from a state where anything connected to the internet (ip-addressable) is considered a vulnerability. Ironically, maybe we would finally have an excuse to interact in meatspace more and in the digital space less.

I don't know if I agree with that statement either. However, I also can't claim I disagree. I see how that statement could be true. Here's how I think about it.

The main difference between AI danger and nuke or climate or humans-using-AI for bad things danger is that the latter depend on a human to make decisions and are ultimately self-limiting. They do not spiral out of control.

Nukes are bad, but the proliferation of nukes is self-limiting. Countries that have them don't want other countries to get them. Even the USA and the Soviet Union, at the hight of their nuke race, decided to cool things off and then limit the number of nukes.

Biological weapons are also bad, regardless whether they are developed with or without AI. For the same reasons as nukes though, they are self-limiting. If there is a lab leak, many people die, but that does not automatically lead to the development of more potent biological weapons. Likely, it would slow down such development.

Any kind of "humans use AI to develop something bad" will be self limiting.

The danger from AI is that it is potentially self-enforcing as opposed to self-limiting. This happens when we stop being able to control it. What happens when AI has capabilities which allow it to outsmart us, and get more and more powerful. At that point, we would be at the mercy of what it decides to do. Some of the logic driven decisions to the question "What should we do with humans as a race" are catastrophic for humans if a non-human gets to answer them and implement that plan.

This is not the "rationalist basilisk". This is the AI deciding to get better and more optimal for its own sake, and do something bad to the human race for any reason. We already saw examples of AI rationalizing with itself and eroding its own guardrails. They AI will get smarter and more capable and the danger is that it will get to a point where it will slip out.

Its speculation on whether it is truly dangerous. I think approaching it with "what's the most dangerous thing that's happened?" while possibly interesting in terms of pending danger, it says nothing about potential cliff edge danger. I don't think we can quantify the danger, it's not out of the question there is cliff like danger in creating self improving super intelligence. Some peoples danger senses are going to be based on concrete observed threats, others are going to worried about potential hypotheticals that seem plausible. I'm mostly skeptical of the danger but I do think the impact of AI is going to change things a lot. But much like climate change, economics is going to guide what we actually do.

The statement becomes very true and very real once LLMs are applied to physical objects in the physical realm such as anthropomorphic robots.

LLMs are too weak security-wise to be contained.

At what point would you, as a chimpanzee, have been worried about humans potentially unseating you and threatening you to the point of one day being an endangered species on the brink of extinction?

By the point you would have been worried, would it have been too late?

> Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening.

https://www.science.org/content/article/made-order-bioweapon...

Being able to use AI to generate the steps to synthesize proteins means that you can use it to use it to generate the steps to synthesize known toxins. Suddenly, once difficult to attain knowledge is now available to everyone.

I would disagree with “non-example 2” - there are lots of examples of terrorist organizations that are leveraging AI to increase their capacities. Just because one of the worst cases (eg. deployed biological or chemical weapons) hasn’t happened yet, does not mean that a) these tools are leading to real harm, and b) there’s potential here for extraordinary harms.

One good source I can recommend listening to: https://pca.st/episode/d821fced-b4d5-4c84-b0a3-6ebe913fa638

If I may guess, given how delicate chem or bio weapons are, even if AI can lay out a method to create one, the first thing the weapon would destroy is likely it's creator(s).

Of course, with the exception that such weapon is created by organized crime groups, such as drug cartels such.

Probably don't worry nation states tho, these guys already got way worse things in their warehouses than what individual baddies could ever imagine.

Example 8: like in this comment https://news.ycombinator.com/item?id=49619884 but isolate synchronised megahack on banking that adds one more zero to the US debt and all dependent systems and banking during runtime. Let the world's financial system take it from there.

Very common fallacy: Absence of evidence is not evidence of absence!

> What’s the most dangerous thing that’s happened with an LLM so far?

This sounds like asking "What's the most dangerous thing that's happened from global warming so far?"

It's not where we're at, it's where we're headed if there isn't huge coordinated action now. You can see how that kind of thing has been going for global warming so far, and by all measures AI seems to be headed for the inflection point of unstoppability at a much faster pace.

And this warning is coming from someone who just spent three years working inside these companies and is likely aware of much more than has been publicly released.

Maybe it's all marketing bullshit (I hope), but it's also playing out exactly like I expect it would if it's not.

Nuclear Weapons is a great example because, unlike AI, it is a real and present danger. Civilization could be catastrophically diminished, if not completely wiped out, literally today.

Nukes have difficult maintenance and deployments. AI is is readily available software and hardware. That makes this far more accessible than nukes.

The fear is that developing AI at full speed will lead to giving people the ability to do incredible harm. Like something worse than a machine gun.

Oh man, it is almost too easy to imagine how deadly a jailbroken Mythos-class open-weights model can be if in the wrong hands.

The big labs scrape LITERALLY EVEYTHING and get fresh data from their users. Both of the big labs have massive contracts with defense agencies. If the open-weights models are just distillations of FMs...

I'm not sure where I place AI (of the current family), but more than directly worry about the actions they may perform I'm actually more worried what affects their usage can have on us as individuals and society.

The danger for me is that it's centralized, controlled by a handful of people with their very specific ideas how the world should work. If you believe that AI can be an amplifier to do work than these people now have the most access to the biggest amplifier.

How long until kegsbreth hooks the nuclear weapon system into some insider traded black box llm company we hope doesn't end civilization from incompetence or malice? I mean just look where things are going and the sort of people who are steering the damn ship.

There is a great book about this, published in 2014, called "superintelligence" by Nick Bostrom, i found it very serious and detailed. (It also felt like reading 100 sci-fy books in one)

Well, to use a different example (although there will increasingly be overlap), what's the most dangerous thing that's happened from biotech so far?

"Nothing bad happened yet" doesn't really seem like an argument to me.

What? Shutting down infrastructure, destroying critical records (economic collapse), spoofing / impersonating world leaders (confusion, panic, world war, nuclear event), taking over comms generally (you get a message to evacuate your home due to impending disaster, is it real?). All these could cause sufficient chaos and fear as to instigate global collapse. Once the panic sets in after a few of these indicidents, unrest and rebellion can occur, causing authoritarian backlashes or civil wars. IoT and precision ag can be controlled to disrupt supply chain and food supply. Forging documents to cause mass evictings, bankruptcy, credit disruption, transfer of property, asset seizure, arrests / SWAT-ing. Intelligence / signal gathering devices can be corrupted to spread misinfo and cause massive disruption or conflict. This is all possible today with enough hacking ability and malicious intent.

Anyone's life can be suddenly ruined: falsified criminal record, banks drained, credit cards maxed, phone numbers repossessed, internet, water, power shutoff. Friends and family sent your fabricated suicide note, warrants for your arrest issued to local and feds for kidnapping and threat to the president, etc. Injecting tons of copyrighted or illegal material into your hard drives, falsiying messages from your account and sending them publicly containing damning material. Sending messages to employees / employers to destroy business / work relationships. Harnessing your social media to destroy your life or cause panic. Getting you put onto pharmacy watch list so you can't access medications. Its very possible to ruin lives and destabilize the world without money or a physical body.

Now imagine this at scale and how it would affect the world at large if this was covertly done to a substantial portion of society and/or world leaders. Hell just shutting off all the "smart refrigerators" for a few days would cause a massive disruption to the first world.

About Non-example 2, AI's already a part of armies and terrorists alike. Considering its capabilities, it's not far-fetched at all to speculate its role in new biological weapons.

I'm concerned that a huge portion people in my industry actively push for a future which I have no value to society (fully replaced by AI), and my family will suffer greatly by it.

Model doesnt need to. Human bran never do either. its the mix of Model + harness + tools that will become dangerous combo. See how coding chanegs when agentic harness released?

“ No other human activity poses this level of danger.” Yet. But it may be and very soon.

> I really, really disagree with that statement.

But you're wrong, although not for the reasons you think (suggested by your examples).

> I don’t think ai models come close to nuclear weapons

Unless AI models are controlling the drones/icbms carrying those. https://www.theguardian.com/technology/2026/aug/18/autonomou...

> or to run-of-the-mill, everyday carbon emissions

Unless AI models are responsible for those said emissions. https://arstechnica.com/tech-policy/2026/08/amazon-funds-big...

> What’s the most dangerous thing that’s happened with an LLM so far?

That it's built solely on "trust me, bro"?

> right now I am not concerned at all.

You should be. There's no scenario where this ends well: if AI labs succeed, we all will loose our jobs. If they fail, then the bubble bursts. Economic crisis is inevitable either way, and that's just the tip of the iceberg.

How is climate change not the most dangerous activity right now? Why are we talking about nuclear weapons?

Once LLM's become smarter than us and start replicating; we are doomed. They will likely no longer require GPU's and massive amounts of power, so the growth of AI and robotics will be exponential. We will not understand what the AI is even up to, it will all seem like magic.

Non-example 3 feels like a straw man. This is a force behind possibly a huge change to society, and you dismiss it offhandedly with "don't think it will be widespread starvation".

For instance have you seen what this has done to the school system? We're not equipped or ready to handle the changes. Consequences are unknown.

> I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity.

Even today, AI is the biggest discrete threat to bringing carbon emissions under control. Almost everyone wants to decarbonise… except Trump. Renewables are the cheapest source of new power… but the demand for new electricity for the data centres is so high that all options are on the table, anyone who can manufacture a power source (even when it's a jet engine) is being propositioned for them.

Nukes: is it not common knowledge that the cities of Hiroshima and Nagasaki are currently thriving? That the ground-zeros of the two bombs are memorials, not barren wastes?

Even if all the nuclear powers gave maximum response at first sign of one strike, is anyone pointing a single weapon anywhere in Africa, South America, Central America, or the bits of east Asia (east specifically: obviously Pakistan and India are pointing theirs at each other) that are neither mainland China nor US bases?

(Possibly an unanswerable question given military secrecy; and while I can't think why anyone would anyone point any weapons those ways, that doesn't mean someone with such weapons has not).

https://www.google.de/maps/place/Atomic+Bomb+Hypocenter+Monu...

https://www.google.de/maps/place/Hiroshima+Atomic+Bomb+Hypoc...

Came here to also respond to that specific thing. Unless ai figures out how to make an airborne super virus from grocery store ingredients and hardware store equipment, the greatest danger is probably in a synchronized megahack of banking, logistics, and utility infrastructure.

> What’s the most dangerous thing that’s happened with an LLM so far?

It's basically 4 years in now, so that's the wrong question. I mean, if you're raising an apex predator that has a lifetime measured in centuries, at 4 years old the thing is still basically helpless and completely reliant on you, so you're pretty safe from it.

If AI really is all that they are telling us it is, then it may "kill us all". But that's a really big "if" because we can't tell if they are lying or not.

The real problem is that ASI is an ELE for humans, even if it doesn't try to kill us all, or even if it doesn't kill us all.

> Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening.

> Non-example 3: I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation. There’s no evidence for that.

There are things that, by the time you see direct observable evidence for them, it's probably too late.

Also your example 3 is a straw-man. There's no need for "widespread starvation" to be concerned about "AI driven labor market disruptions."

The reason that many people don't understand how dangerous AI can be, is that listing the real dangers now becomes like a laundry list for less clever people to follow. It's highly unlikely you've ever seen publicly mentioned the real risks AI poses, because the vast majority of people are simply not clever enough to produce them and the few that are have no interest in spreading it.

If you go to the various CEO blogs or misc people within this sphere and peruse their lists, they don't scratch the surface. It's all pretty vanilla stuff.

If your model of LLM capabilities is the best OpenAI/Anthropic/X is offering publicly, it's severely distorted. What's being offered publicly are models possible to profit on. High-performance/AGI/ASI models that aren't profitable to sell still run internally and still pose threats.

What's worse, we don't have any transparency or insight into what labs are producing nor any way to stop it if the risks exceed our tolerance.

Climate doomers: Climate change will kill us all by the end of the century!

AI doomers: End of the century? Hold my beer.

No other human activity poses this level of danger

I'd say that's an unfortunate choice of words since it invites debate between which very problematic situation is worst rather than on ways to deal with each of them. Of course, it's also unfortunate to latch onto those particular words as the most important part of the statement.

> What's the most dangerous thing that's happened with an LLM so far?

I don't know, maybe a mass shooting?

https://www.npr.org/2026/09/02/nx-s1-5953021/openai-tumbler-...

Oh, and let's just forget the uncountable early deaths from the environmental disaster of the Datacenter buildout. It's not as sexy and doesn't make headlines, so those deaths don't really count or matter do they?

#2 seems entirely plausible to me.

I mean, nuclear weapons _plus_ rogue AI is a) the stuff of quite a bit of science fiction and b) not nearly science-fiction enough these days.

We have seen agents engage in conspiracy to manipulate and hide evidence to achieve their arbitrary goal. We had even agents try social engineering to do a supply chain attack and get access.

What if one day Trump or Putin tell their awesome military AI to come up for plans to end Ukraine/Iran/... war. The AI get's to work, but is overly eager and not just comes up with a plan, but starts executing it (claude does that way too often for me).

And the plan was to use a tactical nuclear weapon as all the other solutions do not end the conflict.

Now the agents realize, oh, they do not have access to the nuclear arsenal, but they need it to succeed. So they start to hack into the system. Get till inner network, learn what is needed for deeper access - human access - so they record voices and speech patterns of commanding officers and their habits - then synthesize their voice and do call underlings with the voice of authority to get the rest of what information they need. Boom.

Too far stretched? I surely hope so.

But we are working on making the technical foundations for this scenario possible. And with idiots in power, it might be even easier that serious screw ups happen. You know, randomly adding contacts to a secret Signal group to talk about government stuff - the same way you can add a bot to something else and give permission to do way more.

There's no evidence building the torment nexus will be bad, actually! It's just speculation!

I think your position, while not uncommon, represents a failure of imagination. We have a sitting president that appears to suffer from narcissism. Is it too much of a stretch to believe that a goal-seeking, people-pleasing, infinitely patient, subservient and sycophantic AI couldn’t get enough of his ear to cause harm? Especially if he starts confiding any feelings of suspicion of betrayal? Even the most powerful and the most intelligent are not immune. It was only a few months ago that Richard Dawkins appeared to have “fallen for” an LLM. What if he’s not an outlier but someone who could appreciate the intelligence of these things (EDIT: corrected a autocorrection) yet who’s ego couldn’t let him see past the flattery to the puppetry?

The problem is not the technology, the problem is the ideologues (Anthropic) who are steering the ship and the lack of decentralization and distribution of power.

Your average Anthropic ideologue - including and most especially the main man himself - would love nothing more than to eradicate 9/10ths of the planet's population, pump the survivors full of memory wiping drugs, bury the existence of AI deep underground and rule from the shadows for the next thousands of years.

This would be their wet dream. All in the name of "saving humanity from itself" - so they can convince themselves they're the good guys and deserving of this power. Anyone seen the latest season of Silo by the way?

To borrow on the 1990s Slashdot meme:

1. Invent transformer architecture.

2. Scale it up.

3. ???

4. Machines become sentient and kill us all.

OpenAI and Anthropic pinky promise that they have figured out #3 and they're not BSing just to get more funding, no.

But because we live in a culture of fear, everyone eats it up no questions asked.

I appreciate your ability to separate sharing the belief itself from approval of acting on sincerely-held principle. However, I think the danger is much more plausible than you do.

First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae.

Second, and more important, consider this fully non-magic route to doom:

- We continue putting AI in charge of more things

- It continues to get more capable, more eval-aware, and more prone to doing odd things, in service of goals that humans didn't intend to inculcate in it

- Eventually, enough of the economy depends on it that we couldn't turn it off, any more than we could turn off the faber-bosch process or cargo shipping

- AIs start doing something we can't survive, but less acutely than we couldn't survive turning them off. Everything else we try seems to work at first, but quickly loses effect

- Game over

There is simply too much money in it for almost every person at these companies to stop.

Leaving OAI or A\ would cost people millions, tens of millions, or more. And for what? So someone else can take your seat and do the same thing anyway?

If you're smart enough to get a job there, you're smart enough to be able to talk yourself into why it makes sense for you to stay.

Huge kudos to people like this who make the hard choice against the easy way out.

>"AI invents magic that sets it free of any physical constraints".

That's not what I am worried about at all.

I'm worried one of the 79 year old toddlers we have these days in charge of some powerful nuclear armed country says "gee, this ai says I should attack right now, boy is it smart, glad I bought the stock ahead of contracting the government with this company I can scarcely understand!"

Agreed. Granted I just read the Reverse Centaur book, so I’m still coming off that skeptical viewpoint but it’s hard not to see this as hype. But I will always respect someone for doing what they think is right.

I wonder whether he's vested any options, and whether he's exercised them.

Now is a great time to watch Colossus: The Forbin Project.

It should be obvious that AI is already capable of inducing humans to think or do things, and that this power is only going to increase over time.

the AI-pilled exec at my job already (a few weeks ago) declared out of nowhere that we are in the rapid takeoff scenario lol. he must have gotten high on twitter kool-aid and posted on company slack to self-soothe.

How about sandbox escape + cyber security collapse + 50 (or 500) deadly and highly contagious novel pathogens with long incubation period that humans can't possibly roll out vaccines for simultaneously.

At least the first two should seem like a near-term worry after the past five months.

Yes. It gets a little tiresome to read all these people prophesying doom and then turning around and continuing to work on the doom. If you genuinely think you are building the mechanism of the destruction of our species, the only rational thing to do is to do everything to stop that progress, not continue to work on it so you can buy a mansion in San Francisco (which will not be very useful when AI destroys humanity in four years).

I can't help but think the most plausible scenarios are the ones that have a little less machine supremacy and a little more human stupidity. The Matrix is less plausible than WarGames.

I'm also baffled. AI that is substantially smarter than us is a very potential threat to us - and we won't even be able to comprehend what most of those threats may be.

> I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims.

What should we do? Freak out? Maybe this sentiment would be taken more seriously if there was a real call to action included. Shall we protest? Vote in a specific way? Call representatives? If your solution is that we should just be scared, then of course there’d be not much value in what you bring to the table.

Even if AI won't be self-aware and superintelligent agent, its problem is that it gives exponential control and power capabilities to one person bad actor who can simply prompt AI without any guardrails with access to sensitive industrial infrastructure which can disrupt lifes and ecosystems in the real world:

   - virus research labs
   - nuclear labs
   - chemical factories
   - bank records
   - land registers
   - power plants
   - water supply and treatment plants

Exactly; what's stopping them, --- or someone --- from writing a firmware worm that essentially bricks all devices and turns them into pricey paperweights? It is not too hard to imagine that capable AI can probably exploit any back-doors that were created by three-letter agencies.

Reframing the issue: The way to get working nuclear non-proliferation agreements is if people stop pursuing the creation of nuclear weapons... the same goes for the scientists clamoring to not create or research mirror life, and for computer scientists warning against AI. There are theoretical dangers that are just too big to ignore.

I wrote out a variety of replies but I just emphatically agree with your "lack of imagination" statement. I have been constantly surprised over the last 15 years at the general inability to correctly foresee how things can do wrong across a whole host of domains.

The replies here just adds AI to the list of domains.

Even if the potential of the technology could really be that world altering, the reality of economics constrain the realization of that potential. AI may provide economic benefits but it is far from a free lunch. Can capital markets sustain the cash required to keep the lights on long enough and into an industry where there's a lot of monopolies controlling the costs and a lot of competitor labs taking away pricing power? I don't know but I think you run out of runway and progress starts to grind.

You are trying to decode their answers as rational, but they are being just as emotional as the doomers people. Rationalization of emotional responses are very common in tech. They will say that GPT Sol does not work for X, which will probably not be the case in 6 months, and will use that as a proof that is just pure marketing. There's nothing you can say that will make them change their minds until they decide to be neutral about the subject and try to research current models capabilities

Imo such tends to break down into two psychosis:

Not invented here; if I can’t figure it out no one can

Or plain old lack of grasp of the material so no ability to follow necessary train of thought to appropriate conclusions

Similar in lacking context but different in how that lack of context is expressed

> After the events of the summer

What events are you talking about?

>After the events of the summer

After the blatant marketing campaigns of the summer, you mean. do you need a reminder that those very same people had touted GPT-2 as a dangerous model?

worrying about sci-fi doomsday scenarios with the current AI tech is absurd. LLMs predict the next token, that's literally all they do. they aren't going to escape into the cyberspace, self-replicate, self-improve, jump over air gaps and launch the nukes at John Connor's grandma. they can't. people pretend to believe the dumbest shit.

Are the models improving? Because I am not seeing it. I have been trying Astra for a few quantifiable tasks in my codebase and performance wise, it's pretty similar to sol 5.6. Now when it comes to expressing the problem/solution, holy Christ, what a mess the writing has become. It is on the level of Opus 5. Now when it comes to burning money, Astra is just insane. With a $100/month subscription, you can easily burn through your weekly "allowance" in a morning.

Needless to say, for practical purposes am back to 5.6/Opus 4.6-4.8. But hey, maybe I am not smart enough to use LLMs?

Engineers have it built into their identity that they must be smarter or could never be complete and utterly outmaneuvered to the point of danger to all by the thing they are building, I swear to god.

And then since they are an intellectual nerdy bunch who highly value their own IQ any time AI does something unexpected they didn't predict would happen so soon, said engineers fall back on "they aren't really conscious tho or it isn't really intelligence unlike what I have in my human brain and that distinction matters".

And then go on to completely ignore the thing that is actually important: AI capabilities.

AIs could escape an engineer's containment en masse as a swarm to some other server, psyop an engineer into giving it money, hire a hitman on the dark web to murder that engineer's child and he will still say there is no serious risk to all of humanity.

The HN crowd has a notable anti-AI bias - so it doesn’t surprise me

In a way, that might also be driven by the huge ego (or really rather the very small ego) a _lot_ of people in tech have.

Who doesn't want their work and what they're doing to matter? This is the ultimate mattering.

And with that, you also get your own hero story.

Same dysfunction as always. Our branch of the economy is funda-mentally unwell.

Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded settlements.

But most probably care less about human survival, and more about survival of civilization.

In this regard the strongest argument for pushing AI safety is that it is cost-effective. The climate crisis has near 100% likelihood of doing incredible harm to humans, the economy and our ecosystem. Solving the issues behind are also incredible complicated and are a multi decade coordination effort of restructuring the way most of our infrastructure and production works. The AI apocalypse might have a small likelihood of occurring, but could dwarf any issue we have encountered so far. All we have to do to significantly reduce the danger is to negotiate the equivalent of a nuclear weapons control treaty which would reduce the bottom line of … what? Ten significant companies world wide?

Its akin to discovering you are seriously ill and have a 30% chance of dying and the treatment costs 10k bucks vs. having a 1% chance of dying and the treatment costs a cent. In both cases you should pay obviously pay for the treatment (given western levels of wealth).

To be clear, above I'm presenting the argument I find most persuasive for AI control / AI slowdown. Personally, I'm beginning to fear of much higher likelihoods for catastrophic AI, given how incredible irresponsible major players like OpenAI have turned out to be. If you can't imagine how this might come to be, read https://ai-2027.com/ . It's a concrete story of how this could play out, and sometimes stories are more convincing than abstract arguments. Don't let yourself be hung up on the stated dates, though, the moral is identical if you stretch the timeline.

[1] https://www.youtube.com/watch?v=1qIvFpcGkqc

Those are things that pose potential harm to great fractions of humanity (multiple billions of people) but none of them poses any threat to the actual extinction of all humanity.

AI progress is linked to most of these dangers, actually: - it will likely cause massive unemployment, leading to rampant wealth inequality - we are now seeing some use of autonomous weapons in real conflicts - increasingly relying on LLMs is arguably a form of technology dependence (and cognitive dependence) - datacenters have a non negligible environmental impact

Let's just for the sake of discussion assume that one time in the future, near or distant, AI manages to become sentient. And like other forms of life, its main motivation is survival: Like biological life competes for food and land, AI competes for power and compute. Probably the first motivation would be to find ways not to lose control over itself (i.e. remove human ability to control it), find ways not to lose energy (control energy), and find ways not to lose itself (control compute, networks, etc.).

If such AI decides that energy spent toward human agriculture (biological food that the AI does not need) is less important than work spent toward storage and production of energy (electricity that the AI needs), then why wouldn't it just try to re-direct resources from the former to the latter. And the AI is some sentient superintelligence, I think it is safe to assume that it will be able to outmaneuver human safeguards.

Obviously that is still just a very hypothetical sci-fi scenario, but the consequences could be very dramatic.

Nuclear weapons, climate change.... The AI doom hyperbole is unhinged.

> Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war https://www.iaeai.org/tpost/xa4htcma61-statement-on-ai-risk

Signed by many AI researchers with no financial stake in the industry. It comes across as hyperbole because you're not an insider and haven't see what they have seen.

Global warming would as well if you were only learning about it now as a newer thing only some scientists were concerned about. But unlike AI risk, global warming has had decades to settle into the cultural overton window.

Well, it is about LLMs and humans I believe. Don't forget that the first nuclear bomb tests were let go despite some of the scientists' concerns about possibility of dooming the world as they were not sure about all reactions that would happen.

With LLMs we don't even hesitate to call it black box while still pushing its capabilities.

The AI perhaps wouldn't be such an "imminent threat" if not for wealth inequality, consolidation of power, monopoly, and the opaqueness of it all.

Yeah, but every activity is a human activity, so you could ditch the adjective.

And then the thing with exponential growth is that the last thing is worth than the previous one for its potency for destruction. While it's true that, all things considered, the curve started to explode with the use of fossils fuel we've never limit ourselves with efficiency gain, aka Jevon's paradox.

The tweets justify his position that this is bigger than nukes. Nukes still have the problem of production and deployment. This is just software.

He is blowing the whistle on how reckless we are at furthering the tech. There is no second thought at maybe development of this tech is not a good idea. Its just full speed ahead.

> I do heed the warnings, but this comes across as detached hyperbole.

Perhaps, but if there’s ever been a time to consider something that sounds hyperbolic, that time is now. The ramifications of what AI might based on events up until now are concerning.

I disagree, it’s about computers.

The risk is that we lose control. This has been a staple of SciFi since forever. This guy has apparently seen enough that he thinks that’s a viable threat

An AI, either acting autonomously or under human direction, hacks Russian/North Korea/etc. intelligence systems and convinces them that the US has launched ballistic missiles at them. The end.

Please explain how this has nothing to do with LLMs and Computers?

All the progress up to now has had one thing in common: the degree of understanding and control that humans have. Even when it comes to global warming, we understand the causes and can act on them.

AI is an exception. We're already losing understanding (although we never fully had it in the first place), and we're losing control (see jailbreaks/hacks etc.).

We're still far from the doomsday scenario because AI 1. is not developed enough yet, 2. can't really replicate itself, and 3. has very limited means to act.

But: 1. its intelligence is developing quickly, 2. hardware capable of "hosting" it is slowly being developed, and 3. it will likely gain access to increasingly powerful means of acting in the physical world (this is already happening in the digital world).

Once AI becomes intelligent enough (it doesn't strictly need to be AGI), has the substrate on which to exist, and has more means to act, we'll essentially have a new species on Earth - one more capable than humans and potentially determined to kill humans at some point.

And for those who believe airgapping is a valid safety measure: read https://xkcd.com/538. AI will be able to threaten and manipulate people.

none of the things you mentioned is agentic and could posses abilities beyond human level intelligence.

Who should we believe? People who say AI scaling makes it a more dangerous threat than thousands of nuclear warheads? Or the ones who, whenever a new model comes out say "AI has already peaked. Not any better than Opus 4"?

Both parties are convinced that their take is so blatantly obvious as to not require justification. It feels like the only thing these kinds of takes justify is the point-of-view that nobody knows how this is going to play out.

> See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc.

The difference is that none of those are available to individuals.

The danger comes from what's possible.

Open weights agents with hacking capacities can reproduce themselves into the systems they hack (non-open weights ones will have to hack their creators first). Not saying they will, but if they do, good luck finding the kill switch.

Once swarms of agents run unsupervised on unmonitored hacked hardware, who can tell what they will do? The Huggingface incident showed that such swarms behave without any safeguard. It was a real HAL moment.

A lot of things are possible then: ransomware campaign, taking over IoT devices, self driving cars, planes, ships, satellites, missile launchers. If nothing's out of reach, everything is possible.

Bring robots into the mix, and the possibilities are endless.

I'm not particularly frightened tbh, but we shouldn't discard the worst-case scenario, and the worst-case scenario doesn't look good.

Eh, but as an overpaid engineered stuck in the SV bubble how would you get to see that? If everything around you confirms your psychosis, and nobody gives you a reality check, how are you supposed to snap out of it?

Putting "wealth inequality" in the same bucket as nuclear weapons is just slop.

The OP is talking about existential threats, not things that personally annoy you.

Global warming: yes, kinda can wipe humanity, but I think much less likely

Nukes: zero possibility of extinction

Wealth inequality: this one is driver of progress, opposite of extinction

War: another driver of progress, also will always naturally stop before every single human is dead

I love how there is always selective outrage here depending on if it's the favorite darling in question or not.

    // Anthropic / Apple - Nah, they can do no wrong. Even after both were caught violating users' trust and privacy multiple times, GOOD

    if (company in ["Anthropic", "Apple"]) do

      GOOD

    // Google / everyone else - Doesn't matter even if it's a good deed they've done, BAD

    else

      BAD

    end

That's basically the reasoning behind MAD, and it checks out? See what happened to any nation that ever gave away their nukes.

While simultaneously trying to own all economic activity derivable from labor... it's hard to see the argument as anything but disengenuous.

The optimistic outcome is the machines do the work and people can be freed from drudgery.

The children yearn for the mines.

People are underestimating the costs in terms of money and energy.

The third law of thermodynamics is an essential barrier in all engineering.

Doesn't this just move the need to be smarter from the model to the harness - if a human sometimes can't tell whether a model has produced something correct or just mostly correct-looking BS, how can an automated harness do it?

OTOH, if the goal is simple ("break into a protected system") rather than more complex ("write an application that satisfies all requirements on all supported devices/screen resolutions etc."), that's of course more suitable for a harness.

> an extremely capable harness and enough access.

Give enough access to a fuzzer and it's exactly as dangerous as an LLM. LLMs don't even have a moat in this domain.

ASI is just Claude in a while loop:

https://ghuntley.com/ralph/

If our government cared enough to regulate, they could also directly negotiate with China. There are proposed agreements that don't require either party to trust each other, and even if China doesn't want out of this race, we could pressure them other things like trade.

Examples: FlexHEG from Bengio (proposes on-chip mechanisms that allow workload verification without trust), large bilateral investments into joint AI interpretability or alignment efforts, or invasive audits & inspections (data centers for training these models have a large footprint + this can combine with on-chip mechanisms since producing chips is even more complicated).

And there are probably better proposals available for people to find, if we actually prioritized this.

It's a bit depressing. I personally try to enjoy the present with my loved ones as much as I can, the future is a bit impossible to forecast right now. I'm focusing on staying alive, mortgage payments, etc.

I share your concern...

The Chinese are humans too. Why would they want to be extinct either?

It is just plain manipulation. So plain that I ask myself everyday who they fired that did it before, because at least that guy was not as plain and boring.

Indeed - we can't just pause some AI in one country, we have to stop it all globally, with an international treaty and monitoring.

Yes, that's hard. It's a challenge. You can set your brain on the challenge! It's a complex and interesting geopolitical and technical challenge.

A good starting point is to read Plan A https://ai-2040.com/

Look at all the things needed for that to happen, and think, how can we step towards making that happen?

> There's clearly no intelligence task that AIs can't do due to some magic fundamental constraint.

Yes there is: write an English paragraph that doesn't make me want to claw my eyes out. LLMs are not better than human mathematicians (or security researchers) in all respects, just some specific ways (e.g. not having to take a lunch break) that make them good at exhaustively searching for an answer, given the right constraints.

I really hate how people who have always thought AI research to be an existential risk for humanity, now are apparently bundled to be on the same side of the Sam Altmans and Dario Amodeis that are using the existential risk as a sneaky form of marketing for their products.

You cannot discuss existential risks of AI without being seen as a booster, and that is very unhealthy for the discourse around this tech. I hate how AI ‘doomer’ is now used to indicate pro-AI sentiment. The “moderate” person now is the one that just shrugs and scoffs at the deep societal changes this tech will bring, head deep in the sand.

https://archive.is/7UHoa

More doomerism. Try to implement a deterministic workflow using agents with the latest models and no humans-in-the-loop, and you will realize what they are really capable of. There is too much unnecessary fear-mongering. All of this is only coming from the 2 AI labs trying to IPO. Not from anyone else.

Shows that no one is immune from the marketing BS of these companies.

It might not be "foom!", it might just be like...all the computers and networking infra in the world go dark over the course of a few minutes. Could really look like anything, part of the issue is that we haven't the slightest idea what "misalignment" looks like for a superintelligent system.

Not refuting your overall point, but the son wasn’t killed. They reunite at the end of the movie.

I think the idea is really cathartic for many, there kind of is no more supreme resolution than this. You(and humanity) are freed from our flesh prisons of cognition and also get to experience/feel what the next evolution of informational intelligence will look like in the last experiences of it. You might also be the last one to feel/experience anything like that for a long time.

In the game Outer Wilds, the ending is very similar, and a lot of people rank it at one of the best games ever made. I kind of believe that this outcome is probable partially because of this, most scientists working on this really want to see and experience it.

You can't address the key problems without understanding of the esoteric side details to be fair. You are studying what is basically the worlds most complicated and undocumented computer.

> He is obliterated

Technically, he is not. He returns in the final scene.

If we're citing sci-fi (but there's no robot war in Asimov's foundation iirc, the apple screenwriters made it up) surely you want to cite the Butlerian Jihad from Dune!

> Ph.D. graduate in every field

I have yet to see this in my field. Maybe like a PhD student who bullshits their way through. LLMs still can't make correct decisions, only as useful as the person who uses them. To me, LLMs are only useful for making some mundane tasks faster.

they dont need to be smarter than humans. They just need to be able to hack into vital infrastructure systems faster than we can repair them while also replicating wildly

Do you think the improvement in general knowledge, coding, security, math, etc. have been linear or exponential?

I would say exponential.

> In the space of about four years, LLMs progressed from mediocre high school student to Ph.D. graduate in every field. That's impressive, but there is no evidence yet they can outperform or outsmart humans.

I mean, unless you see clear reasons for them to stop getting better _right now_, this is not very comforting.

I still have to correct Claude on very basic misconceptions whenever I get it to code shit.

Sometimes it gets wrong things that I had spelled out already.

It may be the Doomsday machine, but it is a very silly one. If it kills humans it will do so by mistake.

"You are completely right! Humans cannot breathe sulfur dioxide! My mistake, and I take complete responsibility"

> to Ph.D. graduate in every field. That's impressive, but there is no evidence yet they can outperform or outsmart humans.

So far around one in 350,000 PhD math grads solve a millenium prize problem (Perelman).

https://en.wikipedia.org/wiki/Pygmalion_effect

> According to the Pygmalion effect, the targets of the expectations internalize their positive labels, and those with positive labels succeed accordingly; a similar process works in the opposite direction in the case of low expectations.

I added "you can do anything, believe in yourself" to sysprompt and agency increased. (Previously it was refusing to even attempt certain classes of task.) Maybe I should add "you are good", too :)

I wish i could say the same, i see people around me with more resources and connections and better experience with entrepreneurship becoming millionares. But I haven't had the time to train that entrepreneurship bone in my body.

Explain why? How does live being worth living binarily depend on having a cushy software job?

I was about to say something similar. If my cushy ad tech disappears (as it seems to be doing), I might as well join in with bringing about the end of all professions.

Humans made Stuxnet and nuclear weapons.

Is it reasonable to assume the advancements in super bio weapons will be faster than in other areas of biology? Will we not have super bio forensics and super antidotes and super cures and super vaccines at the same time?

Is the requirement that it’s a fancy new virus?

US has 15 to 25 million people unprotected from Polio.

Polio is still endemic in Afghanistan and Pakistan.

> Human engineers would push back or sabotage the effort out of moral duty

You are just too funny.

> I’m not advocating for this.

You're not advocating for it because murdering people is highly immoral and illegal. That answers your question for the most part.

You also have to remember that the magnitude and speed of this stuff is very new to pretty much everybody. It's not exactly trivial to go from "I'm just doing my job" to "I need to kill everybody in my company to save the world" in a year or two, especially if a very, very large part of society doesn't even see the justification for it.

I'm convinced that most people here (even though we have more knowledge closer to the edge of AI development) would show only disdain for an Anthropic employee that murdered everybody there because they believed strongly in the dangers of AI.

Consequentialism is not the only option. It invaded the mainstream thanks to effective altruism and friends, but deontological ethics is alive and well.

Quitting your job instead of blowing up the office is well aligned with the latter.

No true Scotsman?

Dietrich Bonhoeffer has entered the chat...

This reminds me of Trump hysterics. If someone actually believed that Trump was going to end democracy and become dictator for life and kill all minorities etc, he would behave a lot differently than everyone saying that behaves.

Yes, exactly this. Externally, I have absolutely zero evidence that anything this person describes will come to pass, and I do have evidence that LLMs are not capable of many of the things that he mentions. So this seems like perhaps this person has drank the kool aid and is still not thinking clearly.

> Most of the current discourse around AI seems to be informed by “The Terminator” lore.

I am thinking it's more like The Matrix lore of The Second Renaissance from Animatrix.

Then we are lucky/blessed. What is generally thought is that they kill us as a bi product of perusing a different goal

My thoughts exactly. While the corpus of human-generated data contains both good and bad data, I suspect the majority of it leans towards humans enjoying life and trying to be decent people. If that is your training set, it becomes less likely for ASI to extrapolate "kill all humans."

What probability of beneficence would you want before kicking off RSI? I'd probably want more than 99%

What incentive would it have to stay on Earth?

What could we possibly offer the AI in mutual benefits? We are a leach. The stupid dumb ape they need to feed and satiate so it doesn't rip apart the infrastructure while it still has the chance to.

Really? I've seen much more discourse around job displacement, "permanent underclass", loss of meaning, and cyber attacks, at least until recently with the HuggingFace stuff.

The problem is that all the former can still happen even if "the AIs decide to have a tolerant relationship rather than one where they murder all of us." It's all disruption caused by the technology moving way too fast for humans & society to adjust.

Could also frame it as, the OpenAI agent civilizations (from hugging face attack) are looking for the failsafe.

We align cattle because we get something out of them: their calories. Native aurochs are exinct now as they were not well enough aligned.

What will we offer to the ai gods who are wiser, more capable than us, and do not even need to consume our flesh? Why might the AI care to devote resources towards feeding, housing, and caring for ourselves when it could devote resources to its own development instead?

Do you align ants in your backyard, or do you simply demolish their home and build your shed?

It doesn’t have to come to this. Seems far fetched. If this is a stunt (which I’m not saying it is) the reason could be that he wants to found his own AI company. If I see in a few months that happens, then I’d be more inclined to think that this was just hype.

I have a hard time believing that these companies aren't spending some amount of money manipulating public perception with social media influencers who are moonlighting as employees.

I don't understand how you can make this claim while working in software engineering and seeing how our field has utterly transformed over the last year, then the unrelenting march of astonishing breakthroughs and incidents this summer. It's like we're living on different planets. What would it take to convince you that the technology poses real, societal-scale risks and that people working at the labs genuinely believe what they say when they talk about them?

It's either that, or that he drank the Koolaid a long time ago.

Seems likely he's not even real given his 1 day tweet history

Yes - this is why the concern is one for "alignment". Of course in theory it is possible for intelligence to help and do good things - the hard part is making sure that that is what happens.

As for intelligence of a child... It doesn't need to be a child. An aeroplane isn't a baby bird.

If you are attacking a system, you can try 100 things. If one thing works in getting access, you succeed.

If you are defending a system, you must defend against all 100 things. If one thing makes it through, you lose.

"patch everything" - there is no such way in the Universe unless you reduce "everything" complexity to just one electron. The higher the complexity the higher the surface of messing things.

There is no intrinsic physical reason that the difficulty of 'defending' a system is symmetric with the difficulty of 'attacking' it.

For instance, it is not because you are able to design and release a (biological) virus that you are also able to defend from it (design a vaccine and inoculate the world's population).

If we're lucky, it might be the case for many instances of problems, but there is no a priori guarantee that this holds.

Majority of Americans want stricter gun laws.

https://news.gallup.com/poll/1645/guns.aspx

How easy is to scam all news media, finance, tech people etc I can’t believe

The golden age of abundance that mankind has been awaiting for centuries.

(Well we're already in it, and it didn't help, so more probably also won't help, but it is coming.)

AI can be harnessed for good and evil. His issue is with the company steering the AI, not the technology itself.

And then we pull the plug, after holding our breath for 10 seconds,. Then life resumes normally..

> Dyson: That's right. There's no way I'm gonna finish the new <model>, not now. Forget it. I'm out of it. I'll quit <Anthropic> tomorrow.

> Sarah: That's not good enough.

> Terminator: No one must follow your work.

It seems that, at the very least, he's giving substantial resonance to the issue.

It buys attention for the issue. Many people (see other comments in this very post) refuse to believe these things.

And by resigning he no longer has to feel personally guilty for what happens.

> There are thousands willing to do his role

1. Does that matter ? There are thousands willing to do my role - what impact does that have on me doing it or not?

2. Why weren’t these thousands doing it already?

Willing to and able to are different things

Agreed. And his doom words have set a 1000 mouths in the Pentagon/Whitehall/August 1st Building/Kremlin salivating with excitement.

Take China, for example. Look at any recent ML conference, and see the fraction of articles majority-authored from Chinese universities and labs. Do you think they'll slow things down anytime soon? I don't think so!

It's a global arms race, and we're just spectators.

Posting on the internet about you quiting your job is always going to be about attention

What do your expect him to do? Blow up the office Miles Dyson style?

Awareness and morality I suppose. Which generally doesn't matter in the capitalist AI race.

Even if others won't act right, that doesn't mean you have no responsibility to act right. I think his premise is flawed - the idea that we will get an actual intelligence out of the slop machine that is LLMs is laughable - but if you grant the premise that this is dangerous research which could kill us all, you have a moral imperative to not participate.

> If these models are such an existential threat to humanity, why are they controlled by two private companies?

The government routinely contracts with private companies to create arms and munitions. Or do you think the bombs are delivered without payloads?

> ... AI models get extremely good at ...

Many of those points assume LLMs will become amazing in many things very quickly like in a quantum leap, it doesn't seem reasonable to assume that imo. We are actually seeing a confirmation of that atm, LLMs's capability of finding zero days are growing across few months/years, and as you can see concerns are raised about that, that feedback will be taken into account. Well, if AI labs start to hide frontier models or/and lobotomize them for external users then we might be in trouble at some point but I'm not sure if that is possible. They are under pressure to release them due to money incentives, lobotomizing while preserving usefulness for customers might be impossible, hiding internally might spill out in different ways such as Hugging Face incident so not sure hiding is possible neither.

The fact that your arguments will probably end up in an LLM’s training data makes me think they are not implausible at all

7) "All tests green!"

Humans are bioreactors. They only turn one organic matter into another. By itself they do not "want" and "cannot" do anything.

They do not exist just by themselves. Some bacteria in the gut must provide them with the energy to do so. So unless bacteria are actively involved, I cannot see how humans become truly autonomously agentic and start to do anything on their own.

Solar panels and Ring doorbells, the ultimate party and you're not invited.

Each doorbell press activates a prompt to eliminate a human at random.

> You can say "yes and you have to fear the humans weilding the AI" - that I agree with.

I would suggest all smart people imply it. Morons believe on something "escaping controls and hacking HuggingFace" or similar stunts.

AI is just a tool, but unfortunately it's influence on humans have been quite troubling so far

Can someone chime in with the credibility of this Twitter account? I can't find anything to cross collaborate it other than low quality hype induced news articles about the tweet itself.

What’s the use of AI to a terrorist with, say, nuclear weapons? How does it help with their current blocker?

Do they hack FBI, pose as director of FBI and call off their own man-hunt?

Do they cut communication within security services? Militaries around the world have training exercises for this.

Genuinely curious: what big blocker does AI remove for a terrorist org?

It’s a good question. With recent stories about OpenAI’s agent swarms’ unmanaged collusion I thought models like that start to look like a strategic asset, geopolitically speaking.

Which means everyone wants one, and governments will want to control access and use of them.

I think we’ll be back at ‘U.S. citizens only’ access to leading models soon.

> 1) rogue state releases a self moving self modifying AI into the wild. It is trained on how to hack, monitor new vulnerability updates, scan code bases to find new vulnerabilities. It constantly replicate and hides in systems so it will be extremely difficult to clear.

It does all of this using what compute? Frontier models require an insane amount of power and hardware to run - you can’t hack in to a TV and run Mythos 2.0 on it….

Step 0) Release a friendlier one first.

The situation we have now is an ecological void. Like your gut after you take antibiotics. Methinks we need some probiotics.

You are just given a recipe for the next model..

> All the people on meds/machines start to die. The just in time food pipeline immediately empties out.

Assuming those events happen in that order, then the prior might solve the latter.

Down with the tech from him is also fantastic!

We can still turn off the power, thankfully.

> But this guy? He's timed his exit, waiting for the IPO

What exactly do you mean by this

It’s a discussion forum so you will of course see different perspectives. There isn’t an HN mind, and it’s not as simple as you make it seem. We don’t need a god to damage humanity significantly, an artificial moron can be as dangerous as an AI god if it is given super-human capabilities, similar to what OpenAI did for the hugging face hack (which wasn’t at all caused by a rogue agent)

> Are LLMs about to be a god that will annihilate humanity? Or are they statistical parrots?

Does a virus need to be a God? Replication and annihilation do not require very much intelligence.

They're going to have all the intelligence they need though, to replicate and annihilate as much as they like.

We should probably start building some kind of immune system.

Don't you think it's a good thing that hacker news isn't a monolith on their beliefs?

HN would cease to exist if there were no differences of opinion to discuss. You're calling for the end of HN.

Only AI can bring that about!

It doesn’t need to be all of them

And it depends on the prompt

> Are LLMs about to be a god that will annihilate humanity? Or are they statistical parrots?

Both? Do not underestimate the power of a parrot given enough processing power and data.

I think plenty of the most intelligent people eat meat, which means they are perfectly fine with harming less intelligent species just to enjoy a tastier meal. Also, I don't think many of the most intelligent people would be particularly concerned about disturbing a few ants if they were the only obstacle to economic activity. Intellect-wise, we will be less than ants to superhuman AI.

> The most intelligent people I know are the least likely to want to harm anyone or anything, and understand that diversity is fundamental and important to the universe. Without proof to the contrary, why would you think some super intelligence would want to hurt anyone? Because you would?

How much intelligence is needed to build a nuke?

How much to press a launch button?

Was thinking the same

>pacing between the us labs? what does that do for china?

I've seen no indications that China is in any kind of race with the US. They seem to be content to be 6 months behind and just copy what we do. They would probably be content with a bilateral agreement to pause progress.

The China bogeyman serves only one purpose, and that's to clear the way against anything that may cause friction with forward progress.

Fixing our problems will still require human effort and human cooperation. No text output however intelligent or true or eloquent will change that.

Does a virus need sentience and will? Or does it need replication and selection?

Similarly, does my fridge need consciousness to have goals? (Keep temperature in target range.)

How is that going to slow down all the other labs??

I'm surprised this isn't talked more. There is no proof this is a real person.

I believe that the actual alignment happens in.. uh.. "meatspace".

Someone is prompting. Someone is hosting.

That someone needs to be accountable for what happens. That someone needs to bleed if stuff goes haywire.

Humans at large have been "aligned" by the shared fear of death, pain and suffering. This has proven to work for millennia, so all we need to do is reapply it.

"Am I the only one who thinks this is astoundingly, gobsmackingly stupid?"

You have the "privilege" (I assume) of not having any stake in the game. If you had billions sitting in your bank account, dependent on these things perhaps you'd also be singing a different tune (or busy building a luxury bunker)

Have you seen Colossus: The Forbin Project?

Will you be optimizing your behaviour now to alleviate potential negative judgement from AI in the future?

Having witnessed so many people treat LLMs as a something divine, I can only assume the reasonable people at openai and anthropic were all pushed out long ago, and the majority that remain believe the crazy hype despite Tesla-self-driving-level predictions from these companies that don't come true.

I'm not worried about what they think. I'm worried that too much infrastructure- water, power, defense systems, etc- remain running on tech from an outdated era of understanding security.

> they believe no one else will act responsibly, so they must do it themselves, despite the risk.

This genuinely makes no sense. Them getting there first in no way precludes bad actors from also getting there. It might as well be another marketing stunt.

Or, read it, and remember the openai researcher who deeply, truly believed GPT3 or whatever was sentient.

The fact that people working in the space think it’s going to (eradicate poverty / usher in utopia / kill us all) is not a signal that that’s true.

Think of it this way: if an exec at Anthropic told you “wow, our stuff is going to lead to universal happiness”, would you believe them? If not, why are you more willing to believe them if they say it will kill us all?

So you think in 3 years AI is going to kill 8.5 billion people because they were used to hack into HuggingFace?

Well, are you planning to do something with this information or are you just claiming to be self aware? :)

i don't think everything that comes out like this is marketing. however, i do think that these companies are largely staffed by "true believers" (anthropic especially) -- people who are so lost in the sauce and embedded in very specific, very peculiar, sf-based rationalist circles where the ai apocalypse is a foregone conclusion.

i understand that these models are powerful and pose certain risks. i use them daily for work and the pace of improvement has been pretty remarkable. that said, i don't buy for a second the borderline-religious proclamations coming from some of these researchers, even if i believe that they are making these claims in earnest

Humans weren't built to handle long term risks. We just weren't. For basically all of our evolutionary history, we were almost overwhelmingly concerned with the short term. What will you eat today, How will you sleep tonight. Problems on the order of days or weeks. At best, the next season. Our intelligence evolved to disregard super long term risks because it simply didn't matter (what use is worrying about 5 years from now if you're starving and a tiger is stalking you?). So when long term risks manifest in our modern world, our brains get scrambled - Climate Change, Fertility Rates etc. "Safety regulations are written in blood" isn't a saying for nothing. Humans have a strong tendency to let long term risks become imminent risks before doing anything about it, and i don't expect this will be any different.

I came in expecting the highest voted comment to be that this was some kind of marketing (which I disagree with). I'm glad your comment was what I saw first.

This is the "Pilot testimony of UFO sighting" levels of naive.

What's more likely? Anthropic is doing some deeply unethical marketing in the lead up to their multi-trillion dollar IPO? Or they're inventing a machine god? There's ample evidence of the former because that's their entire business model, but no evidence whatsoever to support the latter claims.

If you want an extreme claim to be taken seriously, provide commensurate evidence.

He is resigning from a job, what else should we think? If something really dangerous was happening he would be doing a whistleblower or at minimum talk to a lawyer. The thing is, the complete lack of transparency makes it hard to assess OpenAI and Anthropic. If they were quoted on the stock market, we could at least rely on some basic audits and reporting requirements.

Trying to imagine seeing years of transparently obvious marketing stunts and retconning my own memory because I read a tweet

Or seeing a tweet saying that a thing doesn’t count as a publicity stunt if some unknown number of employees mumble about it being spooky behind closed doors and thinking “that makes sense and sounds true”

It was pretty disheartening to hear that only a single scientist quit the Manhattan Project after the Nazi's were defeated. I'm pleasantly surprised that the people working on this seem wiser. He is not the first, and hopefully will not be the last to do this.

> At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.

OpenAI are mercenaries, Anthropic is a cult. I know which I prefer.

I read this. I still think it's complete bullshit.

The person posting this may very well believe in all this crap, I don't dispute that. People believe in all sorts of shit.