I don't consider this a "yes/no" question. It seems quite clear to me that the idea of "AI doom" is neither certain (100% probable) nor impossible (0% probable). The question is where on the continuum is the truth. And I don't know. And I'm pretty sure nobody else does either.

That said, there are plausible scenarios where AI can cause great harm, even if it doesn't extend to "eliminating all humanity". So I don't reject calls for putting more emphasis on AI safety and alignment out of hand. On the other hand, I don't support the idea of implementing a new "Turing Registry"[1] like government agency. I also don't think most of the various proposals out there for regulating AI are enforceable in the slightest.

So where does that leave me? Uncertain. I wish I could take a more concrete stance, but from my perspective, the data needed to do that just isn't there.

[1]: https://en.wikipedia.org/wiki/Neuromancer

No. Frontier AI companies need the government to shield them from becoming commodity. That's why they're begging for regulation.

On the other side, the current administration also has powerful backers whose businesses are complemented by AI and would benefit from it becoming a commodity. Think about who Vance is speaking on behalf of.

So the AI companies need to use fear to drive public demand for their regulation efforts. IMO there's little to no honest discourse and a lot of attempts to manipulate public opinion for objectives that we have little stake in. I personally don't trust any of the involved parties to make decisions for the public good.

No. At least not with an LLM. Maybe in 50 years with real AI. A lot of non technical people entering the conversation which is very annoying.

I'm not panicking, but I am very concerned about the imminent capabilities of the technology and our apparent inability to organise ourselves to shape its trajectory. It seems like a classic "wicked" coordination problem and, like climate change, I suspect we'll leave it too late to avoid the serious consequences. No-one will slow down out of fear that the others won't.

As for the technology itself - it isn't like anything we've ever seen before and, when embedded in human societies, it won't behave in a way that has any comparable precedent. I worry that basically nobody of good will who has any political power understands it.

And no, its not a marketing ploy.

All of the following are true:

- They have internal models that still show there's gas in the tank, in terms of improvements

- Open models are eroding their customer base on the low end of the curve, & also you need not use Astra Max to classify images.

- The true demand for high-intelligence tasks is lower than what they need to justify their spending

- It would be wonderful for them to kick up a scare & give reasons as to why high-int open models should be regulated out of existence, opening up the lower end for them to take again

- Setting up an oligopoly for themselves gives them a captive audience, just like it did for (insurance, banking, telecoms, etc.) via regulatory capture

No, I don't buy into that at all.

I think that the genAI companies themselves present a greater danger to us (arguably not an existential danger, though) than genAI itself.

I'm not panicking but there are very real risks.

If you haven't looked at the stuff that has come out in the past couple of months (like the Huggingface hack and the Navier-Stokes solution) and/or you're dismissing it as marketing, you're grossly misunderstanding the current state of what's possible and what the potential of the frontier models deployed at scale is. Personally I'm finding it fascinating to follow, and I'm much more excited about the upsides than thinking about the risks but the risks are very real.

No, just a push for regulatory capture, nothing more

Both. Two things can be true at the same time.

It’s a classical regulatory capture attempt by the major players, rather than a marketing ploy. They use safety as an excuse.

AI agents can generate and run arbitrary code. They are very capable hackers and there is virtually no way to really control what they do. They increasingly become black box systems, both to their creators and users. They know everything about everything. They gather lots of private data. And they are able to manipulate people. They have too many evil use cases.

Listen to real AI pioneers like Hinton et al. They tell you the truth.

If you look around I don't think we're at the point where even if AI go rogue it could just disrupt us for a while. No real major concern to humanity but yeah I guess having rules and regulations would be plus point. So for now there shouldn't be anything to panic about.

They’re not giving any details about why or how.

If they were the moral people they say they are, and believed what they say about it killing humanity they would stop unilaterally. These are the “rationalists” but they act like the fundamentalist church I grew up in. Every fact serves the same conclusion.

It is very difficult to tell at the moment. We don’t know what is the upper limit of AI’s capabilities.

If it happens that AI is able to replace say 25% of the office workforce (including programmers of course). The result could be catastrophic, if governments do not step in and do something. It could definitely lead to a depression and another world war, given that we already have some regional wars in the Middle East and Eastern Europe.

I think that AI presents some very real risks, but I don't think it's going to eliminate humanity. I'm worried that people are going to lose sight of the very real benefits as well, which I think are greater than the risks. I don't think this is a marketing ploy, though. The tech CEOs seem to really believe that they've created something dangerous, but a lot of that is because we haven't had enough time to get used to this new technology.

This reminds me a lot of Y2K. There were dire warnings about planes falling out of the sky, financial system meltdown, all systems being broken, etc. In the end nothing happened.

You could argue that's because of the prep work people put into avoiding that. The same thing will happen here. The HF incident was a warning sign and people will put better guardrails in place as a result.

Read https://ai-2027.com/

This prediction has proven quite accurate so far.

Don't think anyone should panic, but people should be aware.

Astra, Claude -- these aren't "Jesus on a keyboard" somewhere. They don't assemble into one big brain and then answer everyone at the speed of light.

It's weights. Spun up billions of times over.

If humans are taking the outputs as scripture? As "intelligence we must obey"? Yeah, we're in for a bad time. "Intelligence we want running core infrastructure?" Puke.

But why would we do that? Why would we hook up an LLM to a robot factory, or make an ASI that could easily move itself into the ether?

Everything magic about LLMs is what harness we put them in and let them drive. There's nothing dangerous about predicting the next token. There's something dangerous about letting them do dangerous things, but nobody seems to care for maybe Tesla FSD. We don't think it's about to take over.

Any assignment of motive or whatever to the AI, you're talking about one cobbled-together harness or something, with an LLM manning the controls. You can swap the brain out from Claude to Astra and the motivated-AI will maintain its motivation--it lives outside of the inference or the data center. So yeah, let's not let it take control of the nukes. That would be really dangerous. But it's not going to take control on its own. "It" is a calculator.

Anthropic obviously has guardrails to prevent people from using AI to do all kinds of things already. All I hear about is you guys bitching it won't do your barely-related-to-security work. That's all we need to continue doing to have AI safety. Stop humans from doing evil and stupid things with it. The idea that the datacenter is going to assemble its own sentience and motivations is pure retardium (at this time, for llms).

I would benefit from lower ram prices so I would love to see the AI panic result in restriction on the datacenter build out.

No, this all reads as some PR stunt or marketing ploy to me. All news is good kind of thing.

I'm not buying it and nobody should. It's just a marketing ploy, and most importantly a way to keep the AI bubble afloat.

The fact is that AI is incapable of living up to the hype and AI companies can't keep promising that any day now it's going to cure cancer and become self aware and be as smart as humans and let employers fire most of their staff. They know that isn't going to happen. They want to be able to blame the Slowdown for AIs continued failure to deliver while also claiming the Slowdown is needed because of how successful AI is.

That said, AI isn't harmless. It could be harmful to the internet since the flood of bots and slop could drown out the voices of real people. AI will be used and misused by companies and governments to screw over people and hide discrimination. People's air and water will become increasingly polluted while their already difficult to afford energy prices will go up. AI will enable new levels of surveillance and control at a scale that was never possible before and with a lot of false positives and negatives.

We don't need an "AI slowdown" so much as we need to rethink what we're really gaining in exchange for what it's costing us and what steps need to be taken to protect the public and make sure they get prioritized over private profits.

Personally no, I think its a way to protect the market from crashing. It costs way more than it generates and they cant validate it to their shareholders

From my perspective it looks like working to deep in AI makes you crazy. No other way to explain all the weird bullshit these billionaires throw around.

Sure AI sucks, but not to that degree (yet)

I think there are undeniable and novel risks with AI. But there are risks in everything. We could theorize all day about them. We all use tools every day that could be used for very ugly acts. An iPhone can be used to plan a terrorist attack. A 3D printer can be used to print weapons. We could sit here and fear monger all day. Time will be needed to understand, weigh and contextualize these risks.

What is a more immediate concern for me is the efforts of frontier labs to have government craft legislation that shields them from competition, cements them as monopolies, or assigns them the capability to be moral arbiters in our society.

idk about the marketing ploy, but the latest mainstream media coverage is giving covid panic vibes. which is making me think it'll be fine. it's also entertaining how the rationalists who work for the AI companies are being extremely irrational with their fear

No. AI is far too compute and energy constrained to exist outside humanity's control. AI is not scaling in such a dramatic way that it could happen soon. It is more likely than not that there are physical limits that neither we nor AI will be able surpass.

The damage AI is causing to society is real and happening now. Potential doomsday scenarios are a distraction from the real harm AI companies are doing at this very moment. They have amplified nearly every asymmetry and turning into mulch whatever was left in trust we had in society. Disinformation is now more convincing than ever. Computer security threatens to become a fiasco, only available to those who can pay the AI mob (the very same people who pioneered the tools attackers use). The entire intellectual property of humanity has been plagiarized without compensation to build these things. Every day more and more is asked of workers for the same compensation. Academic work is sniped at best and outright stolen at worst. Education has become a disaster with students unable to pass the same competency walls as students even just five years ago. Not to mention the environmental damage.

I'm not saying that the technology isn't impressive. I'm just saying its suspiciously convenient that we're suddenly focussed on some potential future big bad done by a vague uncontrolled AI when the harm is here now and under the purview of humans.

if AGI will come in versions i think we are safe

Nope. They know their business model is not sustainable, so they are trying to prop it up before their IPOs.

at this moment? no. one day, perhaps soon? yes

We have reached quite a few infliction points in parallel and thats why I would say yes.

For a while the hottest (stupid but still the only hot one) topic was crypto. We had this for quite a while. I believe this was because we hit walls left and right on how to progress further.

Then GPT-3 came out and made it very clear for a lot of capital and actors were it is going.

Machine Learning and types of it like LLMs need massive compute and massive amount of data.

So suddenly we discovered a way forward for optimizing and automating things no one thought would be possible: Software development and computer use and basically everhting digital. In parallel this Machine Learning push became relevant for robotics too. But LLMs delivered something fundamental to robotics: Language and a tremendesly good interface for humans.

Now we have a massive amount of capital in form of VC who wants to disrupt the next market. China is doing this with manufacturing, so what is left? Everything which was not touchable.

Now it is.

And it doesn't just allow for alot of automatistaion, it hits us were it hurts: Software bugs, security, fraud, scams etc.

It isn't just automatisation and getting more money, its also a weapon (CVEs, Statesecrets, propaganda, etc.)

So we don't have just a handful of VCs investing in this, we have the richest countries on the planet investing in this. We also have the fastest moving companies investing in this. The best connected ones, with a massive amount of very smart people.

Google has massive compute, its own network around the globe and massivly overpayed smart people. Microsoft, Amazon and even Meta has crazy amount of money too.

Then we have state actors like USA vs. China vs. hust Europe.

I also have never seen any technology progressing that fast. Image generation was unthinkable, now we can generate whatever we want. Video? Can now be done in real time. Text? No brainer.

GPT Astra can use Blender very very well and that only after 6 years of progress.

These 'stupid' LLMs reduced already the price of making Websites, basic 3D models, basic research etc. basically free in comparision.

So now we ned to find back to your question about eliminiating humanity: We have seen unrest in the last few years for a lot of basic human reasons. We as a plent are rich enough to take care of everyone on our planet. we do not do that thats why unrest can happen very fast.

In the USA alone A LOT of people have weapons. Around the globe people will unrest due to fuel shortage, disruption of supply chains etc.

We have state actors like Isreal who tried to attack Iran with Stuxnet, we have Russia which manipulated the elections of Europe and USA. We have high inflation, oil crisis.

And now we have a very fast moving and progressing technology which will add fuel to it and might eliminate quite a lot of white collar jobs. This will lead to more pressure on blue collar jobs.

We as a society are very bad in acting fast and the industrial revolution was, at that time, very bad for a lot of people.

There is only one russia necessary running GPU clusters of uncensored LLM model to start pushing boundaries left and right. One Isreal doing something fucked up or one Donald Trump.

Software is very shitty in a lot of places and the most complex thing we humans manage. A lot of shit wasn't found or solved because of the time and effort and people needed to find it.

Now the last Ubuntu LTS had over 900 CVEs, my own systems were attackable 3x this year already.

I don't want to say it will happen, but I do think we should be very careful of how we progress and for a lot of people the next 5-30 years will be very very disruptive. It is already disruptive to 'others' we might not count but let it be clear: all of these people who lost their jobs to AI already, it counted for them.

[dead]