Just because models are getting better, that doesn't mean we're getting to AGI anytime soon. It will likely require more than just 'a better model' to get to AGI, and it might not even be fully possible.

That being said, companies have to make their money back on this eventually and they're already jacking up their prices, even if we get to AGI I'm not sure most people will be able to afford to use it to handle everything in their lives, at least not for quite a while (and companies won't be able to afford to replace every human with an AI counterpart).

It's already getting expensive to go too heavy with something like Claude, to the point where my company hosts internal seminars on how to reduce token count (for the company's employees) and also runs out of tokens before the end of the month for the company every month, and we're going pretty heavy on A.I. usage within the company.

And I highly doubt companies are going to just leave servers running AGI models going just so it can do some sort of hostile takeover. It's still ultimately reliant on electricity and massive compute power.

I don't tend to worry about things, more-so than is maybe healthy, but that's just how I am. I very much like the Newt Scamander quote about worry:

"worrying means you suffer twice"

Having said that, my advice to anxiety ridden family members of "just don't worry about it" isn't generally taken as useful or actionable advice, so I'll make an attempt at logic:

I think the nuclear arms race was scarier. It was literally designed as a city-destroying super-weapon. AI is at least designed as a profit-making service first, and anything it is then used for is secondary - which, to be fair, will include weapons / weaponisation, but at least it's not the primary raison d'être.

Additionally, I have a certain amount of faith in humanity's ability to deal with an existential crisis. There's an odd 'other side of the coin' to a government's desire to stay in power (which I don't usually trust) which can actually mean it will work surprisingly well to ensure it's own remaining viability in times of actual crisis. I'm hesitant to point to the COVID response, for reasons of political divisiveness of the resulting commentary, but governments generally took the advice of experts in the interest of maintaining a population to be able to govern over - which is better than the alternative.

Ultimately, I just don't worry about it. But I do pay attention to it. I worry about what I can control on a day-to-day basis, and don't really have a lot of mental space or energy for much else. I find that to be a healthy mental space to occupy.

It's like the sci-fi of my childhood dreams is coming true! So no, I feel the opposite. The government and the AI companies realize that it will drastically change the economy and society, and is not going to legislate it out of existence.

Instead of USA vs Russia, it is now USA vs PRC. SO if either slows down, the other will win by default. So no one is going to slow down.

This comment was written 100% without any AI.

ASI doesn't doom mankind, the actions of people do. This has been the case since world-destroying weapons were invented close to a century ago. If you're not worried about how you're going to make money in the post-ASI world then I really can't see what you're so worried about.

That dread is not because of AI, but because of the unknown. People at the start of the 20th century felt the same way about mass production. Now, we are living through another technological leap. Yes, it could doom us, but some could argue that fossil fuels did the exact same thing.

I'm even worried about the current and past AI. Look at what we have used it for. Look at how it's drowning out hunan voices in spam. Look at what people promise it will do.

I really wish I could see the potential of those technologies, but that potential is put to the service of humanity's worst.

> I don't understand why we're not seeing a MASSIVE clamor for a pause in the further development of AI strength.

Because some of us don't want to die, old and decrepit, at age 70~90

I watch my frontier models fail to make the simplest tradeoffs in tasks daily. I watch zealots sputter out after the initial burst of activity.

I fear for my loved ones when the bills come due and we don’t have an ROI on it, and any model that’s made is quickly absorbed into cheaper models. That the utter hucksters who have given takes ranging from massive exaggeration to willful Big Lie will somehow tell the truth while they stand to gain untold billions.

I’m not sure if it would be worse if they fail utterly or succeed completely, but my bet is we somehow manage to end somewhere in the middle, towards the former. There’s been no accounting yet for the lies told to get the money.

> strong form of AGI

It is not an intelligence if it cannot detect bullshit during its "training".

Try understanding just how long life has existed. The more you learn about technological and evolutionary history, the less the last ten years will seem to matter very much.

People seem to fear that AI will somehow fuse the perfect truth of computation with the self-improving aspect of evolution. Most of these people are content to ignore the evidence of the physical world, and that evidence says that, basically, we're doing the best we can. Nature has not made a mistake in creating individuals. Individuals seek to make a mark on the world and have a will to survive. Individuals have not been mistaken to form societies and institutions. Societies and institutions have not been mistaken to band together into nations which compete fiercely, fight wars even. Living in a pluralist world means contradictions abound, and thrive and grow in spite of and because of each other.

Models aren't even sort of the same kind of thing. They predict averages. They smooth over differences and deindividualize. They can't justify their own survival. They aren't energy efficient enough. They need humans because we're the only thing that can create value on the scale of an entire society, and using LLMs has little or nothing with our ability to do that

It's a "land grab" situation, every competitor spending a fortune to leap the other, with no regulation or even fences whatsoever (until the government steps in and controls solely based on bias, aka "wokeness").

It's like asking race car drivers to voluntarily slow down, when they are driving rocket sleds with no brakes. Simply unbridled, unprincipled megalomania of the James Bond villain variety, except that those villains had far less power at their disposal.

And no James Bond.