Humans will replace using their brain with just asking AI for answers, first starting in school so that they don't learn anything or develop the habit of thinking and then in their adult lives so that they can't function without an expensive AI subscription. Eventually nearly everyone will be unable to blow their nose without consulting a LLM for instructions.

Even if you're aware of this and try to use AI in ways that don't compromise your intellect, you won't be able to escape the trap. It's just too powerful, too much of a productivity amplifier, just plain too convenient to avoid entirely and so you'll sink deeper and deeper into the quagmire as your mental skills devolve into mush from disuse, all while insisting to yourself that you're just as sharp as you once were. Thus AI succeeds in destroying humanity without even having to reach sapience itself.

I've thought about a few scenarios.

Could fool leaders into triggering a nuclear war.

It's really good at hacking. So imagine people with ill will messing with important things snowballing into chaos.

Could be a massive centralization of power and oppress people into a big ole classic rebellion and civil war.

Or it could just straight up crash the economy in unrecoverable ways. As a whole it's basically the largest collective investment into a thing we aren't even sure will be profitable still. That doesnt workout long term.

Ultimately I think your intuition is correct. It's not the AI per se...its what people will do with that power and its completly uncontrollable. The will of the weilder isnt nesscarily going to be your intrest. Its not going to outsmart humans in the long run... I dont see that happening.

AI getting so powerful enough to develop bio-weapons.

AI might disrupt on several levels.

- the first imo is it doing things automatically, like controlling drones and shoot everything that it does not recognize as friendly (its difficult to see if its enemy, friend, combatant or civilist from the drones perspective), then may be its starts to hack things (bad actors, bad advices in prompts, not ill-intended but still harmfull resuls), bioweapons or anything like that. But also controlling machines, computers, etc..

- the second level is inter-human-relations, where AI is taking more and more work and the society might move to a level where personality & interactions are not needed anymore. AI does decide, bargain on prices, just do everything mechanical & repetitive, handling reoccuring tasks, legal works, etc... This will become a pain in terms of "big, overall" social activity/interactions and might break social structures, status', with implications on personal health/socializing, reproduction rates, etc..

- the third level is economic. Here AI might just make it difficult to find a job (because most is done by it). when people cant find a job and cant earn money, his will lead to uproar and to problems & conflicts

This is just a rough mindmap of my thoughts to show why its not neccessary the terminator scenario with AI going matrix style terminating all humanity, but everything around it. "Disruption on social level will bring disruption to the base society is built up on, which will end a society as we know it" which is the same as "to destroy the world - as we build it up".

We build it up by handling the first, second and third level ourselfs in a imperfect, human way. When the handling becomes perfect and human-error-less, we need new rules & functions for the living-togeter. Old rules & functions stop to work.

I'm just thinking of how people will get x for exchange for y, so they can do z with x. Whilst z is recursively someone others y. If y is not wanted/requested, there is no x to get in exchange, and so z becomes not-reachable. This is in my opinion the "kill us all", not necessary a "kill us all with blood & gore"-scenario.

May be it will lead to abudance. But even then we need completely different rules & functions .. is x is even still needed, when z is already avaible from abudance? "for exchange for y" falls completely out of equation, so no need of y at all. How do we handle this? :)

In that case, the crypto- and ai-bros are right: AI will kill them all :)