It's not impossible, of course.

But AI is also being hyped beyond its competences* by e.g. Hegseth, to the degree that Anthropic saying "we don't want to be used like this" resulted in an illegal attempt to punish Anthropic and ban any of their products in a far more general fashion. And Anthropic is only the least bad of the AI companies: the general consensus in safety circles is Anthropic's grade is "barely acceptable" in objective terms, everyone else is much worse.

Some people treat automated systems (both machine learning and simple computer systems) as infallible oracles; they're not infallible oracles, and people do die as a result. This already happens. So far, humans in the loop have caught and prevented massive disasters (Stanislav Petrov day is coming up soon).

Even though the personal disasters keep killing people, even though the big disasters have only been prevented by humans, even though the people making the automation (e.g. Anthropic) say "it's not good enough for that", there's people who keep pushing for full automation.

* it has a lot of competences, but it also has a lot of gaps. I can see inference suppliers being legitimately valued at multiple trillions of USD even today, but there's a big gap between that and the hundred-ish trillion dollar global economy and the combined market cap this would entail for an AI that could genuinely take over all jobs even if somehow "make more AI" wasn't among those jobs.

There's a big difference between "AI is creating misinformation and intelligence failures that could compound, especially given the fact that in a pre-AI world we came to the brink of nuclear warfare" vs the AI Doomerism of "Super-intelligent AI will kill us all in some mysterious way" of the AI "safety" TESCREAL cult, often used to hype up their own product capabilities.