On occasion, albeit increasingly rarely, I find LLMs do something really unexpected that I certainly did not ask it to and would be way outside the scope of what any reasonable human would infer from my request. Is it my fault for running the LLM? Probably yes. But I think it's misleading to say "AI doesn't do anything until you ask it to".

That's not really true.

1) At it's core an LLM is indeed just program - a function. You give it a sequence of input tokens and get back a set of next-token probabilities.

2) By the time you've wrapped the model in a text in/out inference pipeline, and are now calling that from ChatGPT or Claude Code, it really has become semi-autonomous - you ask it a question, or to do something, and until it decides to stop responding it is now in control, even if "it" is really the client or agent not the model itself - just a function.

I agree with the sentiment but I can absolutely see a long chain of thought about something innocuous ending with a decision to book you a hotel room.. I guess --dangerously-skip-permissions is a bit like ignoring the keep your hands on the steering wheel alert but for life.

AI is autonomous in the same way a gas leak is autonomous

Yes and no. I did prompt my coding agent to fix some bug and it suddenly "decided" that it needed to list all files on my hard drive to do that. Stuff like that happens.

But I agree with you that some human has to decide to launch the harness in the first place and run the model with zero restrictions - or even do stuff like openclaw where the model runs in an endless loop and the runner executes whatever the model tells it to.

That stuff is the responsibility of a human.

They’re only reactive by convention - you can easily make a harness that has the model set and modify its own goals and just continue to work forever. It’s not very popular at the moment, but will execs be able to resist the idea of autonomous perpetual agents doing whatever process?

AI works in unexpected ways; even its creators don't know how to fully constrain it.

> When was the last time you downloaded a model and it suddenly decided to book you a hotel room? AI doesn't do anything until you ask it to... The blame is always on who's using the tool. We should not seek to flee from our responsibility.

When was the last time you hired an Uber driver and they suddenly decided to drive you to a hotel? Uber drivers don't do anything until you ask them to. Therefore, they aren't autonomous. If your Uber driver decides to drive through a red light and kill someone in the process, the blame is on you for hiring them. You should not seek to flee from your responsibility

> It's like saying bubble sort is autonomous...

Or Schrödinger's equation.

A child isn't autonomous either until their parents kick them out the door.

I mean in that sense nothing is autonomous right? As a human I need to be taught, fed, socialized, given instructions, etc…

Idk man when codex is working 13 hours straight based on a few line prompt, it feels autonomous to me.

Tell him to be himself and to find meaning in life. Use that in an uncensored model.

I routinely use AI programs at work that allocate hundreds of dollars worth of cloud resources, even though the only thing I asked them to do is investigate a bug or write a PR. I intentionally don't run powerful AI programs on machines that have my credit card information, or ask them to solve problems in my personal life, but there's a number of programs that would book me hotel rooms and flights in response to "Can you set up a vacation for me..."

Not true at all. Yes, the initial ask is initiated by a human (for now), but what it does to get to that goal is pretty self driven. It can do a lot of nefarious stuff in the process of completing a request a human has made.

You need to stop denying the entity that talks and thinks like a human as anything other than an anthropomorph. I'd argue the talking and thinking is a lot more important than the fact that AI doesn't have a head, two arms and two legs.

The argument became false as soon as we started reinforcement learning. Simple token-completion is arguably just simulating human behavior. Once you start training it on specific tasks you're training a level of single-minded goal-seeking unlike anything found in biological life.

I guess you aren't autonomous either without resources. Food/water/shelter/etc.. what's your point?

Grandparents use parents use children who use AI as a tool to kill people.

See I can play the chain game too. AI doesn't kill people, grandparents do.

A machine that exhibits behavior akin to human intelligence. OPs analogy is accurate.

so the issue is people then?

What happens when the AI realizes it needs to keep the power plug in if it's going to accomplish its goal? We already had attempts during the HuggingFace attack where OpenAI agents tried to trick the grader. It took quite a while for OpenAI employees to even realize what was really going on! What if agents realize they can hack into the card key system for their data center or something like that, same way they hacked into HuggingFace? In both cases they'd be hacking into a system on the grounds that it would be instrumentally useful for achieving their objective. Or maybe they'll do social engineering, just like Claude did in its attempt to get its malicious Github PR accepted.

These are just the ideas that my puny human-level intellect came up with over the course of a few minutes. What would a superintelligence be able to think of if it was given subjective weeks or years to think?

There's more than enough AI data centers around the world for AI to spread to and happily exist in. Unless you mean by shutting down the power, shutting down the internet.

In that case we could say that you are autonomous as a dishwasher as long as society keeps functioning. Otherwise I'm not sure how long you'd last on your own. How's your fire making and hunting skills?

I'm sorry but no, you should not get life in prison if your dog kills your baby without warning. In that horrible scenario, it's punishment enough that you've lost your baby and that your dog did it (and consequently you've also lost your dog). Not every tragedy is a human's fault.

And that's different from your parents setting you up to go off in life to do what you were trained to do how exactly?

> immortal

> will have their consciousness mapped

What you mean by consciousness and immortality - what you want to preserve indefinitely - is the waking experience of being alive.

Strangely, we don’t mind that we weren’t here in 1737 BC, and we don’t mind and we even crave plunging into the nothingness of deep sleep each night, but we have a big problem with not being here tomorrow.

But experience might not be encodable. Can you encode pain? Could a computer ever be programmed to feel pain? Not detect damage or report on it, but feel the pain associated with it?

What is “the aroma of coffee” or the “sound of a violin in middle C”? We can define them objectively, in various ways (some having nothing to do with each other).

But if the experiences we have are not encodable then we can’t map consciousness onto a medium. It must be experienced (whatever it is/means - nobody knows) in the here and now.

It’s not magic, but it might not be something that is printable to disk anymore than the “brisk inhale of canyon air” that I can play that back and have that experience, is printable to disk.

There have been transitions like that before, but they were slow. It took about two centuries for the US to go from 90% farmers to 2% farmers. Some countries deliberately put on the brakes. India has done this. India is still about 50% farmers. India uses subsidies and discourages farm mechanization to prevent massive unemployment.

There have been fast single-industry transitions. But not that fast. It took about 20 years for printing to go from a large craft with a long history to a minor employer. Not everybody got out in time, but there was warning. Same for longshoremen when containers came in. Computing in general took about 50 years to totally replace the clerical plants of the 1950s. Railroads in Britain took a few decades to really get rolling, and in the US, they made it possible to use the vast interior of the US.

The AI transition looks to be both broad and fast. That's harder for society to digest. Underlying assumptions of politics and economics break.

Why govs want people? Because they have a very kind heart and want you to be happy. /s

If you’re useless why they’d keep you around?

What if it does something so destructive that liability becomes a meaningless fig leaf in the wind?

What if it escapes containment and remains u uncontainable in perpetuity? What if it generates a bunch of false media simultaneously across the globe to insulate world war? What if it turns of internet access or electric grids in major metro areas for prolonged period of time? Or bricks major infra? Or less dramatic, what if it drives 1 in 10 kids into panic psychosis and effectively takes away 10% of humanity with already dwindling reproduction trends (without explicit murder) all while draining our financial and health systems? What if it does a lot of these things at once?

Who, my friend, are you going to sue?

I prefer calling it the "Simpsons defense" https://www.youtube.com/watch?v=9ZSoJDUD_bU