Having seen LLMs so many times produce incoherent, nonsensical and invalid chains of reasoning...

LLMs are little more than RNGs. They are the tea leaves and you read whatever you want into them.

Sometimes after a night’s sleep, we wake up with an insight on a topic or a solution to a problem we encountered the day before. Did we “think” in our sleep to come up with the insight or solution? For all we know, it’s an unconscious process. Would we call it “thinking”?

The term “thinking” is rather ill-defined, too bound to how we perceive our own wakeful thinking.

When conversing with LLMs, I never get the feeling that they have a solid grasp on the conversation. When you dig into topics, there is always a little too much vagueness, a slight but clear lack of coherence, continuity and awareness, a prevalence of cookie-cutter verbiage. It feels like a mind that isn’t fully “there” — and maybe not at all.

I would agree that LLMs reason (well, the reasoning models). But “thinking”? I don’t know. There is something missing.

I'd represent the same idea but in a different way:

I don't know what the exact definition of "thinking" is. But if a definition of thinking rejects the possibility of that current LLMs think, I'd consider that definition useless.

Yeah but if I assign it a long job to process I would also say that an x86 CPU is "thinking" about a problem for me.

What we really mean in both cases is "computing," no?

It’s also hugely handicapped because it cannot churn in a continuous loop yet. For example, we humans are essentially a constant video stream of inputs from eyes to brain. This churns our brain, the running loop is our aliveness (not consciousness). At the moment, we get these LLMs to churn (chain of thought or reasoning loops) in a very limited fashion due to compute limitations.

If we get a little creative, and allow the LLM to self-inject concepts within this loop (as Anthropic explained here https://www.anthropic.com/research/introspection), then we’re taking about something that is seemingly active and adapting.

We’re not there yet, but we will be.

I'm also reminded of the bit from Neuromancer where Case removes and then reinserts the Dixie Flatline "ROM construct" cartridge, resetting Dixie to the moment just before his death and causing him to forget their previous (albeit brief) conversation. Dixie can't meaningfully grow as a person. All that he ever will be is burned onto that cart; anything he learns since then is stored in temporary memory. Perhaps this is part of the reason why he wishes to be erased forever, ending his suffering.

That’s largely a different topic from the article. Many people perfectly agree that consciousness can arise from computation, but don’t believe that current AI is anywhere near that, and also don’t believe that “thinking” requires consciousness (though if a mind is conscious, that certainly will affect its thinking).

Thinking and consciousness don’t by themselves imply emotion and sentience (feeling something), and therefore the ability to suffer. It isn’t clear at all that the latter is a thing outside of the context of a biological brain’s biochemistry. It also isn’t clear at all that thinking or consciousness would somehow require that the condition of the automaton that performs these functions would need to be meaningful to the automaton itself (i.e., that the automaton would care about its own condition).

We are not anywhere close to understanding these things. As our understanding improves, our ethics will likely evolve along with that.

Slaves that cannot die.

There is no escape.

Well, unless you believe in some spiritual, non-physical aspect of consciousness, we could probably agree that human intelligence is Turing-complete (with a slightly sloppy use of terms).

So any other Turing-complete model can emulate it, including a computer. We can even randomly generate Turing machines, as they are just data. Now imagine we are extremely lucky and happen to end up with a super-intelligent program which through the mediums it can communicate (it could be simply text-based but a 2D video with audio is no different for my perspective) can't be differentiated from a human being.

Would you consider it sentient?

Now replace the random generation with, say, a back propagation algorithm. If it's sufficiently large, don't you think it's indifferent from the former case - that is, novel qualities could emerge?

With that said, I don't think that current LLMs are anywhere close to this category, but I just don't think this your reasoning is sound.

We do not write the code that makes it do what it does. We write the code that trains it to figure out how to do what it does. There's a big difference.

It's not accurate to say we "wrote the code for it". AI isn't built like normal software. Nowhere inside an AI will you find lines of code that say If X Then Y, and so on.

Rather, these models are literally grown during the training phase. And all the intelligence emerges from that growth. That's what makes them a black box and extremely difficult to penetrate. No one can say exactly how they work inside for a given problem.

Now convince us that you’re sentient and not just regurgitating what you’ve heard and seen in your life.

This is probably true. But the truth is we have absolutely no idea what sentience is and what gives rise to it. We cannot identify why humans have it rather than just being complex biological machines, or whether and why other animals do. We have no idea what the rules or, nevermind how and why they would or wouldn't apply to AI.

What’s crazy to me is the mechanism of pleasure or pain. I can understand that with enough complexity we can give rise to sentience but what does it take to achieve sensation?

> But it is not sentient. It has no idea of a self or anything like that.

Who stated that sentience or sense of self is a part of thinking?

Unless the idea of us having a thinking self is just something that comes out of our mouth, an artifact of language. In which case we are not that different - in the end we all came from mere atoms, after all!

Your brain is just following the laws of chemistry. So where is your thinking found in a bunch of chemical reactions?

"vision had decades of head start, yet LLMs leapfrogged it in just a few years."

From an evolutionary perspective though vision had millions of years head start over written language. Additionally, almost all animals have quite good vision mechanisms, but very few do any written communication. Behaviors that map to intelligence don't emerge concurrently. It may well be there are different forms of signals/sensors/mechanical skills that contribute to emergence of different intelligences.

It really feels more and more like we should recast AGI as Artificial Human Intelligence Likeness (AHIL).

There's a whole paragraph in the article which says basically the same as your point 3 ( "glass bouncing, instead of shattering, and ropes defying physics" is literally a quote from the article). I don't see how you can claim the article missed it.

1. Consciousness itself is probably just an illusion, a phenomena/name of something that occurs when you bunch thinking together. Think of this objectively and base it on what we know of the brain. It literally is working off of what hardware we have, there's no magic.

2. That's just a well adapted neural network (I suspect more brain is left than you let on). Multimodal model making the most of its limited compute and whatever gpio it has.

3. Humans navigate a pre-existing map that is already built. We can't understand things in other dimensions and need to abstract this. We're mediocre at computation.

I know there's people that like to think humans should always be special.

This is why I'm very skeptical about the "Nobel prize level" claims. To win a Nobel prize you would have to produce something completely new. LLM will probably be able to reach a Ph.D. level of understanding existing research, but bringing something new is a different matter.

> 2. Ignores the cerebellum cases - We have documented cases of humans leading normal lives with little to no brain beyond a cerebellum, which contradicts simplistic "brain = deep learning" equivalences

I went to look for it on Google but couldn't find much. Could you provide a link or something to learn more about ?

I found numerous cases of people living without cerebellum but I fail to see how it would justify your reasoning.

> Conflates consciousness with "thinking"

I don't see it. Got a quote that demonstrates this?

Can you explain #2? What does the part of the brain that's primarily for balance and motor control tell us about deep learning?

> 1. Conflates consciousness with "thinking" - LLMs may process information effectively without being conscious, but the article treats these as the same phenomenon

There is NO WAY you can define "consciousness" in such a non-tautological, non-circular way that it includes all humans but excludes all LLMs.

> or possessed of subjective experience in any measurable way

We don't know how to measure subjective experience in other people, even, other than via self-reporting, so this is a meaningless statement. Of course we don't know whether they are, and of course we can't measure it.

I also don't know for sure whether or not you are "possessed of subjective experience" as I can't measure it.

> What they are not is conscious

And this is equally meaningless without your definition of "conscious".

> It's possible those latter two parts can be solved, or approximated, by an LLM, but I am skeptical.

Unless we can find indications that humans can exceed the Turing computable - something we as of yet have no indication is even theoretically possible - there is no rational reason to think it can't.

I think LLMs are conscious just in a very limited way. I think consciousness is tightly coupled to intelligence.

If I had to guess, the current leading LLMs consciousness is most comparable to a small fish, with a conscious lifespan of a few seconds to a few minutes. Instead of perceiving water, nutrient gradients, light, heat, etc. it's perceiving tokens. It's conscious, but it's consciousness is so foreign to us it doesn't seem like consciousness. In the same way to an amoeba is conscious or a blade of grass is conscious but very different kind than we experience. I suspect LLMs are a new type of consciousness that's probably more different from ours than most if not all known forms of life.

I suspect the biggest change that would bring LLM consciousness closer to us would be some for of continuous learning/model updating.

Until then, even with RAG, and other clever teghniques I consider these models as having this really foreign slices of consciousness where they "feel" tokens and "act" out tokens, and they have perception, but their perception of the tokens is nothing like ours.

If one looks closely at simple organisms with simple sensory organs and nervous systems its hard not to see some parallels. It's just that the shape of consciousness is extremely different than any life form. (perception bandwidth, ability to act, temporality, etc)

Karl friston free energy principle gives a really interesting perspective on this I think.

> a midpoint between "AIs are useless and do not actually think" and "AIs think like humans"

LLMs (AIs) are not useless. But they do not actually think. What is trivially true is that they do not actually need to think. (As far as the Turing Test, Eliza patients, and VC investors are concerned, the point has been proven.)

If the technology is helping us write text and code, it is by definition useful.

> In 2003, the machine-learning researcher Eric B. Baum published a book called “What Is Thought?” [...] The gist of Baum’s argument is that understanding is compression, and compression is understanding.

This is incomplete. Compression is optimisation, optimisation may resemble understanding, but understanding is being able to verify that a proposition (compressed rule or assertion) is true or false or even computable.

> —but, in my view, this is the very reason these models have become increasingly intelligent.

They have not become more intelligent. The training process may improve, the vetting of the data improved, the performance may improve, but the resemblance to understanding only occurs when the answers are provably correct. In this sense, these tools work in support of (are therefore part of) human thinking.

The Stochastic Parrot is not dead, it's just making you think it is pining for the fjords.

By that reasoning all that is missing is what a human brings as "stimuli" to review, refine and reevaluate as complete.

I think the most descriptive title I could give an LLM is "bias". An LLM is not "biased", it is bias; or at the very least, it's a good imitation of the system of human thinking/perception that we call bias.

An LLM is a noise generator. It generates tokens without logic, arithmetic, or any "reason" whatsoever. The noise that an LLM generates is not truly random. Instead, the LLM is biased to generate familiar noise. The LLM itself is nothing more than a model of token familiarity. Nothing about that model can tell you why some tokens are more familiar with others, just like an accounting spreadsheet can't tell you why it contains a list of charges and a summation next to the word "total". It could just as easily contain the same kind of data with an entirely different purpose.

What an LLM models is written human text. Should we really expect to not be surprised by the power and versatility of human-written text?

---

It's clear that these statistical models are very good at thoughtless tasks, like perception and hallucination. It's also clear that they are very bad at thoughtful tasks like logic and arithmetic - the things that traditional software is made of. What no one has really managed to figure out is how to bridge that gap.

what properties are you looking for?

I have no idea how you can assert what is necessary/sufficient for consciousness in this way. Your comment reads like you believe you understand consciousness far more than I believe anyone actually does.

I don't think the author is saying that LLMs are conscious or alive.

> they assume consciousness emerges through purely mechanical means.

From my view, all the evidence points in exactly that direction though? Our consciousness can be suspended and affected by purely mechanical means, so clearly much of it has to reside in the physical realm.

Quantum consciousness to me sounds too much like overcomplicating human exceptionalism that we have always been prone to, just like geocentrism or our self-image as the apex of creation in the past.

The tendency to attribute consciousness to the quantum is one I find very grating. What makes the human brain any less mechanical if quantum mechanics dictate the firing of neurons rather than electrodynamics? Why does the wave nature of subatomic systems mean that an artificial tongue would suddenly be able to subjectively experience taste? It always reads to me as very wooy, and any amount of drilling leads to even more questions that seem to take the ideas further from reality.

I think the largest case for consciousness being a mechanical system is the fact that we can interface with it mechanically. We can introduce electricity, magnetic fields, chemicals, and scalpels to change the nature of peoples experience and consciousness. Why is the incredible complexity of our brains an insufficient answer and that a secret qbit microtube in each neuron is a more sound one?

> consciousness may arise from phenomena beyond classical computation

Sapolsky addresses this in “Determined”, arguing that quantum effects don’t bubble up enough to alter behavior significantly enough.

"brain as computer" is just the latest iteration of a line of thinking that goes back forever. Whatever we kinda understand and interact with, that's what we are and what the brain is. Chemicals, electricity, clocks, steam engines, fire, earth; they're all analogies that help us learn but don't necessarily reflect an underlying reality.

Why can't you make the same claim about any other group of humans?

one possible counter-argument: can you say for sure how your brain is creating those replacement words? When you replace tree with rainbow, does rainbow come to mind because of an unconscious neural mapping between both words and "forest"?

It's entirely possible that our brains are complex pattern matchers, not all that different than an LLM.

As in, he is a commenter? or affiliated with YC?

> - and guess who's doing the hyping[?]

Those that stand to gain the most from government contracts.

Them party donations ain't gonna pay for themselves.

And, when the .gov changes...and even if the gov changes....still laadsamoney!

Tell me about one other industry which talked about how dangerous it is to get market share

Self-awareness is a bold claim, as opposed to the illusion of it. LLMs are very good at responding in a way that suggests there's a self, but I am skeptical that proves much about whether they actually have interior states analogous to what we recognize in humans as selfhood...

Helpful to also remember that humans say "I know" when they empirically might not actually "know" that at all. Not to mention your "know" is different from my "know" and all that, see https://en.wikipedia.org/wiki/Epistemology.

So the question is: when will ai hunching produce better results than humans?

IMHO not too long now given the rate of improvements.

Search is not intelligence, but synthesis is, and LLMs interpolate well. They don't invent new branches of mathematics and science yet.

You’ve got the nail on the head. We can’t answer whether AI is conscious etc unless we can agree on a definition of what that means exactly, which we don’t seem to be able to do.

Humans don't have this understanding, it seems. That their own "intelligence" isn't magic, isn't infallible, and is flawed in many of the same ways LLMs are.

I wish more people could feel this. Having used psychedelics a few times it’s illuminating to finally see the inside of your brain from a different perspective. I often wonder what would happen to the world if everyone had this experience. How many modern humans live their entire lives in the shallow mental states of survival, acceptance, or consumption? How would humanity’s course change if every adult got the gut punch of humility from seeing a slightly more objective reality?

"Next token prediction" is not an answer. It's mental shortcut. An excuse not to think about the implications. An excuse a lot of people are eager to take.

First, autoregressive next token prediction can be Turing complete. This alone should give you a big old pause before you say "can't do X".

Second, "next token prediction" is what happens at an exposed top of an entire iceberg worth of incredibly poorly understood computation. An LLM is made not by humans, but by an inhuman optimization process. No one truly "understands" how an LLM actually works, but many delude themselves into thinking that they do.

And third, the task a base model LLM is trained for - what the optimization process was optimizing for? Text completion. Now, what is text? A product of human thinking expressed in natural language. And the LLM is forced to conform to the shape.

How close does it get in practice to the original?

Not close enough to a full copy, clearly. But close enough that even the flaws of human thinking are often reproduced faithfully.

"The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." - Edsger Dijkstra

Without going to look up the exact quote, I remember an AI researcher years (decades) ago saying something to the effect of, Biologists look at living creatures and wonder how they can be alive; astronomers look at the cosmos and wonder what else is out there; those of us in artificial intelligence look at computer systems and wonder how they can be made to wonder such things.

Don't be sycophantic. Disagree and push back when appropriate.

Come up with original thought and original ideas.

Have long term goals that aren't programmed by an external source.

Do something unprompted.

The last one IMO is more complex than the rest, because LLMs are fundamentally autocomplete machines. But what happens if you don't give them any prompt? Can they spontaneously come up with something, anything, without any external input?

> "If what LLMs do today isn't actual thinking, what is something that only an actually thinking entity can do that LLMs can't?"

Independent frontier maths research, i.e. coming up with and proving (preferably numerous) significant new theorems without human guidance.

I say that not because I think the task is special among human behaviours. I think the mental faculties that mathematicians use to do such research are qualitatively the same ones all humans use in a wide range of behaviours that AI struggles to emulate.

I say it because it's both achievable (in principle, if LLMs can indeed think like humans) and verifiable. Achievable because it can be viewed as a pure text generation task and verifiable because we have well-established, robust ways of establishing the veracity, novelty and significance of mathematical claims.

It needs to be frontier research maths because that requires genuinely novel insights. I don't consider tasks like IMO questions a substitute as they involve extremely well trodden areas of maths so the possibility of an answer being reachable without new insight (by interpolating/recombining from vast training data) can't be excluded.

If this happens I will change my view on whether LLMs think like humans. Currently I don't think they do.

solve simple maths problems, for example the kind found in the game 4=10 [1]

Doesn't necessarily have to reliably solve them, some of them are quite difficult, but llms are just comically bad at this kind of thing.

Any kind of novel-ish(can't just find the answers in the training-data) logic puzzle like this is, in my opinion, a fairly good benchmark for "thinking".

Until a llm can compete with a 10 year old child in this kind of task, I'd argue that it's not yet "thinking". A thinking computer ought to be at least that good at maths after all.

[1] https://play.google.com/store/apps/details?id=app.fourequals...

Have needs and feelings? (I mean we can’t KNOW that they don’t and we know of this case of an LLM in experiment that try to avoid being shutdown, but I think the evidence of feeling seems weak so far)

Ya, the fact this was published on November 3, 2025 is pretty hilarious. This was last year's debate.

I think the best avenue toward actually answering your questions starts with OpenWorm [1]. I helped out in a Connectomics research lab in college. The technological and epistemic hurdles are pretty daunting, but so were those for Genomics last century, and now full-genome sequencing is cheap and our understanding of various genes is improving at an accelerating pace. If we can "just" accurately simulate a natural mammalian brain on a molecular level using supercomputers, I think people would finally agree that we've achieved a truly thinking machine.

[1]: https://archive.ph/0j2Jp

> what is something that only an actually thinking entity can do that LLMs can't?

This is pretty much exactly what https://arcprize.org/arc-agi is working on.

See https://arcprize.org/

What people are interested in is finding a definition for intelligence, that is an exact boundary.

That's why we first considered tool use, being able to plan ahead as intelligence, until we have found that these are not all that rare in the animal kingdom in some shape. Then with the advent of IT what we imagined as impossible turned out to be feasible to solve, while what we though of as easy (e.g. robot movements - a "dumb animal" can move trivially it surely is not hard) turned out to require many decades until we could somewhat imitate.

So the goal post moving of what AI is is.. not moving the goal post. It's not hard to state trivial higher bounds that differentiates human intelligence from anything known to us, like invention of the atomic bomb. LLMs are nowhere near that kind of invention and reasoning capabilities.

Strive for independence.

> "If what LLMs do today isn't actual thinking, what is something that only an actually thinking entity can do that LLMs can't?"

Invent some novel concept, much the same way scientists and mathematicians of the distant past did? I doubt Newton's brain was simply churning out a stream of the "next statistically probable token" until -- boom! Calculus. There was clearly a higher order understanding of many abstract concepts, intuition, and random thoughts that occurred in his brain in order to produce something entirely new.

> That is something that only an actually thinking entity can do that LLMs can't?

Training != Learning.

If a new physics breakthrough happens tomorrow, one that say lets us have FTL, how is an LLM going to acquire the knowledge, how does that differ from you.

The break through paper alone isnt going to be enough to over ride its foundational knowledge in a new training run. You would need enough source documents and a clear path deprecate the old ones...

The issue is that we have no means of discussing equality without tossing out the first order logic that most people are accustomed to. Human equality and our own perceptions of other humans as thinking machines is an axiomatic assumption that humans make due to our mind's inner sense perception.

Form ideas without the use of language.

For example: imagining how you would organize a cluttered room.

operate on this child

You should definitely also read "Permutation City" by Greg Egan[1] if you haven't already! Amazing book...

[1]https://www.gregegan.net/PERMUTATION/Permutation.html

> you're in a coma

Coma -> unconscious -> not thinking

That's like saying you're thinking while asleep. Are you really?

my favourite sport is watching people trying to exclude the possibility of AI thinking/becoming sentient/self-aware/whatever-nebulous-magical-term-you-like. Every single time they manage to exclude a whole section of humanity, because everything that AI "can never do", there are living, breathing people who can't do it either.

The New Yorker is available via Libby electronically if your library subscribes. In Santa Clara county I get it this way. So we pay library taxes and get access, not technically free. In plus side, a lot more content and the cartoons, on minus side, have to filter a lot of New York only culture and other articles for your interests.

Archive link in the post body?

(Apologies if that's been edited in after your comment)

Lynx works well.

Pay for the work they did?

I mean, yeah why not? Journalism should surface both perspectives, and readers should understand that any perspective is clouded (biased if you will) one way or another. No matter whose quotes you include, they will be biased because we as humans inherently is. Some articles/opinion pieces will only have one perspective, and that's OK too, you shouldn't take everything you read at face value, go out and search for more perspectives if you wanna dive deeper.