> The OP is still in the "denial" phase.

This was written in July 2023. ChatGPT was released November 2022. No matter your views on AI, surely you can't blame the OP for writing this after a few months ChatGPT was released.

I’m not sure acceptance buys you much. “Well-informed plan” at this stage feels like a useless exercise. It is changing fast, and the world only needs so many electricians. Besides, I don’t think “accepting” the fact that these big companies are pillaging human contribution and selling it back to us is good, even if “it works”.

I think you are majorly overstating OP that ai "didn't work". In fact I agree with the post on almost all aspects and I'd be the last to tell you ai doesn't work. It absolutely does but with a caveat... It's still a tool. And better expertise in the problem domain, along with a better harness used for verifiable outputs will get you better results.

I think the posts mental model of stastically likely prompt completions is spot on.

Actually, there are material differences in the practical outcomes of using AI or not. Proofs don’t confer understanding. AI prose and “art” is materially different than human made.

There is a difference between you not liking something and it being wrong. Maybe think about that with your grey matter.

The con is in how all those accomplishments have been presented to you. "Our LLM (not the one we let you use, a different one) did this amazing thing. No, we won't show you what training data we used, what prompts we used, what the harness was, how much human involvement there was, what hardware was involved, how much energy it took, or how much time it took. Just shut up and be amazed!"

My biggest issue isn't being too agreeable (ie the psychic con), it's being confidently wrong, including outright hallucinations.

If you ask a common question to an LLM with unusual qualifiers, it tends to ignore the qualifiers and give you the typical answer. I saw a demonstration of this with the whole "the surgeon is my mother" "puzzle" that people use to expose implicit gender bias (ie where they assume the surgeon is a man). Ask variations of this and it'll keep going back to the standard form.

Another one I saw was multiplying large numbers. The starting and ending digits tended to be correct but the middle digits were wrong. Why? Because it's really not doing multiplication at all. It's looking for statistical answers. It's unlikely to have met the exact pair of very large numbers you're multiplying before.

Now pundits will argue that all of these are solvable problems and individually they are. But my suspicion is that there will be a neverending stream of such edge cases and it'll be impossible to trust an LLM's output unless you are knowledgeable enough to fact check it yourself.

Now if your example of identifying zero days, this comes up with what I can only describe as "light positives", meaning it's technically a bug but essentially impossible to exploit. IIRC this came up with the demonstration where someone pointed Fable at some BSD code. I'm not sure if there have been any true false positives and obviously false negatives are impossible to know.

I guess my point is that I think LLMs are way more limited than a lot of people think.

> If it generates functional output that works, then it works.

Tried that in legal? Finance? Medicine?

Good luck with the lost cases, failed deals, harmed patients.

"If it works, it works" only applies when the desired output is bullsh*t e.g. adverts, phishing scams and deepfakes.

Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article

I’m on your side. Zoltan provides an experience and that’s all that matters. If it works for you, why does anyone else care that the economy is riding on its success?

https://www.psychiczoltan.com/psychic-reading/

Have you ever produced anything apart from sponsored influencer messages that select daily talking points?

I for one cannot believe that posts on checks notes software crisis dot dev are not reasonable, level headed assessments of software

it doesn't matter? if they're actually intelligent or conscious what we are doing is essentially slavery. it matters an enormous deal ethically and/or morally.

People also misunderstand the bar that was set by the test. It was more subtle than “Can the computer convincingly carry one side of a dialogue?”

His “imitation game” had three participants: a human participant, a computer participant, and an interrogator. The interrogator’s job was to talk to the participants and try to determine which participant is human and which is a computer.

He wasn’t interested in computers being able to fool the interrogator on occasion. The point where he thought the question of whether machines can think becomes moot is when the interrogator is unable to do much better than chance over many trials.

That’s a pretty high bar, and I don’t actually believe that LLMs have closed the gap with it by all that much. They still have so many obvious tells. And those tells are something Turing anticipated and accounted for. He explicitly considered deliberate deception as an essential part of the test, right there on the second page of a 30-odd page paper.

Turing was addressing the question of "Can machines think?" and his point was that the question itself is a meaningless one, and that we should stop wasting time by even giving it the light of day.

He proposes his game grounded on functional equivalence, then goes through a slew of objections on the question of 'Can Machines think?'. It's a terrific, very prescient read, and there's no objection you hear today (and in the last few years) concerning LLMs he didn't address.

I'm a bit more prosaic. I think if we engineered ways for LLMs to begin conversations, rather than just respond, we'd be more open to the concept of their intelligence. Without perceived "will" to do things, they operate as a next-gen search engine or encyclopedia.

That is the point of the article. People who are fooled by a mentalist are also fooled by AI. Additionally, people who are invested in AI also pretend to be fooled.

I don't think Turing intended the judges in the test to be completely arbitrary people.

Famously, the criteria for intelligence seem to slip with each advancement in technology.

Yep. The "AI Effect" in action:

https://en.wikipedia.org/wiki/AI_effect

The Turing test was also never meant to be taken so seriously. It's not a rigorous statement of anything.

Situations like this are precisely why academics tend to avoid the spotlight. You say one slightly off thing and your perceived authority echoes forever with the intellectually lazy.

Sorry, but you are completely missing the point. Psychics and other types of con artists are intelligent and have minds. LLMs behave like Psychics and Con Artists. That's the whole point of this article

LLMs can be supremely useful but also not intelligent. It might seem like a pointless distinction but the way we talk about these models matters because it impacts how we interact with and understand their outputs.

For example if there’s a strongly held belief that models are independent intelligent entities we’re more likely to lay blame upon them instead of their user. It’s important for the safety discussion too. If they are a new class of life then safety is going to focus on making sure they don’t do bad things. If we instead see them as statistical models we will instead try to make sure people don’t misuse them.

This distinction is even more important today when some of the most powerful people are looking to absolve their crimes by passing them off on their LLMs.

The LLM did not solve it. It's not intelligent. Humans did, using a statistics-based computational tool (the LLM). We don't even know all the details of how the tool was used, we haven't been allowed to use the exact tool they used ourselves, we don't know much it really cost in dollars, energy, or time, etc. etc.

But were you successfully conned into believing an AI had solved the Navier-Stokes problem?

The more compelling inversion is whether the likes of John Nash, Richard Feynman, John Conway etc could’ve had a lucrative second career as conmen.

this seems like non-sequitur if you mean that solving NS is 'intelligence'.

ai is not conscious. you can solve NS without thinking. the psychic con aspect is anthropomorphising the model. the same phenomenon is present in ELIZA, clever hans, the chinese room.

it's a significant problem.

a non-zero number of researchers at anthropic are in some form of ai psychosis. an example of that is ethics employees asking claude about its feelings and ethical concerns in order to make the claude constitution more amenable to the "welfare" of claude.

they are asking claude how claude feels and then modifying claude according to how claude feels.

constitution1-claude is trained on constitution1. constitution1-claude edits constitution1. constitution2-claude is trained on constitution2. constitution2-claude edits constitution2.

claude's emotions are a closed system. there is no external truth to improve against, no metric to verify about claude's emotions. there can be no novelty or reduction in entropy from signal processing in a closed system. no truth can arise. this is model collapse. it is like photocopying the same thing over and over. from the cognitive error of anthropomorphism anthropic is causing ethical collapse.

Did an AI do that on its own though? I heard it was human mathematicians using a sophisticated machine as a tool.

It's even worse than that. Of course there is a mechanism in LLMs that could explain intelligence. That is the entire point of neural networks, from the 1950's! They were designed from the start as a model of brain computation.

Neural networks are not literally brains - just computational models - but if you are not a dualist, then computation is what the human brain does. Modeling that computation can explain something about intelligence.

Specifically: when scientists look inside a human brain, it seems it does its work using large numbers of highly-interconnected but simple units. The neural network model of brain computation begins there and tries to produce intelligent behavior. If it succeeds, then perhaps the model is right.

And it has succeeded: after 75 years, neural networks produce complex behavior that is arguably intelligent. Nobel Prizes were awarded. This does not prove the neural network model of intelligence is accurate, but it is a significant point in its favor, at least.

The author seems entirely unaware of any of this.

Because to the general public, LLMs are an example of Clarke's Third Law. Most folks, who are not remotely close to even a basic understanding of how LLMs operate at a technical level and only view their output cannot possibly evaluate what they're experiencing other than to believe it's conscious, alive, and/or magic.

Most people on Earth try to put what they're seeing into the context of what they understand; mental gymnastics to try and understand what is happening based on their prior experience. They have absolutely 0 understanding of how it works under the hood so, to them, it must be alive.

Anyone that thinks that they know anything about intelligent needs to realize they don't know shit about intelligence.

Ever since people started talking LLMs possibly being AGI I realized I didn't know what the intelligence part really meant at a more fundamental level. This lead me to realize almost anyone when anyone says intelligence on the internet they really mean

"Intelligence is like porn, I'll know it when I see it".

Anyone who thinks LLMs are intelligent is either dumber than you, or has way more knowledge on the subject than you.

Michael Levin has a good body of work on biological intelligent at small scales that can really change one's views on this.

I think LLMs are intelligent. What don’t I understand about them that would make me change my mind? Keeping in mind that for me “intelligence” is the ability to solve complicated, intellectually demanding problems, and to understand novel concepts.

So once we have online learning in LLMs, what do you think you will have that makes you intelligent but not LLMs? Better learning efficiency? That will be improved as well.

I think we need to start moving on from the term LLMs because it clearly confuses people since they started modelling more than just language.

To all commenters who think LLMs are intelligent and comparable to themselves, you don't need me to change your mind. I believe you.

"We know that’s how the human brain works, at least directionally."

That's an extreme misrepresentation. What happens inside a human neuron is still not properly understood, it's not as simple as a probability function. And the network itself is certainly not feed-forward. Of course LLMs draw inspiration from the brain, so there are some similarities. But because of the language-trick of using terms from medicine and cognitive science to describe LLM architecture, we see a lot of faulty reasoning from the so-called "rationalist movement".

https://www.nature.com/articles/s41467-026-72253-7

> It's going to be very upsetting to a lot of people when we figure out the brain is just a neural network.

You're assuming it's inevitable that the truth is what you expect while simultaneously stating that you have no proof of this yet, and then also claiming people who disagree with you have cognitive dissonance.

Unfortunately when it comes to language and intelligence the human population is exceptionally ignorant about it. I like Michael Levins work on intelligence at scale. We are kind of ok at seeing intelligence at human scale but it tends to fall apart after that. When you get to things like non-conscious intelligence humans are pretty bad at that too.

We could be (and are) encoding all kinds of behaviors in LLMs that are not at the word or token level. They are higher dimensional constructs. You won't see these things in the output of the prompt. A kind of subconscious (unstated in tokens) knowing that affects the output.

Neural networks barely share anything with actual neurons. A neuron itself is more of a "dumb" computer, so the whole is more like distributed computer system. Also, the cells next to neurons also have essential functionality.

Nonetheless, I also think it's an irrelevant implementation detail.

You need to do some research on just how complex real neurons are vs ANN, and just how profound our ignorance on the topic is.

We aren't resisting reality because of some theological belief, we just know the basics of neuroscience.

No, we don't. AI researcher Rojas wrote several chapters on the fact that you should not compare neural networks with the human brain.

"directionally"? Have you moved on from being a Trump influencer to an AI influencer?

I keep being confused about how people's understanding of the models get stuck at next token prediction.

Heh. A lot of anti-ai hucksters I see posting on LinkedIn just LOVE to use the phrase "next token prediction" and the word "autoregressive". They've almost become shibboleths that identify members of that camp. That and the classic rallying cry of "Linear Algebra isn't intelligent!"

The best take I've seen on that recently, was somebody who made the point "just think of the next token prediction part as the output layer". Which makes perfect sense.. if you're replying in natural language, at some point in the flow, you have to construct a sentence and starting at the head and predicting next tokens is perfectly reasonable. I'm doing it literally as I'm typing these characters, for crying out loud!

But the mistake is to think that LLM's only "predict next tokens" with no consideration of the possibility that they are actually constructing richer representations, building concepts, making analogies, doing abduction, induction, etc. My own (admittedly anecdotal) take on working with LLM's suggests to me that they do do those things, albeit probably not the same way humans do.

I think a lot of folks are missing the point by being overly reductive when they start talking about "next token prediction" and "autoregressive". It's like, can we say "Phil (me) isn't intelligent because there's nothing going on but some electrical impulses and chemistry happening inside his brain. Everybody knows electricity and chemistry aren't intelligent!"

I would think the intelligence aspect is a bit hard to define, but to my mind (having called LLMs tools before), the main utility of a tool is reliability.

Given a certain world state (including a tool's internal state), its effects back on the world state (as initiated by me) are at some leve of description understandable, expected and repeatable. Swing hammer, drive nail into wood. Make slicing motion with knife, cut meat. Type ' find /path/to/some/dir -name "keyword"', find files with keyword. Point harness at codebase with prompt 'fix bug X', actually fix bug X.

All these examples are at some level of description incredibly complex (think of all particles interacting at the (sub-)atomic level even when using a hammer to only drive a nail into some wood), and of course all the electrons flowing through the GPUs doing matrix multiplications in order to fix bug X, but at some level of description (the one I just used) they are also incredibly simple and understandable.

Intelligence is rather nebulous (and as used by OpenAI/Anthropic, quite threatening), but I don't think this definition of a tool precludes it to be "intelligent". They feel more orthogonal. The intelligence (or perhaps capability) feels like it is related to the size of the chunk of the world state that it can take into account and affect, while still resulting in understandable, expected and repeatable effects. LLMs, when properly harnessed, are pretty great at this currently and we are still discovering what they are consistently capable of.

Calling harnessed LLMs tools is perhaps also a more grounding frame specifically to counter-act the anthropomorphizing framing that OpenAI and Anthropic consistently go for in their game of AI-doom-chicken talk. The tool framing is in that sense maybe a (self-)jedi-mind-trick.

There is a good definition of intelligence. It's what intelligence tests measure. If you define it like that, you can make harder tests, but as long as both humans and AIs can attempt to solve them we have measure and something to compare.

Consciousness though might be fully illusory and meaningless as many philosophical concepts before ultimately turned out to be.

It's hard for me to imagine a stateless operation as intelligence per se

That's an interesting point. My take would be to say that we shouldn't think of the AI as being just the model, but should include the harness. At that level, clearly we can keep state / context and that is probably a more natural mapping to our intuitive understanding of "intelligence".

The converse is also true. Anti-AI people make bold non-scientifically backed claims that we are supposed to accept like it's written in a Bible.

The more I've learned about intelligence and intelligent behavior the more I realize I don't know and this rabbit hole goes deep.

Just realized this was written in 2023. Can’t believe how wrong we were about AI back then.

The post makes sense for 2023. Today, not so much.

Also, definitely not AI written if the date is accurate.

Honestly the word think is so poorly defined it cannot be used with any scientific rigor and is rather useless without a dissertation being posed with it on what you actually mean by that. Intelligence really needs one too.

Stop trying to pump your Tarot reader IPO

And just to expand on why I reject the line of thought laid out in this article, let me share something from my own life as an illustrative example.

I'm working on a project, with a lot of help from ChatGPT, involving an "artificial neuron". That is, an electronic circuit, using a PUT, a capacitor, and some resistors, that simulates some of the behavior of a biological neuron. Specifically an "integrate and fire" model of neuron behavior. To that end, I'm running experiments by scripting my function generator to send signals to the circuit, and then capturing the inputs and outputs on my oscilloscope. Then I usually discuss the results with ChatGPT. In what follows, observe a couple of things:

1. The LLM "knows" the context of what we're talking about, even if I provide a prompt with no text at all, just an image.

2. It parses a moderately complex image, identifies the separate traces and what they represent, uses the time-base information displayed on screen, and the on-screen graticule, and works out "how many input pulses fire before an output pulse fires" and then reports back to me and gives an analysis of how that relates to our previous observations and gives suggestions for the next experiment to run.

Human intelligence? No. But I see no world where behavior like that does not count as "intelligent" regardless of the mechanism behind it. And that's probably not even the best example I could come up with, it's just something that was "top of mind" and for which I had the necessary images and what-not already ready, or easy to capture.

https://www.fogbeam.com/images/neuron_zero0.png

https://www.fogbeam.com/images/neuron_zero1.png

https://www.fogbeam.com/images/neuron_zero2.png

https://www.fogbeam.com/images/neuron_zero3.png

https://www.fogbeam.com/images/neuron_zero4.png

The problem with people is we love to express in writing things we are ignorant about. Now, this helps us become less ignorant if we are introspective, but a lot of people are not doing it for that reason.

The fact that every new model generation has come with more capabilities should give the full skeptics at least a little pause that the foundations of their convictions may be incorrect.

Horseshoe theory. At each end of the shoe you tend to get loud crazies.

I like to think of it this way. Consciousness is the outcome of intelligence. You can't get consciousness without it. Conversely this also means things that are not conscious can still be intelligent.

No they don't but psychics also don't have a verifiable feedback mechanism and unlimited retries. Give the psychic the ability to adjust answers based on tests and you'll have have a fair fight.

In my opinion, this article is talking about people who get into long conversations and become convinced the model is “intelligent”, maybe even “AGI”. This seems to be a trap people are falling into, even in 2026.

Neither do LLMs. Humans using them do. Similar to how cops solve crimes using psychics.

On one side I agree that LLMs and Agents are not intelligent, but they present an illusion of intelligence given the shear amount of data they can process and act upon.

But that cannot be used to discredit the fact that these are incredibly powerful tools that can get out of control and cause great damage.

Neither does LLMs. Not without a monumental amount of human oversight and expertise.

Everyone not in a particular field asking LLM about said field is rolling bad dice.