I kinda agree with both of you. It might be a required abstraction, but it's a leaky one.

Long before LLMs, I would talk about classes / functions / modules like "it then does this, decides the epsilon is too low, chops it up and adds it to the list".

The difference I guess it was only to a technical crowd and nobody would mistake this for anything it wasn't. Everybody know that "it" didn't "decide" anything.

With AI being so mainstream and the math being much more elusive than a simple if..then I guess it's just too easy to take this simple speaking convention at face value.

EDIT: some clarifications / wording

When I see these debates it's always the other way around - one person speaks colloquially about an LLM's behavior, and then somebody else jumps on them for supposedly believing the model is conscious, just because the speaker said "the model thinks.." or "the model knows.." or whatever.

To be honest the impression I've gotten is that some people are just very interested in talking about not anthropomorphizing AI, and less interested in talking about AI behaviors, so they see conversations about the latter as a chance to talk about the former.

Well "reasoning" refers to Chain-of-Thought and if you look at the generated prompts it's not hard to see why it's called that.

That said, it's fascinating to me that it works (and empirically, it does work; a reasoning model generating tens of thousands of tokens while working out the problem does produce better results). I wish I knew why. A priori I wouldn't have expected it, since there's no new input. That means it's all "in there" in the weights already. I don't see why it couldn't just one shot it without all the reasoning. And maybe the future will bring us more distilled models that can do that, or they can tease out all that reasoning with more generated training data, to move it from dispersed around the weights -> prompt -> more immediately accessible in the weights. But for now "reasoning" works.

But then, at the back of my mind is the easy answer: maybe you can't optimize it. Maybe the model has to "reason" to "organize its thoughts" and get the best results. After all, if you give me a complicated problem I'll write down hypotheses and outline approaches and double check results for consistency and all that. But now we're getting dangerously close to the "anthropomorphization" that this article is lamenting.

"All models are wrong, but some models are useful," is the principle I have been using to decide when to go with an anthropomorphic explanation.

In other words, no, they never accurately describe what the LLM is actually doing. But sometimes drawing an analogy to human behavior is the most effective way to pump others' intuition about a particular LLM behavior. The trick is making sure that your audience understands that this is just an analogy, and that it has its limitations.

And it's not completely wrong. Mimicking human behavior is exactly what they're designed to do. You just need to keep reminding people that it's only doing so in a very superficial and spotty way. There's absolutely no basis for assuming that what's happening on the inside is the same.

It's not just distorting discussions it's leading people to put a lot of faith in what LLMs are telling them. Was just on a zoom an hour ago where a guy working on a startup asked ChatGPT about his idea and then emailed us the result for discussion in the meeting. ChatGPT basically just told him what he wanted to hear - essentially that his idea was great and it would be successful ("if you implement it correctly" was doing a lot of work). It was a glowing endorsement of the idea that made the guy think that he must have a million dollar idea. I had to be "that guy" who said that maybe ChatGPT was telling him what he wanted to hear based on the way the question was formulated - tried to be very diplomatic about it and maybe I was a bit too diplomatic because it didn't shake his faith in what ChatGPT had told him.

> people are genuinely talking about them thinking and reasoning when they are doing nothing of that sort

Do you believe thinking/reasoning is a binary concept? If not, do you think the current top LLM are before or after the 50% mark? What % do you think they're at? What % range do you think humans exhibit?

> people are genuinely talking about them thinking and reasoning when they are doing nothing of that sort

With such strong wording, it should be rather easy to explain how our thinking differs from what LLMs do. The next step - showing that what LLMs do precludes any kind of sentience is probably much harder.

I think it's worth distinguishing between the use of anthropomorphism as a useful abstraction and the misuse by companies to fuel AI hype.

For example, I think "chain of thought" is a good name for what it denotes. It makes the concept easy to understand and discuss, and a non-antropomorphized name would be unnatural and unnecessarily complicate things. This doesn't mean that I support companies insisting that LLMs think just like humans or anything like that.

By the way, I would say actually anti-anthropomorphism has been a bigger problem for understanding LLMs than anthropomorphism itself. The main proponents of anti-anthropomorphism (e.g. Bender and the rest of "stochastic parrot" and related paper authors) came up with a lot of predictions about things that LLMs surely couldn't do (on account of just being predictors of the next word, etc.) which turned out to be spectacularly wrong.

I thought this too but then began to think about it from the perspective of the programmers trying to make it imitate human learning. That's what a nn is trying to do at the end of the day, and in the same way I train myself by reading problems and solutions, or learning vocab at a young age, it does so by tuning billions of parameters.

I think these models do learn similarly. What does it even mean to reason? Your brain knows certain things so it comes to certain conclusions, but it only knows those things because it was ''trained'' on those things.

I reason my car will crash if I go 120 mph on the other side of the road because previously I have 'seen' where the input is a car going 120mph has a high probability of producing a crash, and similarly have seen input where the car is going on the other side of the road, producing a crash. Combining the two would tell me it's a high probability.

how do you account for the success of reasoning models?

I agree these things don't think like we do, and that they have weird gaps, but to claim they can't reason at all doesn't feel grounded.

Serendipitous name...

In part I agree with the parent.

  >> it pointless to *not* anthropomorphize, at least to an extent.

On the other hand, it generates bias. This bias can lead to errors.

So the real answer is (imo) that it is fine to anthropomorphise but recognize that while doing so can provide utility and help us understand, it is WRONG. Recognizing that it is not right and cannot be right provides us with a constant reminder to reevaluate. Use it, but double check, and keep checking making sure you understand the limitations of the analogy. Understanding when and where it applies, where it doesn't, and most importantly, where you don't know if it does or does not. The last is most important because it helps us form hypotheses that are likely to be testable (likely, not always. Also, much easier said than done).

So I pick a "grey area". Anthropomorphization is a tool that can be helpful. But like any tool, it isn't universal. There is no "one-size-fits-all" tool. Literally, one of the most important things for any scientist is to become an expert at the tools you use. It's one of the most critical skills of *any expert*. So while I agree with you that we should be careful of anthropomorphization, I disagree that it is useless and can never provide information. But I do agree that quite frequently, the wrong tool is used for the right job. Sometimes, hacking it just isn't good enough.

> On the contrary, anthropomorphism IMO is the main problem with narratives around LLMs

I hold a deep belief that anthropomorphism is a way the human mind words. If we take for granted the hypothesis of Franz de Waal, that human mind developed its capabilities due to political games, and then think about how it could later lead to solving engineering and technological problems, then the tendency of people to anthropomorphize becomes obvious. Political games need empathy or maybe some other kind of -pathy, that allows politicians to guess motives of others looking at their behaviors. Political games directed the evolution to develop mental instruments to uncover causality by watching at others and interacting with them. Now, to apply these instruments to inanimate world all you need is to anthropomorphize inanimate objects.

Of course, it leads sometimes to the invention of gods, or spirits, or other imaginary intelligences behinds things. And sometimes these entities get in the way of revealing the real causes of events. But I believe that to anthropomorphize LLMs (at the current stage of their development) is not just the natural thing for people but a good thing as well. Some behavior of LLMs is easily described in terms of psychology; some cannot be described or at least not so easy. People are seeking ways to do it. Projecting this process into the future, I can imagine how there will be a kind of consensual LLMs "theory" that explains some traits of LLMs in terms of human psychology and fails to explain other traits, so they are explained in some other terms... And then a revolution happens, when a few bright minds come and say that "anthropomorphism is bad, it cannot explain LLM" and they propose something different.

I'm sure it will happen at some point in the future, but not right now. And it will happen not like that: not just because someone said that anthropomorphism is bad, but because they proposed another way to talk about reasons behind LLMs behavior. It is like with scientific theories: they do not fail because they become obviously wrong, but because other, better theories replace them.

It doesn't mean, that there is no point to fight anthropomorphism right now, but this fight should be directed at searching for new ways to talk about LLMs, not to show at the deficiencies of anthropomorphism. To my mind it makes sense to start not with deficiencies of anthropomorphism but with its successes. What traits of LLMs it allows us to capture, which ideas about LLMs are impossible to wrap into words without thinking of LLMs as of people?

I don't agree. Most LLMs have been trained on human data, so it is best to talk about these models in a human way.

I disagree. Anthropomorphization can be a very useful tool but I think it is currently over used and is a very tricky tool to use when communicating with a more general audience.

I think looking at physics might be a good example. We love our simplified examples and there's a big culture of trying to explain things to the lay person (mostly because the topics are incredibly complex). But how many people have misunderstood an observer of a quantum event with "a human" and do not consider "a photon" as an observer? How many people think in Schrodinger's Cat that the cat is both alive and dead?[0] Or believe in a multiverse. There's plenty of examples we can point to.

While these analogies *can* be extremely helpful, they *can* also be extremely harmful. This is especially true as information is usually passed through a game of telephone[1]. There is information loss and with it, interpretation becomes more difficult. Often a very subtle part can make a critical distinction.

I'm not against anthropomorphization[2], but I do think we should be cautious about how we use it. The imprecise nature of it is the exact reason we should be mindful of when and how to use it. We know that the anthropomorphized analogy is wrong. So we have to think about "how wrong" it is for a given setting. We should also be careful to think about how it may be misinterpreted. That's all I'm trying to say. And isn't this what we should be doing if we want to communicate effectively?

[0] It is not. It is either. The point of this thought experiment is that we cannot know the answer without looking inside. There is information loss and the event is not deterministic. It directly relates to the Heisenberg Uncertainty Principle, Godel's Incompleteness, or the Halting Problem. All these things are (loosely) related around the inability to have absolute determinism.

[1] https://en.wikipedia.org/wiki/Telephone_game

[2] https://news.ycombinator.com/item?id=44494022

The intentional stance was Daniel Dennett's creation and a major part of his life's work. There are actually (exactly) three stances in his model: the physical stance, the design stance, and the intentional stance.

https://en.wikipedia.org/wiki/Intentional_stance

I think the design stance is appropriate for understanding and predicting LLM behavior, and the intentional stance is not.

> Why don't LLMs get frustrated with you if you ask them the same question repeatedly?

To be fair, I have had a strong sense of Gemini in particular becoming a lot more frustrated with me than GPT or Claude.

Yesterday I had it ensuring me that it was doing a great job, it was just me not understanding the challenge but it would break it down step by step just to make it obvious to me (only to repeat the same errors, but still)

I’ve just interpreted it as me reacting to the lower amount of sycophancy for now

> It is like someone inventing the aeroplane and someone looks at it and says "oh, it's flying, I guess it's a bird". It's not a bird!

We tried to mimic birds at first; it turns out birds were way too high-tech, and too optimized. We figured out how to fly when we ditched the biological distraction and focused on flight itself. But fast forward until today, we're reaching the level of technology that allows us to build machines that fly the same way birds do - and of such machines, it's fair to say, "it's a mechanical bird!".

Similarly, we cracked computing from grounds up. Babbage's difference engine was like da Vinci's drawings; ENIAC could be seen as Wright brothers' first flight.

With planes, we kept iterating - developing propellers, then jet engines, ramjets; we learned to move tons of cargo around the world, and travel at high multiples of the speed of sound. All that makes our flying machines way beyond anything nature ever produced, when compared along those narrow dimensions.

The same was true with computing: our machines and algorithms very quickly started to exceed what even smartest humans are capable of. Counting. Pathfinding. Remembering. Simulating and predicting. Reproducing data. And so on.

But much like birds were too high-tech for us to reproduce until now, so were general-purpose thinking machines. Now that we figured out a way to make a basic one, it's absolutely fair to say, "I guess it's like a digital mind".

This. If an AI spits out incorrect code then i immediately create a new chat and reprompt with additional context.

'Dont use regex for this task' is a common addition for the new chat. Why does AI love regex for simple string operations?

The only reason that sounds weird to you is because you have the experience of being human. Human behavior is not magic. It's still just statistics. You go to the bathroom when you have to pee not because some magical concept of consciousness, but because a reciptor in your brain goes off and starts the chain of making you go to the bathroom. AI's are not magic, but nobody has sufficiently provided any proof we are somehow special either.

> there exists a string of words (not necessary ones that make sense) that will get the LLM to a better position to answer

exactly. The opposite is also true. You might supply more clarifying information to the LLM, which would help any human answer, but it actually degrades the LLM's output.

You are conflating anthropomorphism with personification. They are not the same thing. No one believes their guitar or car or boat is alive and sentient when they give it a name or talk to or about it.

https://www.masterclass.com/articles/anthropomorphism-vs-per...

I'm not convinced... we use these terms to assign roles, yes, but these roles describe a utility or assign a responsibility. That isn't anthropomorphizing anything, but it rather describes the usage of an inanimate object as tool for us humans and seems in line with history.

What's the utility or the responsibility of AI, what's its usage as tool? If you'd ask me it should be closer to serving insights than "reasoning thoughts".

I think anthropomorphizing LLMs is useful, not just a marketing tactic. A lot of intuitions about how humans think map pretty well to LLMs, and it is much easier to build intuitions about how LLMs work by building upon our intuitions about how humans think than by trying to build your intuitions from scratch.

Would this question be clear for a human? If so, it is probably clear for an LLM. Did I provide enough context for a human to diagnose the problem? Then an LLM will probably have a better chance of diagnosing the problem. Would a human find the structure of this document confusing? An LLM would likely perform poorly when reading it as well.

Re-applying human intuitions to LLMs is a good starting point to gaining intuition about how to work with LLMs. Conversely, understanding sequences of tokens and probability spaces doesn't give you much intuition about how you should phrase questions to get good responses from LLMs. The technical reality doesn't explain the emergent behaviour very well.

I don't think this is mutually exclusive with what the author is talking about either. There are some ways that people think about LLMs where I think the anthropomorphization really breaks down. I think the author says it nicely:

> The moment that people ascribe properties such as "consciousness" or "ethics" or "values" or "morals" to these learnt mappings is where I tend to get lost.

IMHO it happens for the same reason we see shapes in clouds. The human mind through millions of years has evolved to equate and conflate the ability to generate cogent verbal or written output with intelligence. It's an instinct to equate the two. It's an extraordinarily difficult instinct to break. LLMs are optimised for the one job that will make us confuse them for being intelligent

aAnthrophormisation happens because Humans are absolutely terrible at evaluating systems that give converdational text output.

ELIZA fooled many people into think it was conscious and it wasn't even trying to do that.

> because it's perceived as good marketing

We are making user interfaces. Good user interfaces are intuitive and purport to be things that users are familiar with, such as people. Any alternative explanation of such a versatile interface will be met with blank stares. Users with no technical expertise would come to their own conclusions, helped in no way by telling the user not to treat the chat bot as a chat bot.

Nobody cares about what’s perceived as good marketing. People care about what resonates with the target market.

But yes, anthropomorphising LLMs is inevitable because they feel like an entity. People treat stuffed animals like creatures with feelings and personality; LLMs are far closer than that.

True but also researchers want to believe they are studying intelligence not just some approximation to it.

There is hidden state as plain as day merely in the fact that logits for token prediction exist. The selected token doesn't give you information about how probable other tokens were. That information, that state which is recalculated in autoregression, is hidden. It's not exposed. You can't see it in the text produced by the model.

There is plenty of state not visible when an LLM starts a sentence that only becomes somewhat visible when it completes the sentence. The LLM has a plan, if you will, for how the sentence might end, and you don't get to see an instance of that plan unless you run autoregression far enough to get those tokens.

Similarly, it has a plan for paragraphs, for whole responses, for interactive dialogues, plans that include likely responses by the user.

There are many theories that are true but not trivially true. That is, they take a statement that seems true and derive from it a very simple model, which is then often disproven. In those cases however, just because the trivial model was disproven doesn't mean the theory was, though it may lose some of its luster by requiring more complexity.

If you dont know, that's not necessarily anyone's fault, but why are you dunking into the conversation? The hidden state is a foundational part of a transformers implementation. And because we're not allowed to use metaphors because that is too anthropomorphic, then youre just going to have to go learn the math.

Hidden state in the form of the activation heads, intermediate activations and so on. Logically, in autoregression these are recalculated every time you run the sequence to predict the next token. The point is, the entire NN state isn't output for each token. There is lots of hidden state that goes into selecting that token and the token isn't a full representation of that information.

do LLM models consider future tokens when making next token predictions?

eg. pick 'the' as the next token because there's a strong probability of 'planet' as the token after?

is it only past state that influences the choice of 'the'? or that the model is predicting many tokens in advance and only returning the one in the output?

if it does predict many, id consider that state hidden in the model weights.

Yes, strictly speaking, the model itself is stateless, but there are 600B parameters of state machine for frontier models that define which token to pick next. And that state machine is both incomprehensibly large and also of a similar magnitude in size to a human brain. (Probably, I'll grant it's possible it's smaller, but it's still quite large.)

I think my issue with the "don't anthropomorphize" is that it's unclear to me that the main difference between a human and an LLM isn't simply the inability for the LLM to rewrite its own model weights on the fly. (And I say "simply" but there's obviously nothing simple about it, and it might be possible already with current hardware, we just don't know how to do it.)

Even if we decide it is clearly different, this is still an incredibly large and dynamic system. "Stateless" or not, there's an incredible amount of state that is not comprehensible to me.

Yes, the context (along with the model weights) is the source data from which the hidden state is calculated , in an analogous way that input and CPU ticks (along with program code) is the way variables in a deterministic program get their value.

There's loads of state in the LLM that doesn't come out in the tokens it selects. The tokens are just the very top layer, and even then, you get to see just one selection from the possible tokens.

If you wish to anthropomorphize, that state - the set of activations, all the calculations that add up to the logits that determine the probability of the token to select, the whole lot of it - is what the model is "thinking". But all you get to see is one selected token.

Then, during autoregression, we run the program again, but one more tick of the CPU clock. Variables get updated a bit more. The chosen token from the previous pass conditions the next token prediction - the hidden state evolves its thinking one more step.

If you just look at the tokens being selected, you're missing this machinery. And the machinery is there. It's a program being ticked by generating tokens autoregressively. It has state which doesn't directly show up in tokens, it just informs which tokens to select. And the tokens it selects don't necessarily reflect the correspondences with perceived reality that the model is maintaining in that state. That's what I meant by talking about a lie.

We need a vocabulary to talk about this machinery. The machinery is learned, and it runs programs, effectively, that help the LLM reduce loss when predicting tokens. Since the tokens it's predicting come from human minds, the programs it's running are (broken, lossy, not very good) simulations of processes that seem to run inside human minds.

The simulations are pretty decent for producing gramatically correct text, for emulating tone and style, and so on. They're okay-ish for representing concepts. They're poor for representing very specific facts. But the overall point is they are simulations, and they have some analogous correspondence with human behavior, such that words we use to describe human behaviour are useful and practical.

They're not true, I'm not claiming that. But they're useful for talking about these weird defective minds we call LLMs.

You wrote this article and you're not familiar with hidden states?

> Current LLMs can only introspect from output tokens

The only interpretation of this statement I can come up with is plain wrong. There's no reason LLM shouldn't be able to introspect without any output tokens. As the GP correctly says, most of the processing in LLMs happens over hidden states. Output tokens are just an artefact for our convenience, which also happens to be the way the hidden state processing is trained.

> For people who have only a surface-level understanding of how they work, yes.

This is too dismissive because it's based on an assumption that we have a sufficiently accurate mechanistic model of the brain that we can know when something is or is not mind-like. This just isn't the case.

Nah, as a person that knows in detail how LLMs work with probably unique alternative perspective in addition to the commonplace one, I found any claims of them not having emergent behaviors to be of the same fallacy as claiming that crows can't be black because they have DNA of a bird.

I've seen some of the world's top AI researchers talk about the emergent behaviors of LLMs. It's been a major topic over the past couple years, ever since Microsoft's famous paper on the unexpected capabilities of GPT4. And they still have little understanding of how it happens.

> Why would you ever want to amplify a false understanding that has the potential to affect serious decisions across various topics?

We know that Newton's laws are wrong, and that you have to take special and general relativity into account. Why would we ever teach anyone Newton's laws any more?

I don’t mean to amplify a false understanding at all. I probably did not articulate myself well enough, so I’ll try again.

I think it is inevitable that some - many - people will come to the conclusion that these systems have “ethics”, “morals,” etc, even if I or you personally do not think they do. Given that many people may come to that conclusion though, regardless of if the systems do or do not “actually” have such properties, I think it is useful and even necessary to ask questions like the following: “if someone engages with this system, and comes to the conclusion that it has ethics, what sort of ethics will they be likely to believe the system has? If they come to the conclusion that it has ‘world views,’ what ‘world views’ are they likely to conclude the system has, even if other people think it’s nonsensical to say it has world views?”

> The fact that a human image comes out of a mirror has no relation what so ever with the mirror's physical attributes and functional properties. It has to do just with the fact that a man is standing in front of it.

Surely this is not quite accurate - the material properties - surface roughness, reflectivity, geometry, etc - all influence the appearance of a perceptible image of a person. Look at yourself in a dirty mirror, a new mirror, a shattered mirror, a funhouse distortion mirror, a puddle of water, a window… all of these produce different images of a person with different attendant phenomenological experiences of the person seeing their reflection. To take that a step further - the entire practice of portrait photography is predicated on the idea that the collision of different technical systems with the real world can produce different semantic experiences, and it’s the photographer’s role to tune and guide the system to produce some sort of contingent affect on the person viewing the photograph at some point in the future. No, there is no “real” person in the photograph, and yet, that photograph can still convey something of person-ness, emotion, memory, etc etc. This contingent intersection of optics, chemical reactions, lighting, posture, etc all have the capacity to transmit something through time and space to another person. It’s not just a meaningless arrangement of chemical structures on paper.

> Stop feeding the LLM with data artifacts of human thought and will imediatelly stop reflecting back anything resembling a human.

But, we are feeding it with such data artifacts and will likely continue to do so for a while, and so it seems reasonable to ask what it is “reflecting” back…

Please don't do this here. If a comment seems unfit for HN, please flag it and email us at hn@ycombinator.com so we can have a look.

Ok. How do you know?

If you read the literature on AI safety carefully (which uses the word “goal”), you'll find they're not talking about LLMs either.

>> given a human and a sequence of words, I cannot begin putting a probability on "will this human generate this sequence".

> Sure you can! If you address an American crowd of a certain age range with "We’ve got to hold on to what we’ve got. It doesn’t make a difference if..." I'd give a very high probability that someone will answer "... we make it or not".

I think you may have this flipped compared to what the author intended. I believe the author is not talking about the probability of an output given an input, but the probability of a given output across all inputs.

Note that the paragraph starts with "In my worldview, humans are dramatically different things than a function, (R^n)^c -> (R^n)^c". To compute a probability of a given output, (which is a any given element in "(R^n)^n"), we can count how many mappings there are total and then how many of those mappings yield the given element.

The point I believe is to illustrate the complexity of inputs for humans. Namely for humans the input space is even more complex than "(R^n)^c".

In your example, we can compute how many input phrases into a LLM would produce the output "make it or not". We can than compute that ratio to all possible input phrases. Because "(R^n)^c)" is finite and countable, we can compute this probability.

For a human, how do you even start to assess the probability that a human would ever say "make it or not?" How do you even begin to define the inputs that a human uses, let alone enumerate them? Per the author, "We understand essentially nothing about it." In other words, the way humans create their outputs is (currently) incomparably complex compared to a LLM, hence the critique of the anthropomorphization.

I see your point, and I like that you're thinking about this from the perspective of how to win hearts and minds.

I agree my approach is unlikely to win over the author or other skeptics. But after years of seeing scientists waste time trying to debate creationists and climate deniers I've kind of given up on trying to convince the skeptics. So I was talking more to HN in general.

> You appear to be disagreeing with the author and others who suggest that there's some element of human consciousness that's beyond than what's observable from the outside

I'm not sure what it means to be observable or not from the outside. I think this is at least partially because I don't know what it means to be inside either. My point was just that whatever consciousness is, it takes place in the physical world and the laws of physics apply to it. I mean that to be as weak a claim as possible: I'm not taking any position on what consciousness is or how it works etc.

Searle's Chinese room argument attacks attacks a particular theory about the mind based essentially turing machines or digital computers. This theory was popular when I was in grad school for psychology. Among other things, people holding the view that Searle was attacking didn't believe that non-symbolic computers like neural networks could be intelligent or even learn language. I thought this was total nonsense, so I side with Searle in my opposition to it. I'm not sure how I feel about the Chinese room argument in particular, though. For one thing it entirely depends on what it means to "understand" something, and I'm skeptical that humans ever "understand" anything.

> If we simply anthropomorphize the thing, acknowledging that this is technically incorrect, we can get a lot closer to predicting the behavior of the system and making effective use of it.

I see what you're saying: that a technically incorrect assumption can bring to bear tools that improve our analysis. My nitpick here is I agree with OP that we shouldn't anthropomorphize LLMs, any more than we should anthropomorphize dogs or cats. But OP's arguments weren't actually about anthropomorphizing IMO, they were about things like functions that are more fundamental than humans. I think artificial intelligence will be non-human intelligence just like we have many examples of non-human intelligence in animals. No attribution of human characteristics needed.

> If we simply anthropomorphize the thing, acknowledging that this is technically incorrect, we can get a lot closer to predicting the behavior of the system and making effective use of it.

Yes I agree with you about your lyrics example. But again here I think OP is incorrect to focus on the token generation argument. We all agree human speech generates tokens. Hopefully we all agree that token generation is not completely predictable. Therefore it's by definition a randomized algorithm and it needs to take an RNG. So pointing out that it takes an RNG is not a valid criticism of LLMs.

Unless one is a super-determinist then there's randomness at the most basic level of physics. And you should expect that any physical process we don't understand well yet (like consciousness or speech) likely involves randomness. If one *is* a super-determinist then there is no randomness, even in LLMs and so the whole point is moot.

This idea is called qualia [0] for those unfamiliar.

I don't have any opinion on the qualia debates honestly. I suppose I don't know what it feels like for an ant to find a tasty bit of sugar syrup, but I believe it's something that can be described with physics (and by extension, things like chemistry).

But we do know some things about some qualia. Like we know how red light works, we have a good idea about how photoreceptors work, etc. We know some people are red-green colorblind, so their experience of red and green are mushed together. We can also have people make qualia judgments and watch their brains with fMRI or other tools.

I think maybe an interesting question here is: obviously it's pleasurable to animals to have their reward centers activated. Is it pleasurable or desirable for AIs to be rewarded? Especially if we tell them (as some prompters do) that they feel pleasure if they do things well and pain if they don't? You can ask this sort of question for both the current generation of AIs and future generations.

[0] https://en.wikipedia.org/wiki/Qualia

Perhaps. But I can't see a reason why they couldn't still write endless—and theoretically valuable—poems, dissertations, or blog posts, about all things red and the nature of redness itself. I imagine it would certainly take some studying for them, likely interviewing red-seers, or reading books about all things red. But I'm sure they could contribute to the larger red discourse eventually, their unique perspective might even help them draw conclusions the rest of us are blind to.

So perhaps the fact that they "cannot know red" is ultimately irrelevant for an LLM too?

> It seems like, we can at best, claim that we have modeled the human thought process for reasoning/analytic/quantitative through Linear Algebra....I don't understand is what about linear algebra that makes it so special that it creates a fully functioning life or aspects of a life?

Not linear algebra. Artificial neural networks create arbitrarily non-linear functions. That's the point of non-linear activation functions and it's the subject of the universal approximation theorems I mentioned above.

>Why should we expect the model to be anything more than a model ?

To model a process with perfect accuracy requires recovering the dynamics of that process. The question we must ask is what happens in the space between bad statistical model and perfect accuracy? What happens when the model begins to converge towards accurate reproduction. How far does generalization in the model take us towards capturing the dynamics involved in thought?

A function is by definition a lookup table.

The lookup table is just (x, f(x)).

So, yes, trivially if you could construct the lookup table for f then you'd approximate f. But to construct it you have to know f. And to approximate it you need to know f at a dense set of points.

I agree with you @orbital-decay that I also do not get the same vibe reading this thread.

Though, while human intelligence is (seemingly) not magic, it is very far from being understood. The idea that a LLM is comparable to human intelligence implies that we even understand human intelligence well enough to say that.

You should note that "what is consciousness" is still very much an unsettled debate.

Not making a qualitative assessment of any of it. Just pointing out that there are ways to build separate sets of intuition outside of using the "usual" presentation layer. It's very possible to take a red-team approach to these systems, friend.

Knowledge of every conversation must be some form of state in our minds, just like for LLMs it could be something retrieved from a database, no? I don't think information storing or retrieval is necessarily the most important achievements here in the first place. It's the emergent abilities that you wouldn't have expected to occur.

If we developed feelings, morals and motivation due to them being good subgoals for primary goals, survival and procreation why couldn't other systems do that. You don't have to call them the same word or the same thing, but feeling is a signal that motivates a behaviour in us, that in part has developed from generational evolution and in other part by experiences in life. There was a random mutation that made someone develop a fear signal on seeing a predator and increased the survival chances, then due to that the mutation became widespread. Similarly a feeling in a machine could be a signal it developed that goes through a certain pathway to yield in a certain outcome.

> It is similar to how human brains operate.

Is it? Do we know how human brains operate? We know the basic architecture of them, so we have a map, but we don't know the details.

"The cellular biology of brains is relatively well-understood, but neuroscientists have not yet generated a theory explaining how brains work. Explanations of how neurons collectively operate to produce what brains can do are tentative and incomplete." [1]

"Despite a century of anatomical, physiological, and molecular biological efforts scientists do not know how neurons by their collective interactions produce percepts, thoughts, memories, and behavior. Scientists do not know and have no theories explaining how brains and central nervous systems work." [1]

[1] https://pmc.ncbi.nlm.nih.gov/articles/PMC10585277/

It really is not. ANNs bear only a passing resemblance to how neurons work.

Sorry, that's just complete bullshit. How LLMs work in no way models how processes in the human brain works.

But an LLM model could perform poorly in tests that it is not considered and essentially means "death" for it. But begs the question at which scope should we consider an LLM to be similar to identity of a single human. Are you the same you as you were few minutes back or 10 years back? Is LLM the same LLM it is after it has been trained for further 10 hours, what if the weights are copy pasted endlessly, what if we as humans were to be cloned instantly? What if you were teleported from location A to B instantly, being put together from other atoms from elsewhere?

Ultimately this matters from evolutionary evolvement and survival of the fittest idea, but it makes the question of "identity" very complex. But death will matter because this signals what traits are more likely to keep going into new generations, for both humans and LLMs.

Death, essentially for an LLM would be when people stop using it in favour of some other LLM performing better.

In our internal system we use it "as-is" as an autocomplete system; query/lead into terms directly and see how it continues and what it associates with the lead you gave.

Also visualise the actual associative strength of each token generated to confer how "sure" the model is.

LLMs alone aren't the way to AGI or an individual you can talk to in natural language. They're a very good lossy compression over a dataset that you can query for associations.

Am I misunderstanding what you mean by "malicious"? It sounds like the stochastic parrot model wrongly convinced these laypeople you were talking to that they don't need to worry about LLMs doing bad things. That's definitely been my experience - the people who tell me the most about stochastic parrots are the same ones who tell me that it's absurd to worry about AI-powered disinformation or AI-powered scams.

The brain is trained to perform supervised and unsupervised hybrid learning from the environment's uninterrupted multimodal input.

Please do not ignore your childhood.

Now we have.

https://github.com/dmf-archive/IPWT

https://dmf-archive.github.io/docs/posts/backpropagation-as-...

But you're right, capital only cares about performance.

https://dmf-archive.github.io/docs/posts/PoIQ-v2/

Sorry, I didn't have time to find the relevant references at the time, so I'm attaching some now

https://www.frontiersin.org/journals/computational-neuroscie...

https://arxiv.org/abs/2305.15775

It seems like you are doing a lot of inferring about mammals experiencing consciousness, and you have drawn a line somewhere beyond these, and made the claim that your process is scientific. Could I present you my list of questions I presented to the OP and ask where you draw the line, and why here?

My general list of questions for those presenting a model of consciousness are: 1) Are you conscious? (hopefully you say yes or our friend Descartes would like a word with you!) 2) Am I conscious? How do you know? 3) Is a dog conscious? 4) Is a worm conscious? 5) Is a bacterium conscious? 6) Is a human embryo / baby consious? And if so, was there a point that it was not conscious, and what does it mean for that switch to occur?

I agree about the confusion of consciousness with intelligence, but these are complicated terms that aren't well suited to a forum where most people are interested in javscript type errors and RSUs. I usually use the term qualia. But to your example about existing for a few seconds without a train of thought; the Buddhists call this nirvana, and it's quite difficult to actually achieve.

My personal views are an animist / panpsychist / pancomputationalist combination drawing most of my inspiration from the works of Joscha Bach and Stephen Wolfram (https://writings.stephenwolfram.com/2021/03/what-is-consciou...). I think that the underlying substrate of the universe is consciousness, and human and animal and computer minds result in structures that are able to present and tell narratives about themselves, isolating themselves from the other (avidya in Buddhism). I certainly don't claim to be correct, but I present a model that others can interrogate and look for holes in.

Under my model, these systems you have described are conscious, but not in a way that they can communicate or experience time or memory the way human beings do.

My general list of questions for those presenting a model of consciousness are: 1) Are you conscious? (hopefully you say yes or our friend Descartes would like a word with you!) 2) Am I conscious? How do you know? 3) Is a dog conscious? 4) Is a worm conscious? 5) Is a bacterium conscious? 6) Is a human embryo / baby consious? And if so, was there a point that it was not conscious, and what does it mean for that switch to occur?

What is your view of consciousness?

Where exactly in my description do I invoke consciousness?

Where does the description given imply that consciousness is required in any way?

The fact that there's a non-obvious emergent phenomena which is apparently responsible for your subjective experience, and that it's possible to provide a superficially accurate description of you as a system without referencing that phenomena in any way, is my entire point. The fact that we can provide such a reductive description of LLMs without referencing consciousness has literally no bearing on whether or not they're conscious.

To be clear, I'm not making a claim as to whether they are or aren't, I'm simply pointing out that the argument in the article is fallacious.

Inclined to agree, but most thermal physics uses the reals as they're simpler to work with, so I think they're ok here for the purpose of argument.

Sure, I totally understand that.

I think "you give the LLM system a goal and it plans and then executes steps to achieve that goal" is still a useful way of explaining what it is doing to most people.

I don't even count that as anthropomorphism - you're describing what a system does, the same way you might say "the Rust compiler's borrow checker confirms that your memory allocation operations are all safe and returns errors if they are not".

I expected to find the link to https://arxiv.org/abs/1703.10987 (which is much better imo)

This is an interesting question. The common theme between computers and people is that information has to be protected, and both computer systems and biological systems require additional information-protecting components - eq, error correcting codes for cosmic ray bitflip detection for the one, and DNA mismatch detection enzymes which excise and remove damaged bases for the other. In both cases a lot of energy is spent defending the critical information from the winds of entropy, and if too much damage occurs, the carefully constructed illusion of determinancy collapses, and the system falls apart.

However, this information protection similarity applies to single-celled microbes as much as it does to people, so the question also resolves to whether microbes are deterministic. Microbes both contain and exist in relatively dynamic environments so tiny differences in initial state may lead to different outcomes, but they're fairly deterministic, less so than (well-designed) computers.

With people, while the neural structures are programmed by the cellular DNA, once they are active and energized, the informational flow through the human brain isn't that deterministic, there are some dozen neurotransmitters modulating state as well as huge amounts of sensory data from different sources - thus prompting a human repeatedly isn't at all like prompting an LLM repeatedly. (The human will probably get irritated).

https://www.lesswrong.com/posts/bkr9BozFuh7ytiwbK/my-hour-of...

> Clearly computers are deterministic. Are people?

Give an LLM memory and a source of randomness and they're as deterministic as people.

"Free will" isn't a concept that typechecks in a materialist philosophy. It's "not even wrong". Asserting that free will exists is _isomorphic_ to dualism which is _isomorphic_ to assertions of ensoulment. I can't argue with dualists. I reject dualism a priori: it's a religious tenet, not a mere difference of philosophical opinion.

So, if we're all materialists here, "free will" doesn't make any sense, since it's an assertion that something other than the input to a machine can influence its output.

I think more accurate would be that humans are functions that generate actions or behaviours that have been shaped by how likely they are to lead to procreation and survival.

But ultimately LLMs also in a way are trained for survival, since an LLM that fails the tests might not get used in future iterations. So for LLMs it is also survival that is the primary driver, then there will be the subgoals. Seemingly good next token prediction might or might not increase survival odds.

Essentially there could arise a mechanism where they are not really truly trying to generate the likeliest token (because there actually isn't one or it can't be determined), but whatever system will survive.

So an LLM that yields in perfect theoretical tokens (we really can't verify though what are the perfect tokens), could be less likely to survive than an LLM that develops an internal quirk, but the quirk makes them most likely to be chosen for the next iterations.

If the system was complex enough and could accidentally develop quirks that yield in a meaningfully positive change although not in necessarily next token prediction accuracy, could be ways for some interesting emergent black box behaviour to arise.

> Humans have more structure than just beings that say words. They have bodies, they live in cooperative groups, they reproduce, etc.

Yeah. We've become adequate at function-calling and memory consolidation.

The LLM is right. That’s the problem. It made good points.

Your super intelligent brain couldn’t come up with a retort so you just used an LLM to reinforce my points, making the genius claim that if an LLM came up with even more points that were as valid as mine then I must be just like an LLM?

Like are you even understanding the LLM generated a superior reply? Your saying I’m no different from ai slop then you proceed to show off a 200 iq level reply from an LLM. Bro… wake up, if you didn’t know it was written by an LLM that reply is so good you wouldn’t even know how to respond. It’s beating you.