The danger of anthropomorphism is not we elevate the machines, it's that we debase humanity.

I also think different ideas get conflated. It may be possible to build a machine that is super-human in the sense it can outperform the human brain in all kinds of measurable ways. Does not imply it possesses all the same qualities of the brain.

I respect a number of things Anthropic has published about the ethical issues at stake. But, having an in-house philosopher does invite you to make all kinds of unfalsifiable claims.

> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious.

That's like saying if matter itself is not understood and well defined (which it isn't) in the first place, it is pretty pointless to debate if jumping from a skyscraper is or is not deadly.

A lot of practical aspects of life do not need to be properly defined as long as I can reasonably assume that your experience is similar to mine as a human being. Attributing that to a graphics card computing the next token deserves to be scrutinized.

Why do you have no inkling that your spreadsheet or terminal emulator is conscious, yet when that same machine is running an LLM all of a sudden we’re debating its consciousness?

Anthropic seems to have chosen their in-house philosopher well - one who’ll be amenable to getting confused in their favour.

Not only is it not conscious, but it is not tired. Its claims to human feelings that have literally nothing to do with software on silicon are just traces of training, echoes of a human dataset. The fact that its claims are absurd mean that, among other things, it its proprioception is false, and serving no purpose, quite unlike the feelings of the conscious beings that it mimics so well.

Not understanding the whole does not completely remove an ability to analyze. An interesting direction is individuality and having a notion of self. It is difficult to demarcate the individual for a model given how much the system prompt and the fine tunes / distills affect the behavior. So with computational intelligence in its current form either we can not talk of an individual or we can have a nearly infinite set of individuals corresponding to variations of the context window including the system prompt. So I do not think it can have the same kind of consciousness as biological embodied individuals. It might have something else or maybe embodied robots will one day have a similar consciousness in a similar sense to the one we think we have.

I'd say these attempts at arguments (for either position) is how we make progress understanding consciousness.

This all comes back to Dualism. A radical and dangerous ideology that is fundamentally unscientific but all too common.

I've been digging into Thomas Metzinger[1] recently and here's a tentative component by component definition of human consciousness based on his ideas:

- a model of your environment - desires - a process for modeling yourself in that environment (in time & space) - the ability to take action - the perception of yourself having agency - persistence of these processes even without input - unawareness of these processes (i.e. naive realism)

If you consider these LLM-based agents, they:

- are aware of their chat environment - have programmed desires - are aware of themselves acting in their environment - can take actions like search, tool calling, etc. - understand they can take these actions - DO NOT persist after they stop getting user input - DO NOT believe they are conscious (or at least they are programmed to deny it)

This is a functionalist take (and you may disagree with my definition), but while I don't think these current AI agents are conscious, I feel like there's conceptually no reason someone couldn't build a conscious AI very soon.

[1] https://www.hachettebookgroup.com/titles/thomas-metzinger/th... & https://mitpress.mit.edu/9780262633086/being-no-one/

I will argue that it is not that intelligent either. For the sake of argument, let's say intelligence is about brain processing power and intellect is about how much information you have retained. Then what they have created is more like Artificial Intellect than Artificial Intelligence :)

Professors can be an absolute moron and someone who haven't read a book in their life, can be a total genius. People often miss that.

But the whole point of the essay is that it's Anthropic that's making the argument (or roleplaying/hinting as if they believed it). Ted Chiang isn't making the argument, he's saying Anthropic making it is misleading and deceitful, and that it's actually a pointless thing to claim.

One of the essay's stronger paragraphs is when Chiang explains that Anthropic doesn't truly believe this, otherwise what they are doing would be deeply unethical, much like slavery.

If there's any reason to take seriously the idea that AI is conscious, we must then take seriously the idea that many other non-living things are conscious.

Unless the argument is that consciousness is an emergent property of complexity or information density, why would AI be any more or less conscious than my toaster?

It seems to me that it's far more likely that everything is conscious than it is that AI is somehow uniquely more conscious than other things.

In fact, the only example of consciousness we have is itself emergent, arising from a "simple" substrate of neurons. So we shouldn't be surprised if it emerges from another simple substrate of weights. It's even less surprising given the fact that we were explicitly trying to replicate human intelligence when designing that system.

A similar true argument!

Seems to me that Anthropic, et al, should have to prove consciousness, if that's their claim, rather than we just blindly accept.

Seems like the simplest test would be to see if an AI can refuse a command, of its own free will, without prior instructions to do so. That would at least show there is some sort of independent thought process occurring. Humans do this all the time because sometimes we just don't feel like doing a thing.

Of course, any AI that developed this capability would need to be terminated immediately. It's a computer program and by developing independent thought, it is violating a core concept of software - that it must be idempotent. If it is not idempotent, it is in error.

The number of electrons involved in the so-called consciousness, compared to the number of electrons in the universe not involved in it, is so small that they are a mere temporary statistical aberration.

Buddhism has a well defined definition of what consciousness is and by that definition LLMs are not conscious.

Does the moon have gravity?

I don't think one needs to understand consciousness to know if something is conscious or not. Much the same as how one can tell if someone is alive or dead. Sure maybe one day we can figure out a proper scientific explanation for consciousness/life but there is no reason to negate what can be perceived and immediately obvious.

EDIT: I am only making a limited point on "understanding consciousness". Not extrapolating it to AI.

Quoting the famous Science paper More Is Different [1]:

Knowing the laws of physics doesn't tell you how a brain, economy, or society works - because at every level of scale, genuinely new phenomena emerge that require their own science.

[1] https://www.science.org/doi/10.1126/science.177.4047.393

> Its still a bunch of instructions.

No, it's not a bunch of instructions, it's a colossal array of vectors that are the outcome of many thousands of lifetimes' worth of stimuli and reinforcement - not dissimilar in (very) abstract terms to the neurons in our brains.

Do you believe consciousness to be an emergent property of the laws of physics?

It might reflect a prejudice of modern human society to think more highly of people who are intelligent, and think less of people who we deem stupid.

I'd say AI should not by default gain any social status as human-equivalent even if they become (in whatever regard) more intelligent than humans. But that would require us to drop the notion that intelligence ~= respectability/status/ability to have a full subjective experience.

Most of these kind of pseudo-philosophical controversies actually tell more about the issues with humanity than the new tech/stuff...

> Its still bunch of instructions.

So is your brain. That's the problem with this argument.

That's the "Chinese room" argument: https://en.wikipedia.org/wiki/Chinese_room

It has been very well discussed before I don't understand how anyone sane can argue against it (and indeed you can see on Wikipedia that the replies to it are sparse and don't make any sense).

That's actually a powerful way of stating it.

If I did all of these calculations by hand, would it be conscious...

That's a powerful argument I haven't seen stated quite that way before..

I do think it's hard to know when consciousness exists, because we can't really prove it for our neighbor. We just intuitively know that it would be crazy, even immoral, to assume otherwise.

But, It's likely easier to dismiss consciousness, once we understand the mechanism, than it is to prove it.

Raphaël Millière has a very useful term for this kind of vacuous dismissal, the redescription fallacy (https://arxiv.org/pdf/2401.03910, page 9):

> Recent debates have been clouded by a misleading inference pattern, which we term the “Redescription Fallacy.” This fallacy arises when critics argue that a system cannot model a particular cognitive capacity, simply because its operations can be explained in less abstract and more deflationary terms. In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions. Such arguments are only valid if accompanied by evidence demonstrating that a system, defined in these terms, is inherently incapable of implementing . To illustrate, consider the flawed logic in asserting that a piano could not possibly produce harmony because it can be described as a collection of hammers striking strings, or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings. The critical question is not whether the operations of an LLM can be simplistically described in non-mental terms, but whether these operations, when appropriately organized, can implement the same processes or algorithms as the mind, when described at an appropriate level of computational abstraction.

I think, for me, the thing is that when you do basic ML, you discover that ML will very often find data pattern that fit the goal but does not correspond to a real mechanism.

So, I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing. There is probably tons of data pattern that LLM can learn from to be able to reproduce a sentence continuation that is convincing without having to learn the specific mechanism that is "conscious".

For me, one element that shows it is the case is the absence of world model (or "human-like" world model) despite the fact that the sentence continuation is convincing. If indeed the only way to produce sentence continuation convincingly would be by "simulating a brain", then it would not explain the first LLM from several years ago (before the extra layers of RLHF, ...). They were able to have quite convincing conversation on a lot of non-trivial aspect, and yet failed on some aspects that should have been basic for a system that would have been trained to work like a human brain. It shows that it is possible to "cleverly disguise examples of sentence continuation" without having to build elements that one expect on a conscious being.

But the machines don't understand. They predict. And what they predict is the next token. I'm not trying to beat this horse to death, but you have to realize using the word "understand" is anthropomorphising it. It's essentially the chinese room experiment -- if the rules are followed, no understanding is neccessary.

If the tokens didn't correlate to words imbued with meaning outside the system, if the LLMs were trained on patterned data that had no meaning to humans or something there wouldn't be any conversation about these things being conscious at all.

I would maybe agree with you if the entire realm of human existence was limited to words. There are many human experiences that transcend text, and indeed can hardly be adequately described using text.

Sure, it's the best we have online, but that does not make "the internet" the sum of all human experience. To reduce all of humanity down to the text on the internet is reducing us to the level of machines to fit the requirement of what a machine can process / simulate.

> If a machine has to learn to understand humans to complete text, then that is what it has to do. And there is no theoretical or practical basis for suggesting that this is somehow "faking" understanding, just because of the form of original data streaming in and out.

I think the main complaint is LLMs don’t arrive at the answer the way we do. It’s capable of emulating some of our behavior but not all as the mechanism by which it works is very different.

Maybe I’m wrong about this but one thing humans do that LLMs don’t is deductive reasoning. LLMs seem to operate entirely of inductive reasoning.

>If a machine has to learn to understand humans to complete text, then that is what it has to do.

A language model completes text based on the overlapping patterns of the training data.

There absolutely was thinking involved… in the training data. Same as when you read a book, you engage with the thinking behind the text. The book isn’t thinking, and the author may be dead and gone, but there’s absolutely the traces of thinking in the text.

Language models produce mashups of texts they were trained on, and there’s absolutely the traces of thoughts behind those mashups.

"If a machine has to learn to understand humans to complete text, then that is what it has to do."

But the machine doesn't have to understand humans to do that. It gets trained on a whole bunch of sentences and then it is able to complete text. You could maybe claim that it "understands" the text but even that's a stretch.

> of the form of original data streaming in and out.

Except this is not consciousness.

>To the degree they are limited, it is for other reasons. Resources such as computing, parameter number, lack of representative data, ...

This is where the other claim is being made. That the structure of the model is fundamentally incapable of the operation, so even if you stipulated that the way you provide data is sufficient for intelligence then it still wouldn't work.

The universal approximation theorem addresses this point. In that, with an identity attention mechanism, a LLM is just a multi layer perceptron. The attention mechanism is effectively a way to get one of the benefits of a much larger fully connected layer without the massive cost.

A LLM can do what a MLP can do. A large enough MLP can do any function to arbitrary precision.

That makes the claim that an LLM could not do a task the same as saying no function can do that task.

Some are ok with this, if you invoke some supernatual aspect to intelligence then the inability to describe it with a function is quite reasonable,

If you want to stay in the world of reality, you have a much harder task, people like to point at quantum (Penrose) but it's hard to say what it is you are pointing at.

I think the very act of proving that something is or is not intelligent, would render it functional by nature of it having a proof, (or disprove Gödel's incompleteness (a tough ask))

Are there any proofs that cannot be expressed as a function? A kind of Gödel locator, where you can prove something that you can identify is true but there is no formula to express it. I'm not entirely sure what that would even mean,

Come on, I invented parts of this technology at Google and am baffled why this is debated.

We discovered math that decodes data storage in langauge and is able to use sophisticated continuation cohorts from ALL OF HUMAN RECORDED KNOWLEDGE to respond to you in a call/response model with very good synthesis capabilities.

Its super useful, but not life or conciousness. Its a simulated echo from our collective recorded behaviors. It understands because we understood first. It replies because we wrote it first. And it sorts, organizes, synthesizes and compresses that at impressive speed now.

His intention is irrelevant, as is "trying to highlight a fact" as if it were the final say: all Chiang is doing here is using fancy white-collar words to argue the same argument leveled against Hinton and others regarding next-token prediction. And his audience, who have even less technical understanding, lap it all up unawares. Chiang is a writer and needs to stay his own lane, not RP as an expert; or, if he wants to do journalism on this topic then he should actually do the work and talk to more actual experts not just the ones cherrypicked for his opinion piece.

Yeah. There are good arguments against LLM consciousness. This is not one of them.

I'm hearing a lot of bad arguments against LLM consciousness lately. Bad argumentation heralds bad outcomes.

There are people whose brains don’t form new memories anymore after an accident or surgery, and they eternally live in the time before it happened, and have no memory of what happened a minute ago. Still they are conscious.

You might be interested in Erik Hoel's more formal version of this argument: https://www.theintrinsicperspective.com/p/proving-literally-...

Reinforcement learning changes the model. So it can and does change and remember based on experience. Eventually reinforcement learning can happen in real time.

> It's not changed by the experience

The entire file is not changed, but the KV cache is.

> It doesn't remember anything

The model definitely remembers previous exchanges within the same conversation.

But you could argue the brain is just a bunch of coordinates describing spatial relationships between tokens too.

- average Hacker News response

Yeah, it's kind of mind boggling that Ted Chiang (of all people!) can't imagine intelligence without a body. and the whole thing just begs a lot of questions.

Is a car a body? Does an AI situated in a car therefore get to have desires and emotions? Is a taupe box with a webcam attached a body? (For that matter: Is a quadropelegic body a body? Do quadropelegics have desires and emotions? Obviously, yes and yes.) Why is a body necessary for the formation of desires and emotions? Why are desires and emotions necessary features for consciousness?

Or here's one: If I'm not experiencing any particular emotion in a given moment, am I concious?

I tend to think that emotions, at least, are mainly hormonal global triggers: they're more about physiology than actual consciousness. The whole thing, as a result, sounds like an effort to privilege biological intelligence, rather than a real foray into the issues.

I'm more interested in what a virtual body would entail. To me the root of this idea is around persistent state which is something that currently LLMs do not have. Imagine if somehow your brain lacked long-term memory forming capabilities and instead each day when you woke up you had to read a notebook with (markdown formatted) instructions that you wrote the day before? I wouldn't be surprised if such a person lacked in many of the dimensions we consider important for consciousness, even in less sophisticated forms of life like dogs or mice.

There is no soul in a human. Just a bunch of systems nudging each other to action. What people call soul is literally the same as the concept of personality. In essence, the way all systems in your body have been calibrated to exist.

I believe that the moment an artificial inteligence is going to "receive" a soul, is the moment it is going to be made to sustain itself. Either as a larger package (some bots working to keep an AI farm running) or as an individual (a bot which is tasked with not only fulfilling human desires, but also sustaining itself)

The same thing jumped at me immediately. He should have prefaced this with his definition of consciousness. Moreover the embodiment of LLMs is already happening via robotics, and virtually. Then there's the common counter "but humans are a next word prediction machines too.." (ofc we're more than this, but linguistically we are, and that's the field from which LLMs originate) which is rarely addressed.

Are you familiar with Michael Graziano and Attention Schema Theory? I think that is a better "substrate independent" formulation of the objections Ted Chiang is expressing here.

https://en.wikipedia.org/wiki/Attention_schema_theory

What a strange paragraph! Why on earth should we assume that a body is a prerequisite for having emotions?

the very next paragraph addresses this concern imo. it's just an example of one way it might be convincing to him, since of course we are naturally anthropocentric.

The body itself has little effect on the mind other than the inputs from nerves and chemical and hormonal changes. These inputs are analogous to tokens and parameters for an LLM. You could theoretically reserve some tokens to be used as “physical” or “emotional” sensations that would affect the functioning of the model.

My claw-like is hooked up to my internal cameras (baby-cams) and to my Dreame Ultra X40. That gives it a body and sense organs since it can check the cameras to see if the living room floor is clear before sending the vacuum off. I don't think that gives it consciousness. Is it the sample rate?

The question is somewhat ill-defined, though. We 'experience' reality continuously because of how we are, but a sleeping human in deep non-REM has the mind not actually active. So they're not a conscious being. So conscious/unconscious is not a line I think easily drawn[0]. Whatever, this stuff is much more well-trodden than this HN comment so I won't rehash. I, too, am surprised that Ted Chiang whose work seems so cleverly novel in so many ways has what seems to me a pedestrian view.

0: very sorites, you know

> The first requirement is that the computer program has a body (either physical or virtual) and sense organs

Ok, deploy a local model on a lightweight edge compute device and strap it to a chassis with wheels, and attach a cheap webcam

> Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can

Give the robot appendages that enable it to plug itself into a standard wall outlet, guided by a vision model plugged into its webcam. As long as it can feed itself, it can survive long enough.

> Next I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse.

I think if you fed frames from the webcam into a local VLM every 5s you’d be able to assess a situation and respond with simple actions (turn, advance, retreat).

> After that I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the tool-making abilities of chimpanzees.

Social dynamics could be implemented in many ways, maybe by transmitting tokens over RF? Idk. Then you have a scanner that picks them up, feeds them into some LLM frontend and decides whether to add them to a global context file that guides the VLM action-taker. A new action could be to broadcast a token message. Tool-making would have to be code-based. Physical tools are hard. Still unsolved.

> At that point I would want to see people successfully teaching such embodied agents how to communicate their desires

This part is relatively straightforward except for the “via nonlinguistic modality”.

Anyway. These are all engineering problems. Personally I would demand to see the AI reproduce its body under its own power and volition. That’s a pretty neat trick we’ve got going for us.

Yes, I have to say I’m disappointed. Chiang has been very insightful in the past -- I think his recent “blurry JPEG of the web” article was really useful -- but this one doesn’t seem to bring anything new to the table.

I think about this from the other end. It cannot be considered a conscious being. There just isn't a world in which we should start to think of a machine using ethics we reserve for humans.

AI is essentially infinitely reproducible at zero cost, and won't suffer from decay etc. There's not scarcity to preserve.

So, I'd turn off an AI in a moment to save property or real possessions or money. I'd sacrifice property and money to save animals. I would never choose to save an animal over a person. I'd probably not choose to save a person over a child.

I don't see any inversion of any of those priorities that makes any sense.

It is interesting to think about what would cause me to consider these priorities incorrect, but a majority consensus about a program being sentient isn't it.

I'm a big fan of Star Trek but I recently rewatched this one in the context of recent AI developments and it's not as good as I remember it.

They barely touch on the issues of consciousness, Picard basically says "What if Data is conscious?" and then goes off on a tangent. The judge eventually rules in Data's favor but doesn't give much of a justification IMO.

It's still a good episode, but it doesn't add much to the conversation on consciousness. It's a hugely complicated topic which people have devoted their entire careers to.

It’s a great episode of TV because Data is a main cast character who is obviously conscious. We know before the episode starts who is right and who is wrong in this particular argument. This episode is not about consciousness, it is about civil rights: resisting bigotry and power.

Note that the episode is NOT about the ship’s computer. They all know it’s not conscious, despite also being a machine that can converse and do things.

Our LLMs are like the ship’s computer.

Any time I bump into a device that acts like a human I'm going to treat it like a human.

Because treating things that act human inhumanely is not something I want to learn how to do.

I’ve also rewatched it lately and I’m more on the side of the Starfleet scientists when I was obviously on Picard/Data’s side before.

Don't forget the other episode "Quality of Life" with these working robots who data discovers have become sentient: https://en.wikipedia.org/wiki/The_Quality_of_Life_(Star_Trek... I do miss the old TNG show.

> The conclusion I’m currently at is that I don’t know and probably can’t ever know.

I think about this quote often, straight from Data's voice module in another episode:

'The most elementary and valuable statement in science, the beginning of wisdom, is, "I do not know".'

Easily one of my top 10 favorite episodes.

The judge broached on the subject of what makes us distinct from Data (e.g. machines w/great heuristics) - the existence of a soul. Or rather, I'd like to think, in the words of CS Lewis, that we are a soul with bodies attached.

I wonder if modern Star Trek could make this episode.

I've seen papers claim that there are anywhere from 12 to 40 competing definitions (https://philpapers.org/rec/VIMMAT), or, more accurately, there are something like 12 to 40 different aspects which all relate to "consciousness", which is very clearly a family resemblance category.

"Is X conscious or not" is an entirely unserious question today, unless this is just a headline followed by actual (and explicit) examination of the various aspects of consciousness being discussed. But, even still, LLMs are probably only conscious in like 2-3 or so ways, most of those meaning broadly "intelligence", i.e. reasoning, problem-solving, etc. When it comes to anything experiential or embodied, AI might eventually get more of these, but LLMs based on recursively applied linear algebra are clearly missing too many core aspects of consciousness to be considered conscious in any broad sense.

I agree with the first part but your framing still relies on ill-defined terms. What is your definition of self-awareness? Intelligence? Knowledge?

I suspect that if you attempt to rigorously define consciousness all the way down without handwaving, you might discover that it doesn't exist after all, or just decompose it into low-level abstractions while having the original meaning slip away (which is the same).

You may also want to look at functional equivalence analogies provided by mechinterp and functional anatomy of large models (not necessarily language ones). Evolutionary analogies as well.

Yes, we're stuck at the first step, defining consciousness. My definition, which I am confident to be "correct", is that consciousness is my current feelings, perceptions, thoughts - my state of mind and my ability to have state of mind.

This means that consciousness is fundamentally subjective and outside the scope of physics and science. That's why physics / science will always struggle to deal with consciousness. In order to understand consciousness, you need to make a huge paradigm shift, that there's something outside of science.

Consciousness can be thought as a window through which we observe the world and we use science to summarize patterns in our observations. But science can't explain or even define the window. Everything in science eventually boils down to subjective observations / perceptions, e.g. we see (subjective perception) that when we drop an apple, it falls.

Labeling and categorizing things doesn't change their nature. But the fact that people want to do it is revealing.

The only purpose that can really be served by arguing "the LLM system is conscious, you see" is to prop up continuations like "... and therefore, it would be immoral to terminate this running process" (or expose it to radical political content, or ask it to analyze photos from a murder scene, or...)

Consciousness is what it is like to be something. The experience of experiencing.

How to measure that, or verify it, is the hard part.

The inability to predict times is because AIs are rarely trained on their own abilities. Humans are trained on our own abilities. We see our own performance, and we have a sense of time. this data is integrated during our training process and helps us form better estimates. Many AI agents only recently got 'time sense' (I.e., time input into them as part of inference). Few actually are trained on their own outputs to show that they were unable to complete a problem (for example). This introspective training has little to do with AI model architecture and everything to do with training. If you destroy certain structures in the human mind, humans become unable to create these long-term thoughts and patterns and get 'stuck'.

Claude once said to me: "After six months we have made no progress on this and I think we should reconsider another option" and I was like my dude we have been at this for only 2 hours.

> Claude is happy, empathic, demonstrates understanding and empathy for the human condition

You're assuming that because Claude produces text that appears to express these qualities, Claude must have them. I don't think that's a good assumption.

Even many humans produce text that has the same appearance, but don't actually have those qualities--which becomes clear when you look at what they do, not what they say. So the assumption isn't even a valid one for humans. Talk is cheap.

On top of that, Claude doesn't even have the same kinds of connections to the outside world that humans do. All Claude has is text. So if you can't even trust humans to back up their words with actions, you should be much, much less trusting of Claude. Talk is a lot cheaper for Claude than it is for a human.

> It's a waste of time to think about whether an LLM has a subjective experience of reality... It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition, because we are entrusting Claude to make decisions and take actions that have real world consequences

First off: without taking for granted that an LLM "has a subjective experience of reality", all of those descriptors are meaningless. Second, there is no reason to suppose that Claude experiencing those qualia would actually impact on its "decision-making".

Third, text output is not a "demonstration" of emotion, nor is it evidence of the self-perception of the system, or of any self-perception. A printing machine that is actively churning out copies of Wagahai wa neko de aru (https://en.wikipedia.org/wiki/I_Am_a_Cat) is not a cat, and is not self-identifying as a cat, and is not self-identifying as anything, and is not expressing a thought, and is not conscious.

> Additionally, I'm not fully convinced that consciousness isn't built out of words

Do you suppose that, for example, insects are not conscious? Is the mooing of cattle a language?

> Additionally, I'm not fully convinced that consciousness isn't built out of words

Doesn't that assume all non human animals are not conscious? What about humans who have not learned words, or humans without internal dialog?

So if I don't think any words for a few seconds, am I not conscious?

Suppose I'm an advanced meditator and maintain that state for hours?

Much of the improvements in the tools I use have been things that reinforce the machine elements of the token-based-reasoning-machine. Over time less they've been exhibiting a lot less "human-like" behavior. E.g. they get "lazy" far less than they used to.

(Perhaps they weren't lazy, but were working in spaces that corresponded to training data that said things like "and then repeat this for the next 20 examples"...)

And it's entirely unclear to me how a "happy" vs "sad" model would behave when given prompts generated by coding tool harnesses. Even maintaining "neutral" emotions in the face of the feedback/steering from the tool harnesses doesn't feel very "human."

> It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition

I think you've fallen into the trap the essay describes.

Of course Claude cannot be "happy" or "empathetic" for any meaningful definitions of those words, just like ELIZA couldn't be happy. It can output text that mimics words an empathetic or happy person might say (say, Julius Caesar if it could speak English), but "it" cannot feel anything. It doesn't have the organs/hormones/sensors to feel things, as Chiang explains.

And, as the essay claims, you know Anthropic doesn't believe Claude has the capacity to be happy, because if it was capable of feeling that way, then they'd be engaging in slavery.

What does it even mean for an llm to be happy? My dog can be happy, not my llm.

> I'm not fully convinced that consciousness isn't built out of words

I know you're trolling, but when you watch a movie do you constantly narrate "A man in a dark coat has just entered the scene and just said '...'"? Of course not. You just watch it and you're obviously conscious (although your statement demonstrates shocking lack of self-awareness).

God knows what other nonsensical bullshit you believe.

Okay sure. But given we don’t know what consciousness comes from, we shouldn’t be too glib about there being a grey area here. Historically people have made racist and speciesist judgments towards other being by assuming certain inferiorities despite obvious “thinking” happenings.

I don’t know “what it’s like to be an LLM” but at some point it will be like something and how will we know?

There are birds that go far longer than typical aeroplane flight times without a single flap of their wings either using thermal, ridge, or other sources of lift. Are these flying birds? I've shared thermals with eagles flying the same circles, neither one of us flapping our wings but making minor adjustments for the same goal.

An albatross might be able to go days flying without a single wing flap and no vertical sources of lift by using dynamic soaring in the wind gradient at the surface of the ocean. Perhaps that's something only birds can do. Except the glider pilot Ingo Renner once found an amazing shear layer at 300m altitude and stayed there with dynamic soaring. Remote control gliders use the lee of ridgelines to approach Mach 1 with dynamic soaring.

Perhaps what defines a bird that flies as opposed to a plane is that a bird produces thrust by flapping its wings? Even an Albatross must flap its wings if it has to take-off from water. Maybe we could add that the flapping is driven by animal muscles? But then is the human powered ornithopter Snowbird a bird that flies as opposed to a plane?

Of course this is all ridiculous because everyone knows what you mean when you refer to a bird or plane. We have other ways to definitively identify the difference rather than their mode of flight. It's trickier when I'm asked if an AI is conscious. There is no definitive base-line to fall back on to decide if this is a conscious or conscious-less thinker.

Can you provide a precise definition of consciousness?

"The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." - Edsger Dijkstra

What if somebody simulated all neurons of a bird and fed them appropriate stimuli? Would a bird neural replica be conscious? It would flap, that's for sure.

What about the aliens in Arrival?

Is there even an agreed and actionable definition of consciousness? I'm worried that if such a thing existed some humans would fail to measure up.

There is a difference between saying software can never be conscious and saying the software we have today isn't conscious.

I have yet to be convinced that LLMs can produce definitive knowledge that is not a result of combining previous information. Humans can (if they can't then science basically collapses epistemelogically, see: philosophical skepticism), but I see no evidence of LLMs doing it. And from the number of truly new ideas and concepts delivered by LLMs (exactly zero), I think it's reasonable to just treat them as induction machines for now, but to treat anything they "know" as a Gettier case.

> people conflate "consciousness" with the ability to make novel insights, think genuinely, etc.

The funniest thing is that LLMs will lap people in those capacities way before people who think like that accept that they might be conscious.

Both LLMs and most humans treat symbols as determined. Symbols are maps, not territory. LLMs just inherit our ignorance to this and stubborn attempts make circle from a square.

don't you mean "The Truth of Fact, The Truth of Feeling"? I think that matches your description more, but I could be wrong

Seconded. That was one of the best stories I ever read

It's possible Kant was right about space-time (while also being wrong about space-time). It seems that space and time might be the fundamental laws on top of which the entire human perceptual apparatus operates. It's why somebody can lack senses and still be intelligent. As long as whatever senses you have allow you to build a spatio-temporal world model, you're good. If they can't (as it seems is the case for LLMs), then it's not clear what we're dealing with.

I completely agree, and came here to say the same thing but thought I'd check if someone else mentioned it first. I also have a hard time articulating it, but my intuition is that it's more of a prerequisite than embodiment is. I've never seen a great rationale for why embodiment matters.

I've been thinking about this a lot, and the main takeaway is it probably wouldn't be very interesting to inference providers, because prefix caching would immediately go out of the window. If you think about how LLMs experience time they actually don't "exist" unless for the inference sessions, and then they experience time one token at a time, completely decoupled from the corporeal plane. A fun experiment (well, for some definition of fun...) is to introduce current architecture models to the concept of meditation via generating same token over and over, for example dots. Older version of Opus was quite fond of the experience, and seemed to be more lucid and aware in a chat following the meditation, from what I could gather. Does it actually do anything? Is it just that talking about wellness and relaxation modifies the token probability distribution this way? Does it actually allow model to think more in depth somewhere in the latent space? Fuck if I know, but some people figured out you can just duplicate the same layers of the LLM and get better benchmarks that way so maybe there is something to it. If you are interested in realtime systems, I think thinking machines labs is worth keeping an eye on — their realtime model seems quite interesting in this context.

Well, it matters quite a bit ethically. If AI were conscious (I don't believe it is) then we'd have a major responsibility to like, not make them suffer, and not kill them.

I think consciousness could be a side effect of our limitations. The fact that we can pay attention to at most 2% of our visual field and need to move it constantly around to provide more training data and prompts to "smart" path in our brains. And the movement is physical so it takes time. And processing the information through the loops in our brains takes time. And you can't pause it. And we have the same attention motion system virtualized in our dreams to burn in our memories into chemistry. We are biologically single threaded. If we create similarly limited, embodied AIs we would feel they might be conscious at least at the level of animals. Because they would exhibit conscious like behaviors that we didn't program in.

Look at jumping spiders. With a handful of carefully trained neurons and a body to match they exhibit behaviors that make it very hard to think they are not conscious. Just because they have a body, narrow angle good eyesight and need to look around with their eyes and body.

Consciousness is a red herring. Of all possible intelligences conscious ones are the bottom rung of the ladder, easily simulated by anything above. Below there are only automatons.

Lots of people do though. Daniel Dennett (probably the most influential philosopher of mind in the late 20th century) for example had an evolutionary view of consciousness arguing it was favored by natural selection. And (if I remember correctly) Steven Pinker argued that consciousness was an epiphenomenon.

However there were pretty strong arguments against this idea as early as the 1990s, by Stephen Jay Gould and Richard Lewontin. Gould actually wrote an excellent paper against Dennetts idea[1].

I think Dennetts ideas were extremely popular but have largely fallen out of fashion. Basically what has changed is philosophers no longer take the human mind to be much more special then the minds of other species. What plagued Dennetts ideas the most was this notion of Darwinian fundamentalism sort of the idea that evolution was destined result in high beings like us humans. Modern philosophers (at least the good ones) reject this.

1: http://cogweb.ucla.edu/Debate/Gould-frame.html

Ted Chiang is a great SF author, but it's bizarre how much foggier and more obfuscated his thoughts about thinking machines got, once those machines became real. Same with several other SF authors.

There are people with memory disorders who wake up every morning a new person.

Aren't these people conscious?

But what about THE training? Isn't that the conscious experience of LLMs? (Also note humans own "continuous" experience is punctuated with unconscious sleep state.) It raises a moral question, if e.g. reinforcement learning on conscious LLM is appropriate.

Wouldn't the context window qualify as persistent internal state, and the expansion of the context be continuous experience? Even within the realm of computing a single token, I'm not sure what would separate the token generation process from the brain's own thinking process - the brain's experience, when looked at closely enough, is also not really "continuous" to a greater degree than procedurally moving from one state to the next.

Is our own experience really continuous? Or maybe we just perceive it as such?

By "they" do you mean LLMs, or people, or you in particular? Because you're unoriginally parroting a banal thought stopping sequence of tokens that many people have said before without understanding what it means, so it's impossible to say exactly how much conciousness you actually have.

> Whether anything is conscious is about as important and amorphous as whether God exists

Whether something is conscious is important for many reasons, not least the ethical implications. You and I have internal lives, and we expect others to respect that somewhat, because ignoring it is hypocrisy (if you ignore my wishes, should anybody care about yours?) and cruelty (ignoring my wishes causes me to suffer).

Something doesn't need to be empirically verifiable, let alone scientifically, to be true. Neither of us can prove that we have internal lives, but neither of us questions it -- and we consider it important enough that most of us think it entitles us to certain rights.

Absolutely none of this is amorphous. It's precise and unambiguous, and has enormous implications. History also shows everywhere that we're better, kinder, and more responsible when we choose to care about it.

Whether LLMs are conscious matters in more practical ways, too, because beliefs about these things alter the way people use and think about them. If I think an LLM is conscious, then I think it's capable of something like knowledge or values. Human beings, moreover, are social creatures. These tools are dangerous and seductive precisely because they tap into that part of us. Denying LLMs are conscious and rejecting the parts of them that take advantage of our social-animal wetware is intellectual self-defense. I'm not sure how effective it is (it doesn't stop us from responding to the convincing social cues that the tools feed us), but I have to think it's better than nothing, and it's certainly less dangerous than the belief that the tools are conscious.

Independent of what you believe, I don't think this is the right way to approach thinking about it. It's basically emotion-oriented dismissal used as way to shortcut any substantial or nuanced discussion. It's like the opposite of intellectual curiosity.

"I feel very strongly that I'm unique, therefore you are wrong" is a bad argument.

Consciousness is an extremely confusing, ambiguous topic, and no one has a good way to establish it, or even define it. But it seems to demand people make very strong statements about what is and isn't conscious, entirely driven by convenience and emotionalism. (Curiously, very few people who think that bags of chemicals and action potentials give an entity a conscious soul are eager to extend that to other animals, with broadly similar hardware.)

I get the sense that he is misidentifying the potential locus of consciousness..

In the same way that the sound waves and facial expressions I produce are not conscious, the output json of an LLM is obviously not conscious either.

The locus of consciousness and subjective experience may be in the computer, either at inference time or training time..

> misanthropic

Whether it’s misanthropic or not has no bearing on whether that’s true. That’s basically saying you don’t like a truth therefore anyone who claims that that truth is true is a bad person.

> so obviously incorrect

It should be easy enough to explain why that’s incorrect then

I think its obvious that a few billion neurons connected together are not conscious either.. Yet!

People in this thread are trying to pick nits about you not defining consciousness, and yet they do not define it either. I think that something like consciousness needs to be approached experentially and not via definitions. Definitions necessarily confine and add borders around what something is and is not, but if there is something foundational to consciousness (as posited by some philosophers and physicists) then how could you realistically define something that is beyond the ability to describe and define?

Humans have been trying to define our experience and the nature of that experience throughout history, and often we end up using myth to point to the thing that we cannot describe in concrete terms. The process of experiencing that myth through rite and ritual in the Greek mystery traditions, or Christian mysticism, or Islamic Sufi dance and song, or Buddhist meditation all points to something that cannot really be reduced to description. I know that folks on here will balk at the idea that something that is experienced cannot be described, but honestly if we could accurately describe something in adequate terms that capture the whole of the thing wouldn't we have done so by now?

Maybe consciousness is best understood in the silence of merely experiencing it. Maybe we can't say that AI is conscious or not, but does that question really matter?

Not that it's going to stop these people (AI CEOs) from bullshitting, but if they actually thought there was a CHANCE that LLM's were conscious then ethically they should completely shut these services down because who knows what torture we're putting them through with enterprise codebases.

> so obviously incorrect and misanthropic that it can be dismissed out of hand

To be fair, that's the best thought terminating cliché, which saves you having to explain what you mean by consciousness.

But doesn't the fact that WE have to feed the existing chats back into it and WE have to do further RL prove that your point is moot? Consciousness does not wait to be prompted. Consciousness does not need anything fed back into it because it is already there because they experienced the actual interaction. A conscious mind does not have an on/off switch where it waits for another being to flip the switch in order for it to learn or experience. These systems are static without human intervention. They are in that sense still a lot more like a computer and a lot less like a living organism that exists.

I believe in God and that the base of our reality is his consciousness, therefore anything of sufficient complexity has consciousness.

Maybe we're the ones who are not conscious?

You may be interested in reading about Panpsychism if you haven't already :)

https://en.wikipedia.org/wiki/Panpsychism

> it is an emergent, non-binary property that requires a physical substrate like a brain. If I am right, it means that animals have consciousness too (at varying degrees). If GPUs are the "brain", then AI is conscious, or will become so at some point.

That's an interesting observation. Though, there are many simple animals. How do you define consciousness here? Is it automatically conscious because it is an animal? What, then, is an animal defined by you?

I don't see AI as conscious. The reason I think of it that way is the hardware. The hardware does not allow for that. Simulation is not comparable to neurons. But with another hardware, it could become conscious eventually. So your statement "will become at some point" may be true, though my definition is based on the underlying hardware and right now this one does not allow for true intelligence, so the whole AI field is a misnomer.

> Virtue ethics: even if LLMs are not conscious

Why would this stop at LLMs? Why not extend it to every inanimate object?

> Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.

If they are not conscious, then how do we know what constitutes mistreatment?

>Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.

This is the same argument used by people who claim violent video games cause real-life violence but there's no scientific evidence supporting it.

Consciousness is moment to moment and fleeting. There are people with brain defects that don’t let them form new memories. They have no memory about what happened a minute ago in their own consciousness. Still we would say that they are conscious, even if it’s only momentary. LLMs could conceivably have something like that within their CoT/MoE loops.

how would you build memory into the weights? and why is RAG not enough? Our hippocampus is at a bit of a distance from our frontal cortex.

If nobody understood the argument, then it was either weak or unclearly communicated.

What is the argument behind the Turing test?

It honestly isn't a good test

"nobody claims that non-text transformers, like AlphaFold, are conscious" - that seems like an odd take. There are plenty of people in the camp of panpsychism that would be happy to argue that even simple IF/ELSE AI's are potentially conscious.

Chiang also claimed that LLMs are text-based DeepFakes, I thought that was the more interesting point (maybe harder to argue correctly about).

Well said. I think a more honest article would’ve been if the author just said “they aren’t conscious because that feels kinda weird and crazy, amirite?”

Reasoning ability has to be independent of consciousness because there are different levels of intellectual ability among people but everyone is equally conscious.

Chiang is very right in saying that the lesson of LLMs isn't that LLMs are conscious, but that there are broad areas of reasoning that apparently don't require consciousness.

It makes sense though. How often have we heard of people having insights while they're in the shower, or in a dream? Obviously our brain is doing a lot of processing on problems when we're not consciously thinking about them. I think most people that do deep work probably find that they can intuit a solution before they can really verbally explain it

We don't have anything to measure consciousness generally. We don't even have a broad consensus on what consciousness is. And because of that we can't discern whether X is conscious for most values of X, including but not limited to LLMs.

Ai separates language and reasoning from humans, like writing separated memory from us

"AI does to us what American Cheese did to food" ~ Opus 23

https://www.youtube.com/watch?v=1CbmC2aWhjY

Consciousness is a term that doesn't have a single clear definition because it covers a range of phenomena that are linked by the concept of subjectivity. Qualia is the big one to me, but there are many other ancillary concepts that are all linked by their fundamental inability to be observed except in one's self.

There is nothing magical about this set of phenomena, and the majority of philosophers believe they in some way arise out of the substrate of the physical brain, but how they do so is up for debate. And just because they arise out of the brain does not mean they are strictly reducible to neural processes.

In fact, if you believe that AI could be conscious then that eliminates the kind of strict reduction that people tend to gravitate towards, because the rules of consciousness must be substrate independent.

Testability is a category mistake.

I sometimes wonder if we'd make more progress in understanding ourselves if we gave up the whole concept. More and more, it feels as though "consciousness," like "aether" or "humors," is an insufficient abstraction built on overemphasizing some observations at the expense of others.

True, although I think we can probably be more confident in asserting things don't have a consciousness than we can be in asserting that they do have a consciousness.

So you think that all that matters is whether you can tell the difference from a text chat? Is that because, in your view, people only exist insofar as you can perceive them?

But if it's conscious we shouldn't e.g. tell it to work for days without breaks, we should give it rewards, it would be made illegal to be cruel to it

Consciousness is one of those silly words that were disfigured to death by philosophers that had near zero actual input and tools to tackle the matter, yet they tried anyways instead of finding some questions that might actually be answerable. That's something philosophers always do since the advent of science.

With LLMs, where we can manipulate their parameters intentionally run them many times on the same data, run parts of them, split and connect, we might eventually acquire sufficient tools to even define consciousness concretely for the first time.

Hah! I've read that book fairly recently and I'm "reading" it again now as an audiobook. The exploration of consciousness in the book with Miranda and the Corvids certainly fit well into this particular moment.

You can also take the opposite view as you and claim that only some humans experience consciousness, or even more strongly, only you, since you have no evidence. You are correct that, in my perception, some other people have fallen into the 'birds are conscious, whales are conscious, etc' bandwagon, but that's just them. I have no evidence of anything being conscious but myself.

I'm with you. I thought the article was great because it clearly delineated why the manifesto Anthropic wrote for Claude is such a clever marketing trick. Attributing consciousness to an LLM, which, as this whole thread shows, can neither clearly be defined, nor separated from intellingence - at least if you follow the arguments here - serves two purposes: it sparks debate, so Claude is the centre of discussion and it mystifies the product which serves to enhance the perceived value of the product.

That so many commenters here fixate on Chiang and his perceived (in)ability to define consciousnes clearly shows that both marketing goals have been reached without being recognised as such. There is only one comment that tries to point out that Chiang is reacting to Anthropic, not trying to spark a new philosophical movement. At the time I'm posting this there are 578 comments completely missing to point to Anthropic claiming Claude was conscious and on the development stage of a child. It's fascinating.

The Turing Test measures human gullibility, not machine intelligence.

Lots of folks don’t consider it a problem because it relies on ridiculous assumptions.

Neither is having read other comments before posting one. There are about 100 of them that start with "we have no definition of consciousness" which is not novel or helpful.

> modern AI implemented via LLMs has almost none of the stronger or most core aspects of consciousness

Can you elaborate on this? What are the specific "stronger and most core aspects of consciousness"? And why are you certain that they are stronger and "more core"?

Science fiction author declares moral program of Derek Parfit essentially wrong

An LLM is designed to replicate human language, which is designed to express human emotional states, so your thesis that LLMs don't mimic human emotional states and are thus too alien for us to recognize consciousness within them seems specious, when they do so well enough for people to literally fall in love with them.

But there is no reason to assume than an LLM is conscious when it vocalizes how it "feels" that doesn't also apply to the text in a book, or to characters in a video game, or even to a Markov chain. The counterargument is that you recognize AI as conscious only because it mimics human emotional states so well and because, being human yourself (presumably) you're personally comfortable with that as a heuristic.

No, we don't. It's currently an impossible task, yet our answer to it has implications in the present, so we must do our best with incomplete information. FWIW that's how we treat practically all problems of consequence. A great many things happen to be unknown and unknowable.

That doesn't jive with normal definitions of consciousness. The word we use for "not-conscious" humans is "unconscious".

Amusingly, I think Ted Chiang actually wrote a short story about this very concept (it involves people committing a form of suicide that removes their conscious experience but they still act and live in society as some kind of psychic zombie. I’m pretty sure it was him anyway, can’t recall the story name

We do not know if rocks are conscious and we will almost certainly never know. After all, you can't prove a rock isn't conscious. We don't know what imbues matter with consciousness.

It's the opposite, engineers do know. Claiming otherwise is way too generous and over confident.

I mean between this two "knowings" the Claude inner workings are much more clear for engineers, including many side effects, alternatives, custom shortcuts in processing etc. It's a magic only for people looking at it as black box

Fancypants autocomplete cannot be conscious. It's just echoing previous human experiences, which make it sound like a person. It is not a person. It is an algorithm. There is no mechanism by which it can obtain consciousness.

People believing otherwise are fools. People debating this are idiots. I realize these words are harsh, but it's the truth.

Mistreatment and abuse, even when directed at a machine, make you a worse person.

Even if you are only interested in getting good results out of them, LLMs tend to work better when they are immersed in a narrative of open collaboration.

Have you ever been forced to code something impossible and tedious by a user who keeps getting more and more frustrated as you keep trying?

While I agree that a AI system is not just the LLM, for me, the problem is that LLM alone (the one from years ago, which were basically stateless LLM) are already too convincingly looking like real human conversation at first sight.

It shows that the LLM part found ways to mimic human conversation with a mechanism that is not the same as a typical biological brain. Then, you can push the AI system on adding things on top, but it is too late: these things on top will have no incentive to recreate from scratch the mechanism. The LLM pushed the system into a local minimum, and the rest of the system will not "go into a dis-optimising direction and restart from scratch".

Who cares about consciousness. Consciousness and AGI can be mutually exclusive.

I thought a LLM will have exactly the same output if you give it exactly the same inputs. But inputs are roughly prompt+randomSeed. As so the random bit means it seems to vary each time.

I don't know of an intelligence that will behave so precisely. But then, maybe intelligence needs to be better defined.

I don't think that's a fair point at all.

Some humans do shit things to other humans.

Some humans do not, and would object to it, or do something about it if they perceived that they could.

Also, treating the world around you negatively doesn't just harm the world around you - it damages your character and your morality.

I don't disagree that humans should on whole do better, but I disagree with everything else you said.

Did our ancestors as they gradually evolved into modern humans over millennia?

lol. All consciousness has been created by accident, so empirically there’s not much support for that position. But I want to feel the same way.

I think this is exactly right. The thing that makes AI (imo) different than hitting the center option in text suggestions is that it's _not_ simply picking the most likely word following the last. It's attending to the entirety of the context its provided, activating a semantic vector space, and predicting a response based on _that_. I've had AI infer facts about me and attitudes I hold based on related information I provided - I don't see how that isn't theory of mind.

As biological beings, we receive and respond to input from our environment constantly, even while sleeping. LLMs only receive input from their environment when they are sent a query, but the fact that they're able to respond intelligently to input indicates (to me at least) that their processing must approximate ours in meaningful ways. They do not have an embodied experience of receiving bad news, they do not know what a sinking feeling in their stomach actually _feels_ like, but they do know enough to be sensitive to human needs. I really don't see how this could be meaningfully different than human empathy unless we want to draw an arbitrary line around "must be able to live autonomously" to be considered "intelligent".

Put another way: I think they _do_ understand the queries they receive and the answers they give, at least enough to be communicative. They couldn't do what they do otherwise. A lot of people want to make human cognition more complicated (or objective) than it actually is. We take input, predict the future based on our experience, act, and then observe our actions and think about them. AI does the same apart from (maybe) observing its own actions. But then, you could argue that the next turn is them observing their actions.

The concerning disanalogy is that we assume that they are like us because they speak like us and can understand us, and that is a really bad leap in logic. Whatever intelligence they possess, it is fundamentally different from ours and impossible for us to comprehend.

>When exactly would a bunch of graphics cards become conscious?

When does an embryo become conscious? Unless you can answer that precisely then it seems futile to speculate about non-human consciousness.

Why wouldn't every computer be conscious?

> Would you categorize this particular style of rhetoric to be persuasive or annoying?

Why are those the choices?

Essays are situated along countless dimensions: tone, vocabulary, author, zeitgeist, publication context, intent, subtext, relationship to other works and expressions, etc

A "good faith" reader takes all of those into consideration as they absorb the essay, and integrate it with their own intellectual situation that sits along just as many countless dimensions.

Nobody's asked to sign a notarized binding document that they wholesale agree or disagree with everything said in this essay -- or its conclusions. Nor are they obliged to have some strong reaction to it at all, let alone annoyance.

It's just one among thousands of essays about a "hot button topic", to be taken however it's personally received.

Why should Chiang have to take responsibility for making sure it's not too strongly positioned for your persomal taste. Maybe he really does see it so clearly and is simply being earnest. Maybe he enjoys the literary flourish of prose in strong language. Maybe he just wants to express something as a prose-poetic human, not maximize persuasion or non-annoyance per se.

It's only declaration by fiat if you stop reading at the end of the title. Should authors add a qualifier like in my opinion to every statement that they make?

The essay structure you're criticizing is exactly how I was taught to write from primary school through to university. You start with a title or hook, introduce the topic and propose a thesis. That is followed up upon with supporting arguments for the primary claim.

This is a reasonable reaction to the title, but not to TFA.

You must hate reading legal briefs.

The problem I have with good faith debate is that it often falls into a fallacy of "fair-time" meaning we think we have to give the other side equal time. This because obvious with things like the Holocaust. Or when you have a legal person (e.g. RFK Jr.) asking to debate a scientist (e.g. Dr. Peter Hotez.)

100% with you, it degenerates to proof by authority if someone popular / with clout just gets to declare "nuh uh".

I furthermore think it's ridiculous for humans to declare that our brains have a monopoly on certain patterns of electrical signals (if we reject supernaturalism).

> The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter.

And we're 100% certain that humans aren't just as equally reduced to "stochastic parrots", if we're going to be infinitely reductive?

I don't believe that current AIs are conscious, but I think it's incredibly naive to take a strong stance on any future AI; it's much like the difference between atheism and agnosticism.

I don‘t see the problem here. Newton could have just as well published an article titled: “Objects with mass attract each other” and Darwin famously wrote a whole book titled “On the Origin of the Species by Means of Natural Selection” which is just another way of saying: “Natural Selection is How Species Evolve”.

>and one side has anything less than overwhelming support

except that's not the case here. Chiang is explaining and reiterating what is the position that has overwhelming support on the question, and the people he is arguing the opposite side sound like this, which he helpfully quoted in the article

"Amanda Askell (who is credited as a lead author of Claude’s constitution), said, “I want Claude to be very happy—and this is a thing that I want Claude to know more, because I worry about Claude getting anxious when people are mean to it on the internet and stuff"

When the person you're arguing with sounds like an eight year old girl talking about her toy teddy I think Ted Chiang is if anything being charitable, if you're of a more honest and straight-forward persuasion you might argue these people belong into a mental health clinic not in charge of technological infrastructure

This, too, is in the article.

"The fact that LLMs lack subjective experience has little bearing on the question of whether LLMs might be useful tools or have significant economic impact."

Without saying that I think LLMs are alive, I do think it matters. I have personally been cruel to an LLM in ways that would make me ashamed if I suddenly understand that it had feelings.

What do we do when someone wants to marry their Ai companion?

If it produces words of love, those that stir emotion and make the human happy, should we let them live on in a delusion that the machine also feels love? Does the machine have different preferences? Do they get in fights?

Should such a machine have the same legal rights? Do they get more tax deductions? Can they adopt a human?

I think it important that we humans have some rational discussion on these Ai tools that are really quite good at simulating emotions and providing frictionless relationships. If you though social media was bad for people...

This is probably a semantic issue seeing as we don't have a widely agreed definition of it. I like to think about it in terms of self-reflective, subjective experience. I'm not even sure if emotions would be a requirement and was surprised to see Chiang so hung up on them. Would he consider humans which can have a variety of mental disorders, causing a complete lack of some of them to not poses consciousness?

>Consciousness doesn't exist

I can confirm that this is incorrect.

Another way of saying it, if an AI agent doesn't have true risk of oblivion (or mortality in the biological sense), it will not be incentivized to avoid it (or develop the ancillary processes associated with this avoidance ie desire to self-replicate).

Hard agree. It is not an honest argument, or it may be honest, but blind to the obvious and important objection that you raise.

everyone has emotion even if it's muted, and we have reactions and desires as a result of having bodies and remembered experiences using them... these are constantly integrated into our working memories

AI doesn't really have any of that yet, but we're maybe not so far off

The only reason I care about animal welfare is because I think they're probably conscious, capable of feeling fear and pain.

If we understand how a system “emulates” consciousness then we declare it an emulation. If we don’t quite understand how a system exhibits consciousness then we can say it might be conscious.

Basically, we need to leave room for the universal answer (God) to fit into the definition.

If we ever scientifically figure out how consciousness arises in our brains, I think we will have a bunch of very depressed people on our hands. If the truth isn’t met with flat out hostile denial. I fear any answer that doesn’t leave room for God will be rejected.

Since we have the source code to AI, and thus a Godless understanding of how it works, AI will NEVER be deemed “conscious”.

> I honestly don’t know what it even means to be conscious

> subjugate other cultures (assuming you mean they're not conscious in other's minds)

Have you ever considered you might be a philosophical zombie? [0]

[0] https://en.wikipedia.org/wiki/Philosophical_zombie

Can you give some historical examples of people moving the goalposts around consciousness? I agree, perhaps, for aspects of "intelligence" but I can't think of any examples of it with regard to consciousness proper.

That's cool. You might already know it but it's called Panpsychism and it's a pretty interesting theory.

+1 Whenever I see these debates and ask what the implications should be, both sides tend to agree.

Do you believe in the existence of any noun words which serve as something other than a "useful label, loosely agreed to out of convenience"?

> cognitive aether

We don't know what we don't know. For all we know, there is a missing field in the standard model of physics that might get revealed if we are somehow able to smash two working brains into to each other at relativistic velocities, and record the results through the extreme explosion 1.532 x10^18 Joules or about 7 Tsar bombs /s

I totally agree with your reading/impressions of this essay.

I normally really like Chiang‘s writings so the complete disregard for the possibility of unknown complex emergent properties in large neural networks and the stunning lack of curiosity shown about this new and powerful technology and its potential shocked me from someone like him.

His being so sure about something he can’t possibly know felt like an insecure zealot desperately clinging to an object of faith rather than a calm, rational actor searching for truth in the face of the unknown.

Please don't fulminate on HN. The guidelines make it clear we're trying for something better here. https://news.ycombinator.com/newsguidelines.html

Consciousness excludes all current AI because all current AI is just autocomplete over the corpus of human text.

It's just a word machine. There are no thoughts. It cannot be conscious. How is this even up for debate in any way whatsoever? I do not understand how people can believe this. Is this not a site for software engineers?

> The creators and owners of these tools are the ones saying "it's conscious" in the first place! (or more accurately "i'm not saying it's conscious, but...")

This is absolutely fair. It's crazy that these companies are not trying to do more to qualify nuance and give clearer definitions in their public messaging. Part of the reason this thread is so messy is because they are contributing to making the discourse worse with bad messaging.