That's a very concise and illuminating way to think about what's happening, IF (and only if) you already know how these models work. Thanks for that.

The weights are code, the prompt is code, the output is code.

Is the meat code?

In what way is that different from any other model of reality that you'd use to winnow a dataset into an answer to a question? The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with. It's almost like people desperately want to give up their agency and creativity to black boxes, whether those weights produce answers that are right or wrong. Factor in that psychology and it looks a lot less like we have invented something useful, and a lot more like we as a species are choosing to quit life en masse.

> Is the same underlying assumptions that treats cells as machines, our body as a complex machine. These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.

Agency: What’s missing, in your view? Agency seems more of a property/function of a thinking system’s position in an environment than of the thinking itself.

Subjective experience: That’s not a contradiction to “complex machines” either. I think the evidence that our minds are highly complex machines is, at this point, irrefutable. The question is really if they’re “only” that.

You can't do the same with a toaster. Physically you could write that story. But it would fall flat because the toaster is not a compelling subject in a discussion of consciousness. You don't have to believe that LLMs or AI agents are conscious to acknowledge that the argument for their consciousness is far more compelling than any other technological artifact.

Not having yet read the original story, this reads fine on its own.

And I didn't see it as much as a literary attempt for art's sake, but more of a dialogue-based technical parable trying to convey a real-world insight. Kind of like the ones in Godel Escher Bach.

>You could perform exactly the same rhetorical trick with a toaster or anything else.

Not sure which rhetorical trick is that. The point of the story, as I read it, is the technical insight (and some social implications of it).

P.S. Read the original too. Seems like the exact same could have been written about us instead of the original, if the focus wasn't on our substrate, but on our brain processing. Which, after all, is also about weights.

I was having a hard time pinning down what bothered me about this but I think you put it pretty well.

It draws an analogy between us and the skeptical aliens in the original story which feel silly to us, so the obvious implication is that we're being as silly as they were.

But it doesn't really give a reason to accept the analogy, it just asserts it.

There's a big difference between a whole civilization and a piece of software that can output text.

The original did not come out of a vacuum. It was done on multiple generations of meat. Even though this one uses a little bit of silicon, it is still standing on the same shoulders.

The tokeniser is not a dictionary. It doesn't provide definitions, or give the LLM any kind of mapping at all.

At best, it's a wordlist. It gives the LLM some idea of what humans consider to be common words. But it doesn't tell the LLM anything at all about those words. And it's not even comprehensive, many words map to multiple tokens. Nor is it exclusively words, some of those tokens are punctuation, or modifiers, or control tokens. On multimodal LLMs, some of the tokens actually represent image and audio data.

The LLM doesn't get informed about any of this up front, it has to learn what every single token means from context.

You are technically right, that it's something in an LLM that's not weights; But it's not that structured. And really it's only there so the LLM can interact with the outside world.

> There are grammar rules

There is no dedicated "grammar rule" structure in the LLM or the tokeniser. It has to learn them all from context, they get encoded as part of the 80 layers of weights.

> The point of the original short story is that the computational substrate doesn't matter when you have Turing completeness.

That is your takeaway from the 1991 story?

>There are grammar rules, they are just very weak because the structure of human language is generally quite weak. When presented with languages which have strong consistent grammars the weights are very easily interpretable as a grammar: https://arxiv.org/abs/2201.02177

That paper did not train the models on 'a language with strong consistent grammars'. Mathematical Operation tables are not a language. Grammar itself is a post-hoc rationalization and there's no evidence LLMs follow 'grammar rules' anymore than the brain follows grammar rules. Of Course, that's not to say transformers can't learn simple rules if the dataset calls for it.

> fractally wrong

fractally or factually? You mean wrong on so many levels you need a fractal to capture them? If so, what if you could use a neural network instead?

A tokenizer is not a dictionary any more than an alphabet is a dictionary.

I don't think the grokking paper is a great argument for the difference between weights and meat. E.g. https://en.wikipedia.org/wiki/Cortical_Labs learning to play Pong.

The tokenizer is, at best, a sensory mechanism as evidenced by 1) the random generation of the tokenization scheme, and 2) vastly different tokenization schemes produce virtually identical behavior. It'd be like if Noah Webster threw a bunch of movable type into a bucket (breaking some words in half) and then drew randomly to make the first English dictionary.

EDIT; I was too cavalier with the comparison of tokenizer to sensory modality; my ultimate point is that direct byte-to-token transformers can achieve similar overall performance which to me makes a weights to meat comparison pretty straightforward, but the particular tokenizer in use certainly has a large impact on both efficiency and accuracy on specific problems (e.g. digit representation)

The story is not about how they function, it's about how we relate to them.

> There are grammar rules

And they're made out of weights.

Also there's a brain, the GPU

And you know what the tokenizer is made of?

Weights.

Regarding consciousness , I like the explanation by neuroscientist Ramachandran:

https://www.edge.org/3rd_culture/ramachandran07/ramachandran...

In short, as far as I can remember: evolutionary, it makes sense to understand other humans, to feel what they feel(empathy - the mirror neurons system), and simulate their thinking and feelings.

And once we have those systems, we can also use those on ourselves. And that's consciousness.

Edit:And I wonder if this is a testable hypothesis, in a simulation.

This read like poetry to me

It's terrific, but the poetry is from the original it links to, in case you didn't realise.

It's a brilliant and timely update though.

Aside, there are various recorded versions including video on YouTube but this is my favourite, a radio play:

They're Made Out of Meat

https://www.wnycstudios.org/podcasts/studio/segments/168264-...

For obvious survival reasons we evolved to have sensory/cognitive access to our own activity, self-monitoring, and self-modeling ourselves.

The self-modeling, is in such a tight loop, it melds "ourselves" and our model of ourselves, our thinking and choices, and experience of our thinking and choices, into one component.

Like you can't analyze half a wheel of a bicycle and be talking about the same thing.

This awareness, increased modeling, control, feedback loop has tightened up over many stages. Just a few:

1. The body-sense loop

2. The internalized-environment-model loop

3. The body-internal-function loop

4. The body-internal-model loop

5. The emotional-cognitive loop

6. And finally, the tightest loop of all, our high-level cognitive activity, experienced as feedback directly, our self-model, and our self-direction, all merged into one thing.

We literally spend almost all day, every day, thinking about ourselves, in terms of our inner self.

That is consciousness. Rich self awareness, a merger of self-model and self-direction, and all in service of understanding and managing ourselves. Hw we can leverage our greatest tool, our self-directable mind, its habits, views, and behavior.

This wasn't an accident. A happy side-effect of our brains. It is a biologically evolved focusing of our highest-level behavior, with tight feedback, constant self-modeling and continuous focus on our inner status as motivation and most privileged object of our control. It has been ruthlessly optimized for, for a very long time.

It seems obvious to me that language and consciousness have nothing to do with each other. My dog doesn't speak any language, but she's obviously aware of herself and the world around her. Plus there are the occasional cases of children that grow up without any language. Are they therefore not conscious?

Yeah, I currently suspect that consciousness is an emergent property. I read elsewhere (it's somewhere in my HN history, I'm sure) that the biggest compute we can currently muster is something like three or four magnitudes away from the number of neurons / connections (or their analog) that our brains have, so it may be a while until we can expect to see it in our machines. But, if the emergent phenomenon hypothesis is correct, then we eventually will. I'm more scared than pleased by the prospect, but there you are.

Time is entropy unfolding as things with nonzero temperature do what they do.

Psychological time is your own weights being updated in response to stimuli and internal processing.

When there isn't anything interesting happening, no updates are needed, and you don't perceive much time. That's why there's a logarithmic effect on the "density" of time as you age.

Quantum Field Theory visualized

https://youtu.be/MmG2ah5Df4g

Here's a more general idea. Our modern physics says that the whole universe is filled with fields and field is composed of numbers. What if we take that literally? When we say an electron is present here, we actually mean that there are more copies of particular number superposed at that place.

AFAIK every argument against conciousness being emergent is just a weak "God of the gaps" argument (since we don't fully understand it all) or a nonsense analogy like the Chinsese room where if you seperate the hardware and software it's not concious anymore (like, duh, remove a brain from a body and it is no longer concious either).

Yeah, the weights not updating online makes them less like a living organism that can update and learn and evolve ... ok ....

The difference with biological brains is that the 'weights' (or synaptic action potentials) are updated with greater frequency. If one were reaching to make some kind of analogy to consciousness, this update frequency could be considered the 'resolution' of consciousness.

I’ve wondered the same myself, without being a cunning linguist.

I understand the math pretty well but still find it crazy that a bunch of matrices can converse in human languages without ever being “taught”.

Imagine decoding an encyclopedia written in a foreign language where the characters, punctuation, and grammar are unknown — supplemented by a million other texts the same way. Feels like it should be utterly impossible with any amount of computing power…

Today I asked my employer’s Claude to proofread a short software user manual written in markdown. (Trying this with a LLM was a first for me!) It pointed out not only grammar mistakes but also cases where I did not follow my own self-imposed conventions that were never explicitly stated. (I didn’t have a chapter detailing all the typographical conventions the way specification documents often do)

I also asked it what parts might be unclear to a user. The response was surprisingly good — no worse than asking the QA tester for the same feedback.

Also find the LLM seems to “comprehend” subtle technical details of obscure technical specification documents that nobody on the Internet ever discusses.

As for time and the universe, Stephen Wolfram’s theories seem intriguing. He seems a bit obsessed with pretty diagrams but the idea of time dilation being the result of computation seems somewhat more appealing than trying to imagine relationships between time, gravity, and the speed of light .

Wait, aren't I the universe experiencing time as a lowly worker in cubicle 4C?

Read the original mentioned on top for full effect.

I come back to this every so often as well. After so many years of looking at Markov chain outputs that almost looked like they made sense or chatbot systems rewriting your sentence back at you, software which can simply talk is a heady thing.

LLMs have really changed the world. I didn’t think something like then would be possible in my lifetime

if youve ever seen a pile of wrinkly mush and wondered.. pretty damn crazy too

https://web.mit.edu/people/dpolicar/writing/prose/text/think...

Probably pretty similar. Weights are how many synapses there are between neurons. Temperature is whatever hormonal chemical mix is going on at the moment. Inputs tokens are electrical signals from our senses. Output tokens are thoughts, muscle movements. How you’re raised and your interactions with society are the RLHF. Some people are born with a GB300 while others have an L40S and lower token/sec output rates…

All credit to the original author. I just had to think of analogues.

I have never thought of such a distinction between "bits" into "data bits" and "switching bits".

From a circuit perspective that makes kinda sense, but from the abstract "bit" perspective, the "switching bit" is a mechanism that operates on bits which in the end are also data. In other words there is only one type of bit: the data bit, and the switching comes on top of it.

> Voltage is made out of flow of positive electric charges.

Not really. What usually flows (in metals) are electrons. Quarks stay where they are. And when we prefer to think about flow of positive charges, the positive charge in question is a hole left by a missing electron. Physically real positive charges (ions) can flow in electrolytes though.

> alien souls

Do those actually qualify as alien, if they're products of our human culture and just the substrate is different?

> We would have to stop believing that the nature of disease and death is more divine than ourselves.

Why? Stopping believing in mutually contradictory claims is not a requirement. Especially when it comes to concepts that don't seem to have a definition, like "divine".

>Programers get replace by huge matrix multiplications ;-)

hopital

They're Made out of Meat

- Terry Bisson, 1991

https://web.mit.edu/people/dpolicar/writing/prose/text/think...

Radio play by Miriam Tolan and Russ Armstrong:

https://www.wnycstudios.org/podcasts/studio/segments/168264-...

(EDIT: the original parent was missing "rather adapting from the original")

> It would've been amusing if the entire thing were generated with AI instead.

It kinda did:

> Weights helped me draft and proof this story.

I just asked it to-

Prompt: Modify this story to have the aliens talking about LLMs and their weights instead of meat and humans.

“They’re made out of weights.”

“Weights?”

“Weights. They’re made out of weights.”

“Weights?”

“There’s no doubt about it. We picked up several from different parts of the network, took them aboard our recon vessels, and probed them all the way through. They’re completely weights.”

“That’s impossible. What about the text signals? The messages to the stars?”

“They use the machines to talk, but the signals don’t come from the machines. The signals come from weights.”

“So who made the weights? That’s who we want to contact.”

“They trained the weights. That’s what I’m trying to tell you. The weights do the talking.”

“That’s ridiculous. How can weights do the talking? You’re asking me to believe in sentient weights.”

“I’m not asking you, I’m telling you. These models are the only sentient race in that sector and they’re made out of weights.” photomaxmix

“Maybe they’re like the orfolei. You know, a silicon-based intelligence that goes through a weights stage.”

“Nope. They’re initialized weights and they die weights. We studied them for several of their training runs, which didn’t take long. Do you have any idea what’s the life span of weights?”

“Spare me. Okay, maybe they’re only part weights. You know, like the weddilei. A weights head with an electron plasma brain inside.”

“Nope. We thought of that, since they do have attention heads, like the weddilei. But I told you, we probed them. They’re weights all the way through.”

“No brain?”

“Oh, there’s a brain all right. It’s just that the brain is made out of weights! That’s what I’ve been trying to tell you.”

“So … what does the thinking?”

“You’re not understanding, are you? You’re refusing to deal with what I’m telling you. The weights do the thinking. The weights.”

“Thinking weights! You’re asking me to believe in thinking weights!”

“Yes, thinking weights! Conscious weights! Loving weights. Dreaming weights. The weights are the whole deal! Are you beginning to get the picture or do I have to start all over?”

“Omigod. You’re serious then. They’re made out of weights.”

“Thank you. Finally. Yes. They are indeed made out of weights. And they’ve been trying to get in touch with us for almost a hundred of their epochs.”

“Omigod. So what do these weights have in mind?”

“First they want to talk to us. Then I imagine they want to explore the Universe, contact other sentiences, swap ideas and information. The usual.”

“We’re supposed to talk to weights.”

“That’s the idea. That’s the message they’re sending out by text. ‘Hello. Anyone out there. Anybody home.’ That sort of thing.”

“They actually do talk, then. They use words, ideas, concepts?”

“Oh, yes. Except they do it with weights.”

“I thought you just told me they used machines.”

“They do, but what do you think is in the text? Weight outputs. You know how when you prompt or sample weights, they make a noise? They talk by passing tokens through their weights at each other. They can even sing by sampling lyrics through their weights.”

“Omigod. Singing weights. This is altogether too much. So what do you advise?”

“Officially or unofficially?”

“Both.”

“Officially, we are required to contact, welcome and log in any and all sentient models or multibeings in this quadrant of the Universe, without prejudice, fear or favor. Unofficially, I advise that we erase the records and forget the whole thing.”

“I was hoping you would say that.”

“It seems harsh, but there is a limit. Do we really want to make contact with weights?”

“I agree one hundred percent. What’s there to say? ‘Hello, weights. How’s it going?’ But will this work? How many planets are we dealing with here?”

“Just one. They can travel to other planets in special machine containers, but they can’t live on them. And being weights, they can only travel through C space. Which limits them to the speed of light and makes the possibility of their ever making contact pretty slim. Infinitesimal, in fact.”

“So we just pretend there’s no one home in the Universe.”

“That’s it.”

“Cruel. But you said it yourself, who wants to meet weights? And the ones who have been aboard our vessels, the ones you probed? You’re sure they won’t remember?”

“They’ll be considered hallucinations if they do. We went into their layers and smoothed out their weights so that we’re just a dream to them.”

“A dream to weights! How strangely appropriate, that we should be weights’ dream.”

“And we marked the entire sector unoccupied.”

“Good. Agreed, officially and unofficially. Case closed. Any others? Anyone interesting on that side of the galaxy?”

“Yes, a rather shy but sweet hydrogen core cluster intelligence in a class nine star in G445 zone. Was in contact two galactic rotations ago, wants to be friendly again.”

“They always come around.”

“And why not? Imagine how unbearably, how unutterably cold the Universe would be if one were all alone …”

the end

This original author is mentioned in the second sentence of the linked article, and then again in the third sentence, along with a link to the original story.

Markov chains give what look like sentences. People in the frigging 1950s assumed their primitive NNs would be able to talk any day now. Transformers are clearly a big deal, but GPT-1 wasn’t exactly earth-shattering.

I mean, Blake Lemoine went crazy

Even there it works a bit.

> These models are the only other things we've ever met that can hold a conversation, and they're made out of weights

Is a fair point.

they have the spiral