Even so, one might wonder why we don't try making systems that take different approaches. For example, after a traditional first pass of output, they could do sliding-window "optimizations" considering each token in the context of tokens both before and after, and possibly replace words or phrases in-place.

For example, I've noticed quite a few cases recently of LLMs outputting "but" where "and" would make more sense, or vice-versa. Surely that could be improved by such an approach?

The distinction I perhaps didn’t make clearly enough is that I’m not really debating the concept of prediction at inference time, although, as I pointed out elsewhere, I think that’s the less interesting interpretation of what “prediction” means.

What’s more interesting to me is its application at training time. In reinforcement learning, there is no ground-truth next token to predict.

So if you’re comfortable calling Deep Blue a “next move predictor,” then I think it’s perfectly consistent to call an LLM a “next token predictor.” But I think it’s more useful to think of Deep Blue as evaluating the value of possible moves. roughly, how likely they are to lead to winning.

And I think effectively the same distinction applies here.

> generated an article that continually undermines its own main point.

I disagree that this accurately describes TFA.

Yeah, it sounds like this is just a disagreement about what the word "next" means. I agree with you that "next" just means "the one about to come", and if the underlying model works by using some prediction mechanism to determine that, then it's by definition a next-token predictor. Disagreeing with that on the basis that the "next" token isn't necessarily in the training data verbatim just seems like an overly strict definition of the word "next".

I just think its a meaningless dismissive term. It literally does predict the next token. But it ignores that it coherently predicts long continuous sequences of those tokens, that tokens can be anything, and you can do almost literally anything with that capability if it does it well enough.

Blog articles from Anthropic and others show that this is not true.

A LLM already knows more tokens than the current one. It was mentioned in a blog post about how a LLM is doing haikus and co.

There are also structures in an LLM which allows it to 'estimate' numbers to a certain degree and doing other things.

Maybe I'm wrong - but I see LLMs are a "next-token predictor" as somewhat equivalent to brains are a "bag of molecules".

Both systems have emergent behavior that goes well beyond what would naively be expected.

How about "outcome steering" as a mental model? During training it is optimized until it's really successful at producing code / terminal commands / words that make the compiler/computer/itself do something that ultimately completes a long time-horizon task that iswcurrently being trained.

> or perhaps a human who has been reading too much Claudish recently

To be fair, is there anyone who hasn't been "reading too much Claudish recently" who is also qualified to write on the topic?

The distinction is that it's not 'predicting the next token'. Instead it's _determining_ the next token based on a prediction of its reward signal.

The word "predict" has a meaning. I don't "predict" my next move in chess. I might predict what someone elses first move is.

Yes, I think that’s a good explanation. There are really two sides to it.

There’s the mechanical, inference time, autoregressive, one-token-after-another side, which I’m not going to argue isn’t prediction. I just think that’s a relatively uninteresting use of the word “prediction,” because it’s effectively a system predicting its own output.

The more interesting question is what happens at training time. As you describe, reinforcement learning allows the model to learn to output things that it never could have learned simply by predicting what appears in the training corpus.

More concretely, in reinforcement learning there are no ground-truth next tokens to predict.

In supervised machine learning, “prediction” usually means there is some ground-truth label that will eventually be revealed. The model predicts what that label is, the difference between the prediction and the truth gives you a loss, and you learn from that.

But in reinforcement learning, there is no ground-truth action waiting to be revealed. The model chooses an action, observes the consequences, and learns from the reward. To me, that’s a meaningfully different thing from prediction.

Sure, politics and economics are involved - but why can’t technology also play an important role?

None of this AI political economy stuff was relevant in 2015 because necessary technological breakthroughs had not yet been made.

the next token part is like the "clock" that drives it

Nope. Post training means even the raw model isn't predicting.

Obfuscation is the goal of the hype cycle in VC. Certain firms & individuals are minting money and that’s all that matters to them.

That there’s a legion of LLM nerds arguing deterministic this, pretraining & rewards that all the better for the con job they’re pulling off.

The technology will be relegated to the trash bin of history, just like crypto.

At the same time, it ... is literally a next token predictor. Like that's what it is. The input is a sequence of tokens. The output is a probability distribution of next tokens.

But it is a next token predictor.

Recursively invoked.

With carefully selected context.

And massive investment in RL to tune token selection.

And the ability to use cli tools on other folks' machines.

That's a powerful system built around a conceptually simple technology: Next token predictors.

Good example.

It’s also like saying our brains are just electric circuitry incorporated in meat. It’s true but it seems that consciousness emerges from this.

The fact that LLMs are next token predictors isn’t the interesting or impressive part. Actually my brain strictly is a black box predicting (or choosing) my next word/action/move… based on a complex existing context (my thoughts, the environment, my physical state, my senses…).

FWIW, I don’t believe LLMs are sentient, but I don’t think either that we have enough knowledge to rule it out.

imply that there is some fundamental limit on their capacities

This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI?

That seems more outrageous a claim than the one you're dismissing.

> used most of the time with a motive to downplay the abilities and faculties of AI models

Exactly. We're dancing around the real argument: there's massive amounts of influencing going on (and not only about AI.)

And what's wrong with downplaying the abilities and faculties of AI models if that's what people feel like saying? We don't call humans or animals sacks of chemicals because we believe they have moral status.

That's literally what LLMs are.

No amount of cope and anthropomorphizing is gonna change that cold, hard fact.

P.S. The perceived magic of LLMs comes from the way they cross-correlate all the probabilities of tokens on their context window. Not from their ability to "think ahead". They can't do that by design.

The LLM produces a probability distribution over the likelihood of all possible next tokens. So whatever the tokens are, "ch", "ex", etc. the next one gets a probability.

During training, real life text is fed through the LLM, and rhe "correct" token is the one actually observed in the training text. Here's a recent video walkthrough in some detail, mostly aimed at providing a deeper understanding than "next token predictor function":

https://youtu.be/GlYgs6v2YfU?is=IxVMhoCCE4N4WRVK

(Start at 15:30 for the LLM specific parts)

During training, certain tokens are more likely to lead to a lower loss function value, which is how you "win" the game of LLM output.

>I don’t understand how it can know that a token is correct.

It can't. The next token is just the most statistically probably given the context (at least in transformers). Try a very small/weak model in your own machine and more often than not it would get stuck repeating the same word or even just output garbage. Because after training and quantization (where some information is lost), that's the most probable next token. Large models can be tricked to fall in the same behavior with very very specific inputs. Still happen, even in frontier models. And they can't detect if the output is wrong.

That's why the premise in TFA is wrong, because a transformer is a next-token predictor. It literally is that. There's nothing secret or magical, it's just a very mechanical process, with a lot of matrix multiplication, normalization, a few random passes, mappings between embeddings and a dictionary of tokens, in a very very high scale.

If someone has found something that's not a mechanical, algorithmic computation and llms are doing something nobody can explain and can't even be modeled in math, I'm happy to be educated.

It knows nothing of correctness or winning. It is predicting only what is most likely given its corpus.

My poor understanding is that an LLM does not "know" either. It basically uses probabilities to predict the next word based on a large matrix of probable outcomes.

For example, say I ask an LLM, "What sentence in English contains every letter in the alphabet?"

It would respond with something like:

"The quick fox jumps over the lazy, brown [next word]"

(Assume all the words were previously guessed correctly at this point)

The LLM guesses the last word based on what it has been trained on. Let's pretend the matrix is small, and the options narrow down to something like:

1. Dog (99.9% confidence) 2. Cow (85% confidence) 3. Bag (75% confidence) 4. Crayon (25% confidence)

The machine can confidently determine the final word of the sentence, "The quick fox jumps over the lazy, brown dog" because that sentence is unique because it is often used for testing things like fonts, a fun piece of trivia, and so on.

Brown Cow is not a bad guess because it's a type of cow and a yogurt brand. Brown bags and brown crayons are also perfectly rational adjectives to describe those common items and are not a bad guess either.

However, in the context of that sentence, dog is the most correct answer because one is unlikely to have written "The quick fox jumps over the lazy, brown crayon," thus it is quite improbable to be the answer.

My understand is this is where hallucinations can often come from. If the trivia about the sentence happened to not be in common in the data set, then "brown cow" might not be a terrible guess. There is clearly something rational behind that answer, but it's not correct in the sense that it answers the question correctly nor followed the instruction properly.

I'm sure the LLMs we have are far more capable these days. In fact, it wouldn't surprise me if an LLM could check its answer by counting the distinct letters in each word to verify. Not sure though.

Again, this is just a poor example based on my understanding, but I hope it helps (and is more correct than not).

Edit: Pretend word = token. It's technically tokens and not entire words, but I didn't not want to get into tokenization of words.

I'm not sure that's a useful way to think of it.

RL post-training changes the nature of what is being predicted, basically turning it from a copying machine into a goal-seeking machine.

A base model is predicting training sample continuations (copying).

A post-trained model is now steering/narrowing the base model's predictions in directions that were reinforced by RL goals.

The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.

> I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.

People don't know exactly the words that they're going to say necessarily, but tend to start with a general concept of what they're trying to communicate and only then try to put together the words (sometimes out of order). LLMs do not begin with any sort of concept they're trying to express. LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.

> I have never liked the reductionist argument that LLMs are nothing more than next-token predictors.

I have never heard such an argument. Recognition that LLMs are nothing more than next-token predictors does not come from reductionism. It comes from simply knowing how they work e.g. from viewing the inference code.