> It can't model the generative process of the humans who created those training set samples because that generative process has different inputs - sensory ones (in addition to auto-regressive ones).

I think that’s too strong a statement. I would say that it’s very constrained in its ability to model that, but not having access to the same inputs doesn’t mean you can’t model a process.

For example, we model hurricanes based on measurements taken from satellites. Those aren’t the actual inputs to the hurricane itself, but abstracted correlates of those inputs. An LLM does have access to correlates of the inputs to human writing, i.e. textual descriptions of sensory inputs.

You say this, yet people such as Helen Keller suggest that a full sensorium is not necessary to be a full human. She had some grasp of the idea of colour, of sound, and could use the words around them appropriately - yet had no firsthand experience of either. Is it really so different?

I think “we” each comprise a number of models, language being just one of them - however an extremely powerful one, as it allows the transmission of thought across time and space. It’s therefore understandable that much of what we recognise as conscious thought, of a model of the world, emerges from such an information dense system. It’s literally developed to describe the world, efficiently and completely, and so that symbol map an LLM carries possibly isn’t that different to our own.

It seems to me that the human authors of the training text are the ones who have created the “world model”, and have encoded it into written language. The llm transcodes this model into word embedding vector space. I think most people can recognize a high dimensional vector space as a reasonable foundation for a mathematical “model”. The humans are the ones who have interacted with the world and have perceived its workings. The llm only interacts with the human’s language model. Some credit must be given to the humans modellers for the unreasonable effectiveness of the llm.

But if you squint then sensory actions and reactions are also sequential tokens. Even reactions can be encoded alongside input as action tokens and as single token stream. Anyone tried sth like this?

> The "world model" of a human, or any other animal, is built pursuant to predicting the environment

What do you make of Immanuel Kant's claim that all thinking has as a basis the presumption of the "Categories"--fundamental concepts like quantity, quality and causality[1]. Do LLMs need to develop a deep understanding of these?

[1] https://plato.stanford.edu/entries/categories/#KanCon

But isn't the distinction between a "passive" and an "active" model ultimately a metaphysical (freedom of will vs. determinism) question, under the (possibly practically infeasible) assumption that the passive model gets to witness all possible actions an agent might take?

Practically, I could definitely imagine interesting outcomes from e.g. hooking up a model to a high-fidelity physics simulator during training.

People around here like to say "The map isn't the territory". If we are talking about the physical world, then language is a map not the territory, and not a detailed one either, an LLM trained on it is a second order map.

If we consider the territory to be human intelligence, then language is still a map but it is a much more detailed map. Thus an LLM trained on it becomes a more interesting second order map.

Animals could well use an autoregressive model to predict the outcomes of their actions on their perceptions. It's not like we run math in out everyday actions (it would take too long).

Perhaps thats why we can easily communicate those predictions as words

We can't see neutrons either, but we have built various models of them based on indirect observations.

> embedding a pre-trained KGE model into a transformer model

Do you have any good pointers (literature, code etc) on the mechanics of this?

> At the limit of prediction becoming as good as theoretically possible given the input data and model size restrictions

You are treading on delicate ground here. Why do you believe that sequence models are capable of reaching theoretical maximums?

One of the things about offline imitation learning like OP or LLMs in general is that the more important the error in their world model, the faster it'll correct itself. If you think you can teleport across a river, you'll make & execute plans which exploit that fact first thing to save a lot of time - and then immediately hit the large errors in that plan and observe a new trajectory which refutes an entire set of errors in your world model. And then you retrain and now the world model is that much more accurate. The new world model still contains errors, and then you may try to exploit those too right away, and then you'll fix those too. So the errors get corrected when you're able to execute online with on-policy actions. The errors which never turn out to be relevant won't get fixed quickly, but then, why do you care?

Also things are not static in the real world.

I can't imagine training took more than a day with 8 A100 even with that vocab size [0] (does lightning do implicit vocab extension maybe?) and a batch size of 1 [1] or 64 [2] or 4096 [3] (I have not trawled through the repo and other wordk enough to see what they are actually using in the paper, and let's be real - we've all copied random min/nano/whatever GPT forks and not bothered renaming stuff). They mentioned their dataset is 120 million tokens, which is miniscule by transformer standards. Even with a more graph-based model making it 10X+ longer to train, 1.20 billion tokens per epoch equivalent shouldn't take more than a couple hours with no optimization.

[0] https://github.com/keyonvafa/world-model-evaluation/blob/949... [1] https://github.com/keyonvafa/world-model-evaluation/blob/949... [2] https://github.com/keyonvafa/world-model-evaluation/blob/949... [3] https://github.com/keyonvafa/world-model-evaluation/blob/mai...

It's a bit unclear what the map visualisations are showing to me, but I don't think your interpretation is correct. They even say:

> Our evaluation methods reveal they are very far from recovering the true street map of New York City. As a visualization, we use graph reconstruction techniques to recover each model’s implicit street map of New York City. The resulting map bears little resemblance to the actual streets of Manhattan, containing streets with impossible physical orientations and flyovers above other streets.

Makes me think of an interesting related question: how aware are we, normally, of our incoherence? What's the phenomenology of that? Hmm.

In the paper 'Hopfield networks are all you need', they calculate the total number of things able to be 'stored' in the attention layers, and it's exponential in the number of parameters. So essentially, you can store more 'ideas' in an LLM than there are particles in the universe. I think we'll be good.

From a technical perspective, this is due to the softmax activation function that causes high degrees of separation between memory points.