I will be open sourcing it soon :) my code is a bit dirty (the whole system is 3 pieces. The game adapter / a websocket bridge between game and brain / the brain itself) and it runs on 2 different (local) machines currently.

Ale is what makes this work locally, I felt a little conscious about it as I am not sure if it is a novel approach or somebody comes out and claims I rediscovered BERT or something (though ale runs at 1/10 the cost of BERT).

New mediums always absolutely knock the humanity organism for a bit

Hello, author here. I was intentionally a little vague about this because this is kind of the thing that makes the whole approach work.

An LLM predicts the next token. If you're trying to predict the next token in a mathematics competition, or while playing a deep strategy game, being a much larger and more capable model helps enormously. To predict that next token correctly, the model effectively needs to model a bunch of possible future states - even if that is a second order (unintended) effect, it is what is seems to be happening.

This is basically the Ilya (and Dario) argument that prediction, understanding, and compression are the same thing (deep rabbit hole) from a few years ago.

----

In my opinion; this is a beautiful idea, but videogames do not need most of that. Videogames (and games in general) shine when character behavior is predictable, and when NPCs are a little dumb (just a little).

We already have very good small roleplaying models — Qwen 3.5 4B/9B/30B-A3B. Nowhere near frontier models at general reasoning. But they can act and write in a very engaging way. Good at roleplaying but very weak at reasoning. They just need a little nudge at reasoning...

And that's the key. The player has already expressed their intent: attack that guy, go look over there, cover me, find the key that shines and is golden, etc. A constrained world, with a constrained set of actions. Instead of asking the model to reason over an enormous space of possible futures, we're mostly asking it to map: player intent + current world state → a small sequence of plausible actions.

As for the "dump context to an LLM". It's basically. "You are roleplaying as X - you experienced Y - you like/dislike (dispositions) Z, you remember Alpha, your journal says Delta. Player orders you to do Gamma. - "What do you respond and do?"

It kind of works (as you can see in the videos I posted). I am not going against the grain, big models are better, but do we need those models for everything?

Yes, the dog has a few levers including a "boredom" counter that will trigger it to do things on its own (like roam around, eat something, chase something (eg a rabbit) if there's one around etc). All connected with its current state (Eg hungry = will probably prioritize eating but not guaranteed). Other things are mapped to game events. Deterministic in nature but its underlying values can evolve over time (both its default "homeostasis value" but also the curve of which it increases). You can think of it as letting an llm design a behavior graph, and then playing that behavior graph until it changes again.

Talking to it alleviates pressure so that mechanism doesn't fire much, but in the void-mode video on the website it chooses to chase down an elk on its own much to my surprise.

After play testing it I had to tone it down quite a bit as it caused continuity issues...

In one scenario the dog was hungry so he went and ate something that was owned (counts as a crime attributed to the player) so we ended up in jail. Being transported to jail fired a new location event and a switch from "exterior->interior" which the dog responded to by saying something along the lines of "Good. Finally indoors, it was getting cold outside". Which made for a very funny moment but a frustrating gaming experience.

Dennou Coil had this exact same concept.

It's expensive, early attempts didn't impress, and many gamers are hostile to the tech because it threatens the livelihood of game creators and impacts gaming hardware prices.

It depends on if you want samey games or anything creative.

You could have complex realistic dialogue for innkeeper #28917 in the little hamlet of ass backwards, but:

One: do you really want that? There is such a thing as too much realism in a video game.

Two: do you really want all NPCs in all games to speak nigerian business english?

Reminder for 2: https://www.theguardian.com/technology/2024/apr/16/techscape...

Because games are entertainment, but outputs generated by current AI are rage inducing and felt hostile, not entertaining. It just isn't past the "make it work" stage.

at the current token prices, far too expensive, but in medium/long term I think we will see a revolution in this field, the concepts are already there (and the expensive proofs of concept already work)

It did, look at the game Suck Up or a variety of Skyrim mods. Problem is gamers are hostile to any obvious trace of IA in their games. Also, consumer hardware is overall too weak to run a convincing llm model.

LLMs have only been really good for a couple years. You can bet that the remaining AAA studios are working hard on figuring this out in current unreleased products.