4 days ago
I might have missed it in the readme, but what hardware did the training take place on? This is really neat.
4 days ago
For the world model training I've used an RTX 4060, but for the agent playing the game that was directly on the pico using stream-ac
4 days ago
This is incredibly cool! Fascinating to see looped transformers be necessary for mini-Doom, but not for simpler games.
4 days ago
Very cool. I didn't know NNs were capable of such determinism, that's very interesting. Is this a similar effect to overfitting? Also, it's the same model for all the games, right?
4 days ago
> Also, it's the same model for all the games, right
It's a bit unclear but it looks like different models, since they all have different parameter and MAC counts. It would be much more impressive with a single world model that could maybe dream up new games inbetween the games in the training set.
4 days ago
It's a self contained net for each game but the data used for training doesn't overlap with the validation. I think it would be easy to merge them into a single model
4 days ago
Is the actor trained on world model's mistakes as well? If so, it should learn to force hallucinations that force it into an easier state.
4 days ago
Yes, there were some hallucinations, e.g: in breakout hitting a corner would destroy more blocks than expected, in doom-lite it could shoot from miles away and get a kill. The fixed was add some noise to the input frame and keep the target clean.
4 days ago
Not the poster you responded to... Your Singleton hack was clever. In some ways it brings game design back into the equation, in that it mirrors the special cases that most game code has to deal with. You alluded to something in the docs that I was curious about, which is state. From reading it, I gathered that both agents interpret the direction of the Pac Man based on the position of its mouth in the previous frame. And one of your graphics seems to imply that the Doom state of where you are and which direction you're facing is magically grokked by the agent reading the map in the corner. Is that so? In that case, I wanted to ask, in pong or breakout, what is it using to figure out the direction and velocity of the ball? Faded prior shadows in other cells, maybe?
4 days ago
No, not shadows. I use the heading as a token, in Doom the minimap has a player token and the facing (the current 3D view), I tried using only the frame but it was harder to learn only from the wall patterns. So the agent isn't magically grokking position, its using the map. As for the Pac Man, it's based on mouth and has tokens for "keep going" or N/E/S/W direction. For breakout it uses 4 ball tokens, one per diagonal and it moves one cell per step in that diagonal, and for pong the ball has 6 tokens (combiantion between [up, level, down] with [left, right], e.g up right, level down)
4 days ago
4 days ago
Honestly fascinating. I almost feel like you'd be better off avoiding any discussion of the fact that this runs on a Raspberry Pi Pico 2 W because it takes away from the crazy world model capabilities (of course, the fact that this does run on a Pico 2 makes the whole thing much more impressive).
There's gotta be a good way to incorporate this logic into a larger, more "traditional" game in a meaningful way. Seems like an approach for procedural generation that could solve some of the issues usually found with actual procedural generation.
4 days ago
I honestly believe that these neat projects (smallest possible, locally run LLMs) are the real future of AI on the long run.
3 days ago
There is still immense potential for optimization. Future local intelligence on your phone.
3 days ago
Yep, I agree. You can already run them locally on a top-shelf gamer PC with decent token throughput: https://github.com/Niko1221/Strata That's today.
As for tomorrow, consider that literally every chip maker is working on better and better NPUs to boost local performance (Ryzen AI, Intel Ultra, Apple M7, Broadcom PON, Huawei Ascend, Snapdragon X Elite, etc.), and imagine optimized models running on these, yeah, insanely huge potential.
As of 2026Q4 I wouldn't bet on any cloud AI, that's for sure.
3 days ago
This is really interesting! How hard would it be add this into like, sort of a traditional game?
3 days ago
I guess it depends on how you would add it into a game, I think that would be pretty hard to not have hallucinations if it's something like an open world, but otherwise I think it's doable with enough time and effort.
4 days ago
If this runs on a PI... can it run on webgpu ?
4 days ago
The core of it uses rust with no_std and on the pico we use embassy-rs with no os. At the same time, the core can be build for webassembly but it doesn't use webgpu.
4 days ago
Unlikely, Raspberry Pi Pico 2 W is a microcontroller, not a computer. It runs either bare metal or an RTOS.
3 days ago
[flagged]
4 days ago
[flagged]
4 days ago
[flagged]
4 days ago
[dead]
4 days ago
[dead]
4 days ago
[dead]