> Robots: they're just like us!

Hah I dunno, I think most human players wouldn't find that very fun and would instead take that as an opportunity to refactor the factory for optimized performance (i.e. the core gameplay loop)

i think its more of an effect of factorio's hidden information.

youre not really sure how to set up some of the logistics/how much of what you will need where ahead of time

It certainly sounds like an actual factory in my experience

Where are the recordings of these games? I know this kind of stuff must be happening, but it seems oddly difficult to track down the actual recordings, which feel like the really fascinating thing people should be publishing if they’re putting in the time and tokens to do stuff like this.

Like I want to see what Pokemon strategies it uses: Does it try to brute force with its starter? Does it try to load all of its HMs onto one or two Pokemon? Does it use Flash? If it needs to catch a new Pokemon to beat a gym leader, how much does it remember about which species it encountered where and what their levels were?

If you reading every wiki, forum, dev blog/patch note, watching every video, etc before playing, I’d be more surprised if you could NOT play a game.

How do they feed the game screen to the models? I imagine the models are too slow if you send 60FPS, but I feel like that’s also the case if you maybe feed it 1FPS? If you go any slower, you’ll miss too much info?

Your last sentence mirrors my model choices recently. Opus can do things very well but Sol can do them well for less than 1/4 the price.

Opus and claude models have always been lagards in vision. It just hasn't been a focus like coding for anygropic. Gemini and ChatGPT have always been much stronger in this area

I would like to see it play Slay the Spire (or 2) and beet it consistently. Mainly to see what strats it uses. I suspect it wouldn't have too much trouble.

Some of these Games are definitely in the RL environments.

Do you have link for the VII bonus boss run?

Not sure about LLMs but I yearn for someone to train neural nets to play the NPCs in a shooter game well but without having aimbots or seeing through walls - basically a “true” version of F.E.A.R. AI where they play simulated squads that overwhelm the player with various tactics and adapt to their playstyle. Or even just make increasingly complex state machines and goal oriented systems which might be more performant but still encode the main rules.

I have been playing with this. Having tiny models provide more contextual information when an NPC speaks is cool but the "small enough" models tend to go off the rails.

It's fun if you have a handful of people playing, but it would quickly go towards negative reviews if you tried to sell it.

It's a funny situation at the moment because most attempts are just sort of shoehorning LLM chat into a game that doesn't really need it, and then gamers quite rightly mock the results which don't really add anything to the game.

I think there's a bit of a lack of knowledge of game history on both sides, because we already know it can be fun when a game is designed around it. It worked in Façade (2005!) and it worked in Starship Titanic (1998!!), and both of those could have worked even better with the technology we have today.

One type of game that seems like an obvious one to me, is something a bit like The Sims except that the characters' decisions and speech are LLM-controlled. Maybe one of the characters is player-controlled, so you can interact and talk with everyone else. Or to describe it from the other direction, it'd be a kind of interactive fiction like AI Dungeon invented, but grounded in a physically simulated setting.

There has to be a AAAA studio building the new, greatest MMO, that also utilizes LLMs as part of the game loop, surely?

(Not sure how prohibitive it would be to do anything interesting with local LLMs of any kind in a big game, given how many platforms/setups you have to make the stuff run smoothly on, and the sheer amount of RAM they would bind)

I built a factorio "sidecar" LLM harness that would look at production stats every few minutes and identify issues. Even sonnet was good enough at finding potential things to alert on. coal production dropped and then iron production dropped two minutes later --> top potential issue.

Not useful enough that I actually play with it, but it was capable enough to be useful even just with sonnet.

With the aid of SOTA LLMs, you can encode specialist intelligence into computer programs without going through the roundabout route of training small LLMs. And even when you do go the route of ML, it doesn't necessarily have to be an LLM or even a Neural network.

I hope to experiment with using classic tabular reinforcement learning in game AI one day. Imagine a framework that makes setting up small Q-learning tables easy, and you can just sprinkle dozens of Q-learning algorithms throughout your traditional AI scripting.

These old basic algorithms often have mathematical proofs that they will converge to optimal behavior, and they do so fairly quickly. Turning the complex world of the game into a Q-table would require some creativity and judgement, but is not that hard I think.

In the end we would have a controllable AI that actually does adapt. That's my hope anyway.

There are already AI-based mods for games like Grand Theft Auto V that make NPCs talk back and follow your verbal commands: https://www.youtube.com/watch?v=3_Eng-F4akI&t=106s

A little unpolished because it was bolted on by fans and not the original game developers, but otherwise very impressive.

I cannot wait for NPCs having a casual conversation. Doesn't seem too hard, as current chatbots are already role-playing.

I understand https://neal.fun/infinite-craft/ uses an LLM under the hood each time a novel combination happens.

This... Hopefully inference can become so cheap games can have simple LLM's integrated into them. Imagine an RPG with LLM's and with rich worldbuilding; like Fallout NV with LLM's. Honestly, scary!

Fable seems to be a potential franchise to do all this; imagine a "living fantasy world" where everything truly unfolds based on your decisions, giving NPCs all the functions/actions that a player has.

SS13 but the ship's AI is actually an AI instead of a player in disguise

i think hooking in an llm to dwarf fortress for turning the very mechanistic descriptions into prose. same with the adventure mode talk. instead of a constant "well weve got beasts" an llm could read a bunch of the language tree to make a unique introduction for each.

same with the df artworks. get kitfox to design some art styles for dwarves, elves, goblins, and humans, and then instead of just reading

"this is a fine engraving of a dwarf. it concerns the events of 102 when ulrist fastbuttons was elected mayor. ulrist fastbuttons is sobbing"

SOTA LLMs would probably be fun opponents in turn based 4x.

I would love the revival of the RTS genre with better computer enemies.

impossible, unfortunately, because LLMs suck at creative writing. without handholding, they are incapable of producing anything but cliche-ridden purple prose for young adults. you can feed it the first half of a book with lots of sex, violence, and profanity, ask it to continue the story, and it will degrade into safe slop within two chapters.

both American and Chinese labs aim their models at coding and knowledge work, so they penalize adult themes in post-training.

>These models have read and have access to every guide and video ever published on the subject.

This has been true for every model since 2023. It's not nearly as big of a factor as people believe it is

Isn’t the point of arc agi 3 to measure general game intelligence even if it’s not a AAA game? Frontier models have obliterated that. And they do keep most of it private so no gaming there.

The models these days have 3 components that is for sure going to get better and better

1. Very good vision (it can identify almost anything in a picture) 2. Keyboard and mouse actions (slow but it does it well) 3. General world understanding (this one is toss up but given Astra or Fable, it surely does well even if it takes a while)

Now combine all these with specific RL environments that may use specific games which also generalizes to an extent and we have a system that can only get significantly better from here.

As for speed, I’d say it’s a question of how much money you are willing to throw (like Cerebras for example).

Good point. There are also a huge amount of open source calculators, planners and other tools that these models have undoubtably been trained on [0]

There are full blueprints for factories online available to copy-paste in its training data [1]

I wouldn't be surprised if you can beat the game by just copying a bunch of blueprints from the web and chaining them together.

This is an old game relatively speaking and its been optimized to hell.

0: https://factoriolab.github.io/2.1

1: https://factorioprints.com

These are really two different challenges - playing the game for the first time with exploration, and beating the game with all the prior knowledge.

Shhh you can't say anything anti-hype here.

What if the player's inventory has a higher throughput than belts?

Part of the reason players try to automate manual feeding is the annoyance factor, which AI doesn't care about.

If you don't have enemies, you have unlimited time. Doing it thru player is what it learned and wasn't pressured to do better (it has no fatigue as opposed to humans).

Coding agents are bad at architecture, Exhibit no. 12057.

It's just easier to do manual stuff when the complexity of scaling the automation becomes cognitively demanding. A new player does mostly everything manually while experienced players have a natural intuition for planning and scaling the automation.

To be fair, I do do the same.

The biggest thing the bugs introduce from an LLMs perspective is a necessity for realtime response. Without them, the LLM can take as long as it wants to decide and do its next operation on the factory. With them, not responding fast enough results in death (particularly early game where a lot of biter combat is player driven). I presume it doesn't do great at this.

That was exactly my thought when reading the headline, before even looking into the article. Sure it's still somewhat complex to get to a rocket launch, but without the biters where is the challenge, other than complexity? It's like a big excel sheet with conveyor belts, pipes and electricity poles at that point only.

The nests stop being a concern once you have significant production and a few hundred logistics bots. Then you just build walls beyond the pollution range and place a gun turret every other square (or lasers at level 10 or above). I only use nauvis for uranium now because my Aquino production is weak (fusion). Late game you rarely go back to nauvis, just send your fleet of platforms to resupply reactor fuel autonomously.

The issue is most of these play-throughs happen on “normal” response speed. Way too slow for these LLMs to send back and forth and process this many tokens in a quick enough reaction time.

I’ve yet to see running these LLMs on Ultra-fast (8x speed on Sol). Wondering if that’s fast enough to do real time

"AI found decompiling game to fix a bug after being instrcuted to beat it" sounds like an IPO friendly headline.

I've been watching streams of 9 different models playing Factorio together [1] and it seems the answer is — not very well at all. They run into the same issue they have elsewhere in the game as well as in coding tasks: inability to create structured plans for the future.

So they get into a cycle of: 1. Get attacked 2. Build a few turrets 3. Get attacked. They don't consider buffing up their defenses or going out to deal with the threats or preemptively placing turrets when they expand. Not without human probing.

Also, they had the same issue with hand carrying resources around as some other comments here mentioned Sol did.

[1]: https://factorio.nickwinter.net/

To them the player character is just as good as a belt for transporting materials.

The player character actually does have an extremely high capacity so if you are constantly running between chests your factory will still be reasonably high throughput.

> In some sense it proves to me that llms are not just copying and are actually creating a bespoke solution.

Is the weirdness of the designs meaningfully beneficial in any way that the AI has figured out, or could it for example be that the training data the LLMs learned from are too different from the type of data it would need to learn from in order to interface properly and play the game better? And then because of that incompleteness, the LLM has to hallucinate a lot and it results in weird outcomes.

I think one of the problems is that AI has no sense of laziness or time. Unless it's in their training data or prompted they're not going to bother automating something

Sounds like an experiment I'd like to give a try--what integration layer did you use to get LLMs to control the civ?

To be fair, Deity on Civ games is brutal. I'd be interested in where the cutoff is.

Doesn't the AI (the computer players, I mean) "cheat" at that high of difficulty level?

> Having an LLM gardener that could expand outposts to connect to the main network, or just pop in to observe if there's any major problem to report

Why even play the game?

https://factorio.nickwinter.net/

(Author) I did this on the 'Plus' plan over a series of 5h quota segments so $20 for me, but at API pricing the tokens would have been $52

Who determines "over investment"

I'm sure you could if you put your mind to it. It's not so bad, on peaceful mode. Just takes time.

Boring though without something to attack you imo

[dead]