Not only that, measurably improves coding ability too (according to our tests finetuning llama 2).

Back in 2023 we sold a training dataset (~500B tokens but we could have gen more) with exactly this kind of data. The dataset was a bunch of ~1-3KB text file examples with a code snippet, then we would show some variable values when the program was at line X, then ask the LLM to predict some more print statements at line Y.

If you wanted to train on stack traces or control flow graphs we offered that too.

Our mvp used rr to attach to large programs like Excel and Chrome, then used a heuristic to filter samples for more "interesting" examples. The theory at the time was sure LLMs can learn a few information theory bits from many noisy examples, but why not spend a little intelligence on our end and knock out a couple low hanging "noise" sources. We used a prolog engine to find traces where all the information required to predict the final memory address was present in the initial program description. This turned out not to matter too much because the LLM would learn from non deterministic examples too (shrug). Eventually we ended up with a monkey patched chromium browser wandering around the internet to collect javascript examples.

We sold datasets, and we also offered an on-prem agent where you could generate training examples on demand using the spare CPU cycles on GPU nodes.

It seemed like a beautiful idea at the time but sputtered out because every serious outfit we talked to would ask a bunch of questions and seemed to come to the conclusion that they would rather do it in house.

I would bet dollars to donuts that most of the AI scraping load that people are complaining about has nothing to do with grabbing the text content of their forum and more to do with executing their javascript.

Speaking from experience: we didn't really understand V8 internals so we relied on continuous navigation and reloading the page to drive more execution rather than something smarter: snapshotting or other more fine grained manipulation of VM state. Our training data harvesting bot would reload web pages over and over to take more samples rather than do something more efficient with isolate snapshots.

Edit: email in my profile if anyone wants to talk about this. I'm feeling a wave of nostalgia thinking about this project again. Despite commercial failure, it was arrestingly/dangerously interesting.

I'd be extremely surprised if AI labs are not doing or planning on doing this already.

The same way that reasoning models are trained on chain of thoughts, why not do it with program state?

Just have a "separate" scratchpad where the AI keeps the expected state of the program. You can verify if that is correct or not. Just use RL to train the AI to always have that correct.

Yes. My business is in the code review space, synthetic data is very helpful for evals

Probably. You don't know until someone tries.

In the case of Common Lisp, yes. Other lisps (e.g. Clojure) don't really have interactive debugging, value inspection, live disassembly, or anything like the condition system (which allows programmatic access to the debugger and automatically recovering from certain conditions).

The compiler, debugger, and runtime are always present. When optimizing Common Lisp code, I found it very useful to add types, refactor structs, recompile functions, then disassemble the functions and see if the compiler generated efficient machine code. This is all natively supported by the runtime. Editor tooling does little more than create a nice-looking UI. Doing this in Clojure is not really possible, since it's hard to guess what HotSpot will do with a given sequence of JVM bytecode.

That's a really cool project. Why did you stop developing this? For me, it's a lot of work and I don't have bandwidth to productionize this

Tangent: I now want a 10 hour YouTube video where an LLM gets stuck in some debugging reasoning loop and the loop is just recorded for 10 hours.

Preferably with some lofi music under it.

This isn't difficult to do, once you have debugging capabilities exposed to the llm in vs code. You just need the proper launch.json and to expose "step back".

I added support for SSE transport (and so Cursor) in "claude debugs for you" - an autodebugger I've been working on. It works via MCP server, so you can use via Composer (Agent) in Cursor. I don't pay for premium in Cursor myself, so would be very excited about you testing it out!

https://marketplace.visualstudio.com/items?itemName=JasonMcG...

https://github.com/jasonjmcghee/claude-debugs-for-you

Extremely LLM-like writing style, do you translate your comments through one or something?

Theoretically there's no reason you need to use vscode, it's just very easy to make extensions for.