> Is success mostly the result of it being able to test its output reliably, and of how easy it is to set up the environment for this testing?

I won't say so. From my experience the key to success is the ability to split big tasks into smaller ones and help the model with solutions when it's stuck.

Reproducible environments (Nix) help a lot, yes, same for sound testing strategies. But the ability to plan is the key.

жопа -> jopa (zhopa) for those who don't spot the joke

Claude will also tend to go for the "test-passing" development style where it gets super fixated on making the tests pass with no regards to how the features will work with whatever is intended to be built later.

I had to throw away a couple days worth of work because the code it built to pass the tests wasn't able to do the actual thing it was designed for and the only workaround was to go back and build it correctly while, ironically, still keeping the same tests.

You kind of have to keep it on a short leash but it'll get there in the end... hopefully.

You can either use the orchestrator mode and tell it that it must run a subtask that reviews changes after every successful sub-task is done (works in RooCode, I’m guessing KiloCode should also have the feature).

Or you can just switch the models in a regular conversation and tell one to review everything up until now, optionally telling it to get a git diff of all the unstaged changes.

My own (very dirty) tool, there are some public ones, probably I'll try to migrate to one of the more mature tools later. Example: https://github.com/ruvnet/claude-flow

> is this possible with those?

You can always write to stdin/read from stdout even if there is no SDK available I guess. Or create your own agent on top of an LLM provider.

Well, just told it to use gdb when necessary, MCP wasn't required at all! Also it helps to tell it to integrate cpptrace and always look at the stacks.

MCP is more or less obsolete for code generation since agents can just run CLI tools directly.

That's Multimodal Dynamic Launcher. A very nice thing actually, a scripting orchestrator.

That is perilously close to the usual:

"AI DID EVERYTHING IN A DAY"

"How do you know it works?"

"... it just looks like it does"

Like when I ask AIs to port sed to java, and it writes test cases ... running sed on a CLI and doesn't implement the full lang spec no matter how much prompting I give it.

Claude won't handle a project of that scale. Even with Java 7 modernization project, which is much simpler than full javac translation, I constantly hit context limits and Claude throws things like "API Error: 400 {"type":"error","error":{"type":"invalid_request_error","message":"messages.3.content.76: `thinking` or `redacted_thinking` blocks in the latest assistant message cannot be modified. These blocks must remain as they were in the original response."},"request_id":"req_011CVWwBJpf3ZrmYGkYZLQVf"}" at me.

I've been looking for a way to decouple legacy parser from the rest of the compiler, plus create a way to dump parser output in a readable form. Unfortunately, the coupling is too tight there. In my own compilers all the outputs of all the phases are serializable.

In the end I've just reanimated the original parser generator and progressed to full Java 7 syntactically (-att5 branch), but there are some major obstacles with bytecode.

https://github.com/atgreen/openldk

Jikes didn't.