One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup.
Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something like Pi.
It does seem a bit odd to say "We tested all of these bicycles with the same rider" when one of the test cases is actually a bare high-end frame with no components on it.
It's even weirder given that another of the test cases is essentially "We put parts on the frame (OhMyPi) and it went faster!"
I'm not sure your analogy holds, because here the optimization metric is clear and unanimous: every one wants max pass rate at min costs.
With the bicycle, some may prefer comfort, others speed, others offroad, etc., so it would not be obvious which one is "best".
I think it's still a useful data point. For example, omp, which is pi with some default extensions, scores worse. I do agree that adding more configurations of Pi would help though.
Interesting to see harness benchmarks for coding agents. The same problem exists for conversational and data agents but I dont see anyone benchmarking them yet. Seems like manual-spot checks are the norm
Testing it against Kimi is potentially skewing the numbers massively. Kimi has a number of quirks that requires behaviours that e.g. Claude or GPT doesn't.
Harnesses that are built around needing to work with "weird" models will need to deal with that, such as Kimi's tendency to get stuck in tool-call loops.
Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that.
Claiming on the blog that this gives Kimi Code no home field advantage seems like a dicey assumption. I haven't dug into the newest Kimi Code much, but the older Kimi CLI included several tools that were clearly specifically aimed at working around that behaviour - when I copied their checkpoints and "dmail" mechanism into my own harness, the performance with Kimi improved dramatically, but it made zero difference against Anthropic models.
That doesn't make the data worthless - it's clear you shouldn't use Clade Code to work against Kimi. But it does significantly limit the utility of it.
This is great. People always talk about how important the harness is and yet we have so few harness benchmarks. Agree with sibling commenter that ‘harness x model’ is needed.
People always talk about how Cursor harness has some secret sauce; would like to see how that one stacks up.
Thanks for making this and filling a real gap!
There really should be error bars on those measurements. With just 30 samples, the 95% confidence intervals for accuracy should all be more than 35 percentage points wide, so comfortably overlap. For cost it's harder to say, because outcomes aren't constrained to {0, 1}, but I also expect a lot of variability there.
Putting only the tasks and results in the repo is a poor decision. These conclusions would be far more credible if anyone could re-run the benchmark.
Is Github Copilot (integrated in VS Code) not a thing? I use it all the time and don't know what more I could wish for.
Disclaimer: I don't let agents run on huge tasks for hours. Almost all tasks I give them are done in under 30 min.
I also feel Copilot is not getting the attention it should considering its user base and its importance in the enterprise context. There is a lot to be learned and I found https://news.ycombinator.com/item?id=49256057 (mitm copilot investigation) interesting. Playing with Copilot and asking it what was the biggest part of its standard payload I was surprised to learn that VSCode extensions plug in their MCP servers without me being much aware of it.
agy has fewer users than grok, both are ~1% based on some surveys I've seen, not every harness needs to be evaluated
Incredibile result for pi. The ratio quality / complexity of the harness make me think that all other harnesses are very bloated.
Same, I want to bench PasClaw on it.
I just started using Hermes and it's pretty good. Guess I'll try Pi. I'm hesitant on DSH.
Very interesting. As models become commodities, the harness will be the next optimizing game.
I've been confused a lot why there isn't a benchmark to measure a harness's performance rather than the model's one. Now there it is.
Good start, but needs to be harness x model to be useful. 3 top harnesses x 3 top models would be more interesting.
Version 1.1 will add more harnesses and evaluate a broader set of models. Rather than holding the model fixed, we plan to evaluate the full harness × model matrix to reveal interaction effects (which harnesses work best with which models) and produce a harness-model compatibility matrix.
It's nice to see time reflected here.
Deepseek is fuckin fast! Seems like the bigger the task, the faster it gets, which is kind of unfortunate because nobody is going to be doing 17 benchmark passes on a $50-100 task. I'm assuming the brief pause before it avalanches out 16kb of text at upwards of 250tk/s (multiples beyond anything resembling a comfortable reading speed) is some sort of workload evaluator directing sessions to individual/multiple cards, occasionally waiting for what it thinks is best to become available.
I couldn't believe it at first, it shit out a damn fine multithreaded physics simulation fabric (integrated into a massive codebase, tests passing) in under an hour. Anything I could find online says they average like 80 but my logs average ~3x that.
Software is constrained when you write it. Agents have to be constrained while they run.
The Exo harness is the most interesting one in the benchmark, as it has the potential to complete tasks with fewer turns and cost.
Harness wars are the next browsers wars!!
But wow, opencode is that bad?!
Forgive my stupidity but how do you run Claude Code with non-Anthropic models?
Claude Code supports the base URL env var so you could tell it to talk with any LLM API endpoint that receives the Anthropic style request format, e.g. DeepSeek.
Claude Code allows you to change the config such that you use the harness with different models - you can actually ask a coding agent to configure it for you
Codex is the best harness to me because of its GUI and subscription.