so then what I’m really interested in are benchmarks of OSS models vs closed source frontier models, using Claude Code as a harness

> Minimal: Just some 100 lines of python for the agent class (and a bit more for the environment, model, and run script) — no fancy dependencies!

It is way more than 100 lines. Why keep advertising something that is no longer the case?

Minimal agents also allow room for more focused add-on tools/infra. I'm working on a context management layer[1] and it's very difficult to do well. In our benchmarking, we've observed simple harnesses like Stirrup[2] outperforming more elaborate ones.

[1] https://www.induction.ai/docs/context-management [2] https://github.com/ArtificialAnalysis/Stirrup

I'd really love to see more studies about effectiveness of AI in general. As in, what works best and how to use it and such.

Because I feel that the technology and space is - so - hyped and fast moving that a lot of cultish feeling rituals seem to pop up, none of which are backed by evidence. Anthropic openly recommend giving the agents.md file an architectural overview of the code, and the one time this was studied they found the opposite - that the agents.md file is best for concrete commands about how to build stuff and such, and - not - huge overviews. This was, and still is, the official recommendation from Anthropic as far as I can tell.

And then there are the benchmarks, how feel vague and not concrete, and everyone kind of knows they're not the best cuz you can't just assign these tools one fixed number ( for multiple reasons ), but everyone still looks at them and compares them.

People share skills and superpowers and plugins and mcps and very, very few of them have and kind of proof they do much at all.

It all feels a bit weird to me, and I've been on the lookout for exactly these kinds of studies more lately, because I think having this research, even if not done on the exact newest models or not the exact, newest thing, are still - vastly - superior to the alternative.

The focus of the study was the different harness approaches and how they scale across model sizes. The fact that they used any particular set of models is irrelevant.

I have to say, I am starting to hate this line of reasoning. Yes, LLMs move extremely fast and a lot of improvements are done in a short amount of time.

And there might be a point to these arguments, vaguely. However:

There never seems to be - any - kind of counter example or reasoning behind the rationale. You have an in depth and empirical study, done by researchers who, frankly, now their shit (most of the time)

And on the other hand a random internet comment saying "nope" because...the models aren't the latest.

If the latest models really would make a difference, you should at least provide some kind of evidence towards that. As it stands though, every time these comments come up this is missing.

There seems to just be a vaguely defined understanding that "everything changes all the time, and nothing you ever research is transferable to state-of-the-art models"

Which brings me to my second point about these kinds of arguments:

LLM models often - aren't - fundamentally different. Yes, they are vastly more capable. And yes, there are emergent properties. But at their core, they function very much similarly. And for quite a while now, there have not been any of these drastic changes we saw when LLMs first become "good enough" for agentic coding.

I am tired of dismissing empirical evidence and studies every. single. time for reasons without evidence and seemingly a vague sense of "no, but my model is different"

How do Nemotron models + mistral compare to "the current frontier" in your experience?

I've been pushing the context window well into the 400-600k+ token range lately (mostly Opus 5). I prefer not to since I'm aware of context collapse, but I've been leaning that way lately.

Lately, I've been finding that the game of telephone of handoffs causes more mistakes than just letting the session run longer. Not too mention the wasted time waiting for agents to poke around as they bootstrap a new session from the handoff.

Of course, I still have handoffs for the large-scale plan being accomplished, but I've been having better results letting an agent finishing what it starts. The game of telephone is a painful waste of time when it goes wrong, which is far too often for me.

I think it’s sort of self-defined. If a model is able to use bash well enough to not need specific tools.

The research seems to agree with you, though. The paper calls out that for “bash capable” models, adding tools to do things bash can already do doesn’t improve performance.

Vaguely the same result as RAG. Unless you’re in specific domains, you won’t beat handing the agent a shell and grep.

I think it just means “is there some parser program that consumes the LLM token stream and spawns shell processes with the detected command strings”.

> bash-only tooling has higher success for bash-capable models, compared to using predefined tools for models that aren't good at bash?

No. The conclusion is that:

bash-capable models + bash-only tools > bash-capable models + predefined tools

In other words, MCP was just a bunch of bullshit that maybe helped a little bit until the models got good at bash, and now it's basically useless.

"Everybody knows foul air causes sickness."

"Duh, of course Mars has canals."

Testing the "obvious", "duh" things is incredibly valuable science. It provides a more solid foundation on which to build because it reduces the assumption space.

> using the harness + model made by the same lab, has vastly better experience compared to more "general purpose" (for the lack of a better term) harnesses

not really, there was a recent benchmark with claude and codex and it showed no difference in ability with a harness like pi agent compared to their native harnesses, pi was in fact cheaper per task.

It might be 'duh' but it means we need a formal list of what each model is good at, and to pick or change harnesses to closer fit the model. Like an llm recipe book. Not just for remote models, but also local ones where how you run the model is critical too.