Right now, sol-xhigh is my favorite model. I feel that Opus 5 is dumber than 4.8. Fable is too expensive to do anything (limit of $50, started a prompt at $25, ended up at $75, is bullshit, but at least it's "free credits").

DeepSeek is okay for random API-based stuff, as it's cheap.

Local open models running on a 5090 are hit or miss. I feel that most GGUFs/quants are awful...

One thing I've really noticed with Luna Max is its speed. I've got a review script setup on a custom Pi extension. Luna finds some issues/some false positives, while Sol finds issues but disregards false positives. The biggest thing is Sol finishes in about half the time.

Curious about what you found. I agree harness for browser automation is vital - I work on https://libretto.sh

Exactly! I am not opposed to cloud-based models, but I do only stick to open-weight models because I know I can move my whole stack to local (given enough hardware) and continue development without any of the LLM interaction contracts being broken.

I would like to see some development where proof of authenticity certs are generated alongside the actual output of the model. Prove to me (or at least claim to me liable to breach of contract) that this output was generated by FP8 DeepSeek V4 Pro 0813. Not some cheaper quantization of the model.

When it comes to quality of outcome, since at least Feburary, the harness has almost equal, if not more weight than the model itself. It's no longer "which model is the best?" it's "which model + harness is the best?"

I get drastically different tool call failure rates using Claude SDK vs OpenCode using Qwen 3.6 models

I don't think so. there is a lot of tools with similar usage, some harness even bring their own internal tools for accurately manipulation.

also, even if some models claim that they have full 1M context window, some only work effective with the head or tail of the window, a proper harness tool will know about the limitation of the model and act accordingly.

then also the output format, the tool calling syntax, the quirks and gotchas of each model.

it is not simple as just throwing everything at the model, especially when your project has hundred of files or so.

Because complex problems can be decomposed (a skill in itself) into easy parts and hard parts; and the hard parts are almost always bottlenecked on understanding concepts and principles (i.e. things that are either in a model's weights, or not), not on having certain facts available. Models can solve complex problems insofar as they can decompose those problems, and have learned the concepts and principles relevant to approaching the hard parts of those problems.

Whereas tool-use isn't a capability problem, but a context problem: the thing that makes models fail by default is that they have no idea, when first summoned out of the aether, what kind of conversation they're having, who it's with, what that person is trying to do, what tools they have available, and how those tools can be invoked.

Think of the difference between how you'd respond to a casual programming question asked by a person sitting next to you on a flight, vs. a programming question asked of you by someone you're pair-programming with with your IDE open in front of you. Now imagine waking up blind and deaf and needing to discern which of the two situations you're in. LLMs know how to approach both of these problem-contexts (and more besides), but they need to be given context to know which problem-context they're in (and everything else about that problem-context: which IDE they're using, which OS it's installed on, what other tools are installed+accessible, etc.)

And before you say "but why can't they just experiment to figure these things out" — if you think about it, knowing how to interface with a shell and an IDE are bootstrapping requirements for any kind of experimentation, in about the same way that "knowing how to open your eyes and move your head" is a bootstrap requirement for a human gaining information about the world around them. These capabilities are necessary to explore the world to "discover" and "probe" other capabilities.

---

Also, a lot of the work LLMs do "needs" (i.e. is heavily improved by the use of) some kind of structured scratchpad, that they have been trained to manipulate and "look at" through tool-use. Even for a human who could accurately visualize a canvas based on a coordinate system, you still wouldn't expect said human to succeed at the pelican test if they had to write the SVG entirely in their head and then write it out sequentially with no rewinding to fix mistakes. You'd expect them to ask for at least a whiteboard, if not a text editor, to be able to write and rewrite the SVG XML.

(Really, they'd ideally want to run the SVG and look at it to see how close it is, and optimize that way. I'm not sure if we're letting LLMs do that part in the classical pelican test. It feels like that would vaguely violate the "zero-shot"-ness of the test, though I'm not sure if we're currently considering a conversation to be "zero-shot" if it involves the model iteratively interacting with a third-party system [such that there are repeated model -> system -> model conversation turns] but holding off to responding to the user until they think they've fully solved the problem.)

---

And also, on a lower level, all of these external capabilities are getting exposed to the LLM through MCP. Models can and do understand how to speak MCP itself. But there's no standard for how a given harness's capabilities (e.g. "execute command line in new shell session", "send patch edit command to active tab in IDE", etc) should be modelled to be exposed through MCP, either in their encoding or in their semantics. There's no MCP equivalent of WASM's WASI meta-standard, such that models could learn these specs and "assume by default" that things work like them until told otherwise; and nor are there even open harnesses that LLMs could learn about during training, and through them, learn some de-facto MCP-endpoint specs. Instead, there are mostly just proprietary harnesses, that hide all that info from public access, sharing it only with the LLM during inference, and even then, only at the moment the LLM needs it.

the single shell command is the terminal bench.

This. The same goes for “skills”, skill type “subagents” and other bullshit - powerful models don’t need any of that anymore I noticed.

Not exactly. See:

https://alexzhang13.github.io/blog/2026/mgh/

For me, flash 0731 was much better in omp/opencode than in Pi.

Anyway, it might be so that they are rolling out deployment. There haven't been an official announcement post yet (this submission is a link to openrouter). Some people have been saying they are getting results worse than GLM-5.1, that's obviously broken.

You can't even run a benchmark without a basic harness, of course the harness has some effect.

Their leaks would confirm this sort of attitude. They're not trying to become the top player or anything like that - just working to play their part in pushing LLM tech forward and going from there. It was quite refreshing from the 'here's how we're going to dominate the world' nonsense. It's undoubtedly the same attitude that just lets them shrug and cancel the fund raising round after the leaks came from said funding round.

Actually, yes. I just didn't know about Grok's release because they aren't on the front page of HN.

It still matters as a point of comparison until other providers come online. If the consensus price from other providers is much different that can be compared then. But for now we have $0.435 / $0.87 for v4 Pro 0813 (with increase announced but we don't know the new pricing), and $2 / $6 for Qwen3.8-max. So until we get other data points that is what we have to look at.

We have a first-party figure from the system card [1]:

> Mythos 5 reproduced 83.8% of targeted vulnerabilities on a single try, and produced at least one crash in 99.4% of tasks. This is comparable to Claude Mythos Preview, which reproduced 83.1% of targeted vulnerabilities and produced a crash in 97.1% of tasks. By contrast, Claude Opus 4.8 achieved a score of 78.1% (95.7% any crash).

So their quoted figure exactly matches the figure for Mythos Preview, although they don't state the provenance. It could also quite possibly be an independent measurement of Opus 5.

[1]: https://www-cdn.anthropic.com/57a52ea7d8f0e54e8a542e90826608...

I'm not the most LLM-savvy person around, and I'm not gonna say I've put a ton of effort into practically compared these open models. But, a month or two ago I did do some "practical evaluates" testing GLM 5.2 versus DSv4 (flash/pro) with OpenCode's subscription with some late 80s Unix clone-type work, and this jives with my experience.

GLM ended up being far slower, and far more expensive, for approximately the same results. There was never a problem that GLM could solve that DS couldn't solve, faster, and significantly cheaper.

I strongly agree that you shouldn't pick a model based on benchmarks. But for me, I found GLM really underwhelming given its cost and speed.

DSv4 isn't as good as GPT or Claude or what have you, but it's fast, and pretty darned effective. I can run a 3-bit quant of DSv4 locally on my system with ~15 tokens per second, and for a local model it might be the most overall effective at coding. For what it is, it's extremely impressive.

It isn't. I run both at home. GLM5.2 Q4 crushes DSv4Flash0731 Q8. I reach for DS for speed and for medium effort level work. If I care about quality I'll reach for GLM5.2 Looking at this release, I'm comparing it to GLM5.2 and it seems to beat GLM5.2, only time/experience will show. If true, then I'm happy. It's much easier to run than Qwen3.8/KimiK3

In my little social circle DS4F generally substitutes for GLM 5.2 except it's the next best thing to free.

I think instruction following carries outsized weight in these evaluations.

Hey this looks good! Maybe consider adding a hover-over popup for the rectangles explaining what each thing means to a lay person. I see it at the bottom, but that is below the fold.

How much do you pay for the subscription?

yeah, I've definitely noticed one has to be quite precise to keep Flash on the straight-and-narrow

I use a plan -> implement wotkflow for this reason.

pro plans, flash implements. I am super happy with how flash behaves like that.

So not worth it over flash? Even at ~7x the size it isn't worth the price hike. Flash may be a monster of a model due to all the RL it received from free usage everywhere.

deepseek-v4-flash feels so fast and snappy, i'm loving it. Happy to trade speed for the the 5% degraded benchmarking performance.

exciting. it's almost like 3 models in one. that variety would matter when trying to solve a creative problem.

I don't think so. I specifically kept this PR to test model capabilities, and I've already tested a bunch of models. Current test results show that the more advanced the model is, the easier it passes. For example, GPT-5.5 Medium fails the test(has bug), but High passed.

V4 Pro 0813 isn't offered by other providers. I can't find this model on hugging face. It's probably not open, or not open yet.

Unfortunately cache reads with third party providers are all 10-50x more expensive than with DeepSeek, so they're not even close to as cost efficient for multi-round agent use.

i dont see any price increase there... what am i missing?

isn't it the same old pricing? did they increase V4 Pro pricing already?

I'd have to ask for you to be more specific, otherwise, to take your answer at face value, it comes across as a contradiction.

> [Opus 5's output] is beyond the comprehension of virtually all engineers and developers

That would make it pretty bad? The key defining quality of good software, is clarity, and the ability to simplify a complex problem to the point of it seeming trivial.

> Math, science, and engineering are all contained in one model. We may be experts in one field. The model is an expert in everything that humans know.

The bar here should absolutely be to judge this against the expert level within each domain. I have time and time come across LLM output being woefully underwhelming in every single request where I am an expert. For all areas that I am not, it sure seems plausible. It is far more likely than not, that it is equally inadequate in the areas I lack the necessary knowledge to tell.

If the AI is being subpar in every field and category compared to an expert in said respective field, then, what a strange gauge of a tool's usefulness. Are we attributing higher value because a single model is "attempting to solve all knowledge and fields at the same time", why is that of any importance, or excuse?

We should not define "intelligence" as how effectively it can convince a non-expert of something being plausible. That sounds like the absolute worst tradeoff. You'd have to waste the experts time in filtering and refuting incorrect postulations that are cheep to generate. The perfect storm for bullshit asymmetry.

I don't think that's because of its "intelligence". It speaks obtuse techbro-ese: stringing together words that sound smart to obscure the simplicity of the thing it's describing. In many ways it's the opposite of intelligence.

Opus 5 and Fable 5 in particular suffer from this issue at worse level than most models in the same class.

You mean Fable 5 right? Opus 5 makes lots of stupid mistakes about anything that requires any domain knowledge.

Based on what the political climate looks like nowadays it's entirely possible the US bans federal contractors from associating with any company that uses the models themselves, regardless of data provenance or where they are hosted. Or they create AI safety rules that make it impossible to release open source models (for example, making it so that closed-source models can be evaluated with a harness but open-source models need to pass the benchmark with the weights alone, which isn't really possible). Or they just declare Chinese models a security risk like TikTok (claiming that the model would be trained to respect Chinese interests).

It may not be likely but it's definitely possible enough to be something people worry about.

Running on AWS GovCloud isn't necessarily an option, some places prohibit running Chinese origin models even locally.

So is your concern more about reducing the risk of an authoritarian China "winning" the AI race? And less about reducing the risk of your data being used against you personally?

To me, the risks of an individual helping China to continue to develop their AI by being a customer is pretty marginal compared with the personal risks of my data being used against me.