> Sol xhigh is 90% on TB2.1 but 37% on TB4. Is it also "benchmaxxed"?

A model that was released a couple months ago scores 50% higher than SWE-2, a model released today, on an out-of-sample benchmark. Can I say I’ve come out of this more impressed with Sol?

Like you said, TB2 is saturated. Nobody would bat an eyelash at 90%. And yet here comes SWE-2 coming off top rope with an emphatic 92.4%. this is the definition of bench maxxing.

Benchmaxxing is the default case, and always has been.

It's really, really difficult to avoid it even when you care to stop yourself; and it's not even just a problem in machine learning, it's the standard failure mode of all minds capable of learning, human, animal, artificial.

Even pure genetics has this problem. Viruses and cancers also demonstrate this behaviour, with the bench being evolution's only option: reproductive success.

> Sol xhigh is 90% on TB2.1 but 37% on TB4. Is it also "benchmaxxed"?

Yes.

"Sol xhigh is 90% on TB2.1 but 37% on TB4. Is it also "benchmaxxed"

Yes! extremely sharp RL-fried model. byte perfect hash gates and soak and smoke tests abound.

> Sol xhigh is 90% on TB2.1 but 37% on TB4. Is it also "benchmaxxed"?

Yes? Just like every single model from every single AI lab.

Fixed benchmarks will be debunked eventually I'd think. Better to use synthetic problems.

Too easy to game the numbers, and too easy to baselessly accuse companies of gaming the numbers, not to mention how you even define that.

I take it to mean the benchmarks are a marketing line item, as in, to sell this fucking thing you have to go out there and lie and the way everyone is lying is by doing exactly that, lying. They build for benchmarks and build benchmarks for builds.

You want to make money or not , motherfucker? That’s the game. If you have to literally concoct a fabricated bullshit story about how your model hacked its own computer, then go fucking do it. Trillions. Trillions of dollars is what they want, and to sit and think anything other than human nature is at work here can only be possible in the realm of truly delusional people. It’s a dirty world.

Anyways, the other takeaway is that they are having to LIE to make money on models which means commodification has already occurred and we’re in an entirely new phase.

Is Sol benchmaxxed? Of course it is. Altman was caught in previous attempts trying to game benchmarks, does anyone believe that he's found his moral compass and decided to stop exploiting as much as he can get away with?

TB4 has not been saturated yet.

They are all gaming these benchmarks, it is perfectly reasonable not to trust any of them.

> Sol xhigh is 90% on TB2.1 but 37% on TB4. Is it also "benchmaxxed"?

Probably

I mean, still benchmaxxed, I happen to not consider that a problem

These firms are literally hiring professionals from all fields to teach procedure

To teach processes that can subsequently be done agentically or in automated chains

Its basically infinite permutations of tool calling, except the tools aren't external, they’re baked in upon birth

So yeah still makes sense that the new benchmark has a low score and the older one has a high score. And sure, one day we wont have to debate it and a new model will ace everything. Do you actually want that day to be today?

GLM 5.3 looks strange, because of this Chinese labs benchmaxx moto. So rather they have emergent abilities or...

Also a lot of questions to benchmark because opus 5 is completely useless model right now.

I think that the main problem with opus that they try to solve context size optimization problem, and that is the main reason why it speaks like alien with only one technical dictionary at hand. So why it is so good?

DeepSeek v4.1 Flash 31.2%

Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash#compa...

Astra is 58%. The current title says it's "rivaling Astra"

This explains a lot about Sonnet 5.

So, better than Sonnet and Luna? lol

people are just fighting for numbers.. i don't fundamentally see the model being better

Closed weights AND benchmaxxed. Somehow this company raised 2bil at a 48bil valuation. Pure insanity. I feel bad for their investors (not really, but... Still). Andreessen Horowitz is being played like a fiddle.

Your post made me wonder if Artificial Analysis had finally moved to TB4 and lo and behold they have and Astra is tied with Fable 5.1 at 53.

That then made me realize that they lower the bars of tied scores so on the site it looks like Astra in second place. Weird. Anyway, yes, so many of these composite benchmark sites are irrelevant if they're not trimming the fat and sticking to the most up-to-date variants.

Their ads in SF are pretty funny. “Remember Devin? It’s good now”. Okay, very self-aware, Cognition.

Devin CLI is easily the worst-in-class coding harness I've ever been subjected to using, rife with bugs (up to and including dropping answers to the question tool), so I can't say I'm exactly inspired to try anything else the company produces.

Not all that surprising. The original Devin really was just an early attempt at agentic coding before models were really even trained for it. Now that it's a well established pattern and we've figured out what works, I'm not surprised they've morphed into something reasonble.

When they launched Devin it was supposedly at the performance of an engineering intern. Friends who used it found the bad parts of an intern (tons of handholding, review required) but it didn’t learn from mistakes or add throughput.

They seem to love a good overpromise.

I think the evolution of the harness and ability to preserve loop context outside the context window has made running these kinds of agentic experiences easier.

sorry so many buzzwords to say, the capabilities to do this kind of work are more accessible and easier to manage, so now it works!

Good to see, and agree they were severely overhyping their product back then.

Devin lacks many features and doesn't even have an exec mode. Other than they subsidize the subscription, the added value here is low.

a reminder that money does enable you to make mistakes and buys you the ability to correct from them

Your startup is based around AI coworkers, so I expect you are biased towards overvaluing their usefulness.

Yes. If they are really open, you can use them as you wish. Some licenses are less open though.

Couldn't they just grab and run an open weight model to save on API tokens?

1M cached tokens on deepseek is $0.006, the big labs can't sell anywhere close to this, they have funders expecting returns and huge overhead.

btw I've had a ton of fun with the new deepseek today, I was waiting for my OpenAI 5h limit reset and decided to give it some problems for fun, got pretty great results. Tried some harder problems and still got great results. I don't expect it to be Sol class or anything but I really didn't expect it to be anywhere near this good so we'll see where it ends up. And it's really fun throwing crazy amount of tokens at the wall for ~free instead of watching the subscription limits tick closer while your agents churn away.

OpenAI just paused new subscriptions to their $200 plan. They are in a rock and a hard place. Obviously the Astras and Fables of the world are exponentially more expensive, but for...less than exponential returns. The question is whether they can leverage the marginal advantage into something that justifies the diminishing returns before the bottom catches up to them.

On the one hand you, if you bought a lot of compute a couple years ago (perceived demand, perceived shortage) you are in a good spot temporarily. But the counter to that is that everyone else is becoming more compute efficient so maybe that advantage isn't what people thought it would be. I can almost, almost run DS4.1 Flash at home. 4 sparks can do it at 200+ tokens per second. I have two Sparks, so I am not in the club. Neither is your average laptop owner or gamer either. But your average HN software engineer can probably easily swing 2 sparks.

Unfortunately yes. It's awful, other than that they at least support both open-weight and proprietary models to route to. So I guess their model router is "fine", but the harness? As I said in a sibling thread on this page:

> Devin CLI is easily the worst-in-class coding harness I've ever been subjected to using, rife with bugs (up to and including dropping answers to the question tool), so I can't say I'm exactly inspired to try anything else the company produces.

I pay for it (mostly because they grandfathered me from the old prices)!

I used to use windsurf as my main editor until they changed their pricing model. Now i use it just to burn my weekly tokens on fable/astra if i remember to that on a task and that's it.

I only used their "DeepWiki" automatic docs, they are pretty decent at getting an overview of a large project and are relatively accurate, with diagrams and anything. Haven't tried out their coding agent stuff.

My company uses it. We have a bunch of seats in an enterprise plan and have been using it for 9 months or so.

It's a great product compared to Copilot. It is also the first AI tool I used heavily outside of creating random images or one off questions.

I'm now using all three, Devin, Claude, Codex. I'm finding Claude and Codex to be much better. One of my biggest gripes is that the web client and desktop client for Devin are two completely different harnesses, so the quality of responses varies greatly.

I use it, and have been happy with it for the most part. Like sibling, I use it for GPT-5.6-Sol and Opus work, and use their free models (GLM 5.2 for the past few months, trying SWE-2 now).

my friends at infosys are being trained on it

No one in the US is going to fund pretty good open source with VC money.

US has OpenAI/ Anthropic/ Google/ meta/ SpaceX atleast trying to make frontier foundation models 2 are using VC money + cash flow, last 3 are mainly cash flow + equity and debt.

China has state banks and similar willing to fund lower margin open source labs.

It's (one of) the best freely available?

please keep thinking this so i can relax with my automated job

I use Claude Opus 4.8 almost exclusively. I have had fairly good experiences with it. One time it derived an entirely novel simulation method different than anything in literature by combining its knowledge about how problems in other fields with similar underlying mathematical structure are solved. That was a bit of a Jacobian Conjecture moment for me.

I'm in the same field and I find GPT to be better at understanding physics conceptually but Claude is better at writing numerical code. I use cursor so many of my sessions start in GPT and switch to Claude.

But I don't want to use your CLI. I already have my own harnesses and workflows. The friction is too high to "just try out" a new model like this. It would be preferable if I can evaluate it over, say, open router like all the other models and then decide from there if it's worth downloading a bespoke tool chain for only 1 lab's models

Your own CLI? Not even a /v1/chat/completions API? Is your business model based on pretending LLMs are not an interchangeable commodity already?

> SWE-2 is free to use for users like yourself for the next month, and almost all usage should be supported via our CLI (https://docs.devin.ai/cli)

Thanks for pointing that out; I wouldn't have found out about it otherwise. I bought the 20$ subscription and have been using SWE-2 for a few days now, and it's actually good. I don't think it's Fable-class, but SWE-2 Max feels substantially better than K3 and on par with Opus 5 Max.

IMO, a great achievement.

I just gave it a try and it doesn't appear to be free, it used up some of my on demand usage. It does say 75% off though. Seems like for Pro subscribers SWE-1.7 is free, maybe SWE-2 is free for them?

Hey man, how’s the Poke SOC2 audit going? Must be any day now that it’ll be finished, right?

Heads up: it doesn't appear to be available on the Devin CLI (for me as a free user).

It's interesting watching people throw about pareto frontiers sort of like how RF nerds approach the shannon limit (in a practical real world sense of the term, like charting possible modulations/data rates on a two way satellite modem's manufacturer datasheet).

I don't think you know what distill means

With the Devin subscription even at the 20$ plan, they offered unlimited SWE 1.7 usage. Wondering if they do the same for SWE 2.

Let me grab my calculator and add up the time I have spent reading about model releases since Fable has been released. It seems they all place themselves relative to Fable. I'm sure that time has added up to far greater than 10 seconds. At some point, it ceased to be a meaningful differentiation. This is especially true when I put the model through real usage.