It’s funny, in oh-my-pi I’ve had much better luck with DS 4.1 Flash than GLM 5.3 flash. Glm seems to just constantly reason in circles before attempting a tool call

Makes sense, GLM 5.1/5.2/5.3 is a lot better than DS v4.1, I found the same results in other domains.

Given how fast and cheap DS is, it's just an ideal model with enough "IQ" to let it loose. Another thing they left out of the article, DS becomes really good if you provide custom tools for the task, on it's own it's mediocre.

If you are talking about publicly known vulns, it's a bit moot since they should be in the training sets. If not, you just burned the vulns to that inference provider's training data (and any intermediary), and future benchmarks will be meaningless.

In your example, couldn't you parallelize DS's work more ? You could have 11 times as many agents for the same price.

A little off-topic: where does one use those models such as GLM or DS for this kind of reverse engineering tasks? I think I read many of them refuse to help with tasks like those on their official platforms.

you can run 10 ds-f agents at the same time for the same cost exploring a much wider surface area.

I think it really depends on what the data DS was fine tuned on. If your use case is very specific, it wouldn’t have distilled that knowledge well.

Thing is that GLM 5.3 is many multiples the cost to run, and slower.

I have good results with DS4.1 flash because I can iterate faster. I either provide it with correction, or it discovers its failures via the harness. And seems to respond well to empirical evidence rather than go in circles.

So it might need some prodding, but it's likely in this case it was able to brute force after several runs and collecting some evidence.

What I find surprising is that DS Flash can do it at all.

I love DS flash, it is an amazing workhorse to implement plans created by more robust models (such as GLM). But a more fair comparison would be of DS Flash with GLM Flash.

Notice the qualifier "our", that is the one they have access to.

What if there would be a separate category for distilled models?

Good luck building something of value with "day to day coding". If you're going to compete in the future economy, you're going to compete at the frontier. The low hanging fruit will be of no value. If I can recreate your SaaS with an open weight model at a fraction of the cost, then your SaaS has no value.

Apparently they updated it perhaps based on your comment?

> . The accepted runs cost $4.65. Failed attempts and replacement runs increased the complete cost to $5.14.

You can have 100 runs for the price of the Claude Max plan?

> When the history books are written and all is said and done, the hubris of this moment where all the American labs decided to punk their investors and join hand in hand in agreeing to let the Chinese win forever is going to be the main story.

Come on. Lecturing about hubris when your message is damn the consequences, full speed ahead?

If it's a race to build the torment nexus, or a race with a nonzero chance of building the torment nexus by accident, I don't care about winning.

Win what? The race to the bottom always has this competitive language.

“If we ban CFCs now the Chinese will win!”

“If we ban chemical weapons, nuclear weapons, etc etc our enemies will triumph! They won’t stop!”

“If we switch to biodegradeable plastic then our rivals will have an advantage.”

“If we dont externalize the costs to our population, then they will, and then will win!”

I think workflows can do the job agents do, 20x cheaper and more predictably and safely. They can completely displace agents, just as HFCs displaced CFCs and then we were able to ban CFCs and phase them out through international COOPERATION. The language of COOPERATION is what saves us vs COMPETITION is all about cutting corners and externalizing costs. Google the Montreal Protocol, Geneva Conventions, Nuclear Non Proliferation Treaty, Unleaded Gasoline etc etc.

Agents have got to be marginalized. They are just popular because the labs need to make a ton of money for their investors and recoup their massive spending on training models.