I used Space Bunny from some providers that offered it for free and it was a very much a "you get what you pay for" experience. If the release is comparable, I'll be sticking with Deepseek v4 for sure.

> The perverse incentives around benchmarks

I am not sure. The people who actually care enough to look at benchmarks will run evals, and find out quickly. If you post silly benchmark scores too often, you will simply be ignored. You are paying with your reputation.

This is my experience with all MiniMax models: They're good on paper but fail to live up to expectations in practice. IMO, at no point has any MiniMax model been the superior choice over others, all things considered.

"When a measure becomes a target, it ceases to be a good measure." https://en.wikipedia.org/wiki/Goodhart%27s_law

Opus 5.5 is according to AA cost per task, 23x more expensive than Deepseek with max effort (which is dumb, you shouldn't use) and 13x more expensive on xhigh.

Claude sub has a 60x subsidy, so a $20 sub gives you $1200 in API.

Based on this math, Claude sub is a much better deal than Deepseek. And it's a stronger model of course.

What harness do you use to drive the models?

There are hundreds of benchmarks (all noisy), dozens of model families, and dozens of modes and variants within each. Announcements are crafted by publicity teams and involve cherry picking a chart that looks compelling.

As a prospective user, none of of the benchmarks details will matter to you, but the family of models and settings they peg themselves against can give you a hint of where you might want to evaluate them within your own workflows.

As an enthusiast or lay researcher, the more interesting thing is to look at their techniques and any innovations they're exploring in training, inference, etc. Those are usually going to be details tucked below the fold and won't be summarized in a benchmark chart.

As anybody else, why do you even care about this in the first place?

This doesn't just apply to AI. You should know better than to take marketing announcements at face value, nor trust that most "influencers" or "journalists" or "commenter" repeating and discussing those announcements are doing a lot of work as they buzz over what's said.

Don't be a victim to PR spin, but also don't discount potentially interesting work just because it got channeled through a marketing team. Just look for the information you need amidst the noise or move on if there's none there.

Because they selectively choose which benchmarks to highlight and which competing models to compare against; they don't always compare against the SOTA or the Pareto frontier of performance/price, a lot of times you'll see them compare against the last generation of competing models, or the like, so that they look better if you don't carefully consider what they're comparing against.

For example, here you see they are comparing against Opus 5 and GPT-5.6 Sol, when Opus 5.5, Sonnet 5.5, GPT-6 Astra, and GPT-6.1 Sol would be the latest SOTA models from those labs to compare against.

Also because a lab's own benchmarks tend to run about 6.4% better than third party runs of the same benchmark. Source: https://whatmodel.sh/claims This can happen due to a number of factors, and doesn't necessarily indicate anything nefarious. It could just be that they run multiple times while fixing issues that aren't related to the model, and have a selection bias towards stopping when the score is on the higher side. But since you see that sometimes their own scores are lower than the independent runs, some of the variation is explained just by variance in the model solving problems, since LLMs are nondeterministic there can be variation from run to run and you don't see labs always cheating, just potentially a bit of selection bias about when they stop tweaking their benchmark setup and publish the results.

And yes, there are independent third party runs of benchmarks for just this reason. The most popular with broad coverage of a range of benchmarks is https://artificialanalysis.ai/ but there are a number of others as well.

You have to read between the lines (or wait for someone on HN to do it for you).

It's never "cheaper and faster and more intelligent", it's pick 2. Sometimes (as was the case this month), it was "cheaper per token, but reasons five times as much", so your costs go up.

This seems worse across the board compared to latest DeepSeek Flash, except on some automation bench.

So, I can quickly judge that this is an interesting option only if cheaper or faster than DeepSeek.

Based on Terminal Bench 4.0, I'd also guesstimate behind GPT 5.6 Sol, so of course behind the SOTA as expected.

I run a benchmark and independently check all new models. They are very spiky.

If you are doing coding only then official benchmarks are mostly right, but as many people are starting to do for instance marketing work with LLMs now there’s very little correlation between coding and creativity for instance.

I have this idea that the big labs internally have much more capable models which are prohibitively expensive to run though, and the competition is mainly around getting the price down by releasing models that are quantized/otherwise made more efficient at the cost of a lot of performance.

I've got a corollary question, which is "how big is the market for these never ending model announcements?" ... I guess its mostly people coding with agents. Most general AI users are just asking ChatGpt to word an email for them or whatever. How many AI users are on here looking for the next model announcement and working out if the price per token is better? Are we talking like 50,000 developers? More? Less?

It's always cheaper (*than the next more expensive model) and better (*than the next worse model).

And often they also don't mention how verbose it is, that the same text splits into more tokens than in competing models, how slow it is, that the pricing is temporary, that the cost per task actually isn't that good despite being both cheap and capable.

https://x.com/pashabitz/status/2105486786382406087?s=20

If anything I am confused by how high cache costs still are across many models

The really expensive part of inference - the compute and memory traffic required to perform prompt processing and decode - is paid for roughly once per token in a multi-turn session when prompt caching is working properly.

The difference to the provider's costs between:

- Submitting 100 back-to-back requests, each adding 1k tokens to the context

- Submitting 1 100k token context request

Is negligible. Yet the consumer is going to pay a lot more in the former case for all those repeatedly submitted input tokens that you get charged the cached rate for (but which required no additional compute on the provider side).

DeepSeek V4.1 has an insanely optimized KV-cache that takes around 900MB per million tokens, making it feasible to move to disk between requests. Makes sense they're cheap

Not sure why ds cache costs are confusing? Their kv cache is absurdly compressed compared to almost everyone else.

That still isn't the whole story. Compare DS 4.1 Flash [1] to GLM 5.3 Flash [2], and consider the effective price. DS's price is lower than GLM on input tokens, but higher than GLM on output tokens. Output price is an order of magnitude higher than input price. So for output-heavy workloads and long reasoning, DS costs more than GLM.[1] https://openrouter.ai/deepseek/deepseek-v4.1-flash#pricing [2] https://openrouter.ai/z-ai/glm-5.3-flash#pricing

Yes the official DS provider has a low cache cost, for now (it's just distributed KV). But people are still concerned about ZDR, sovereign AI, getting more model choice in their subscription price, etc. They want more bang for their buck. So when a new model comes out, the bigger impact is really its performance across providers, not the price at a single provider.

And in about 5 years (or less), API prices will stop being a thing. We all used to pay per-minute for phone calls, texts, data. Nobody does anymore. Virtually every mobile provider is now a flat prepaid or contract fee. Carriers improved their technology (going from analog to digital), and it turned out consumers will spend more money on a packaged subscription than on pay-per-use. That's why everything in the world is a subscription now.

Your modern $70 big wireless plan isn't giving you $70 of value. Similarly, your $20 AI subscription that's a great deal right now, will be less valuable in 5 years, when hardware & models are more efficient. You'll still be able to buy a local AI rig, but it will still be very expensive, due to AI models continuing to be large, and NVidia keeping new hardware expensive, as these things together reinforce their dominance and profits, and our reliance on them.

This also follows another pre-existing trend. Why do people keep having to buy a new iPhone or laptop every other year? Because each year companies make software & data less efficient/more bloated, so people need a more powerful machine to run it, which enables them to make even more bloated software. Almost everything people do with a computer today is what they did 15-20 years ago; it's just a higher bit-rate. Despite this, people keep re-buying the same machine, so companies can keep extracting more wealth.

AI works the same way. Rather than cost to consumers going down over time, it will likely go up. Not because it can't get cheaper/more efficient - but because it increases companies' profit. This isn't just a theory, btw; the US economy's growth is now almost entirely reliant on this happening.

The FAQ on their website says Minimax 3.1 flash is 428B total, 23B active

Which is roughly similar to deepseek 4.1 flash.

I find it interesting that we’re calling these “flash” models now, even though they aren’t “small or light” and in many cases aren’t particularly that fast.

I suppose it’s more so that the standard for “regular models” is becoming 1T+

https://agent.minimax.io/tools/m3-1-flash-preview

I think flash usually means MoE, not dense

It did okay in Pac-Man. Not great, but a usable example. Unfortunately, it did take half an hour. I'm trying to figure out what the pricing would be if it wasn't free. But zero dollars is a great entry price.

https://jonclegg.github.io/pacman-bakeoff/#minimax-m3.1-flas...

13 TPS is painfully slow?

20 t/s is about the speed you can read at, so slower than that and it always feels like it's buffering. 35 t/s is where it starts feeling decent and 50 t/s feels nice.

I went to check and Opus 5.5 wasn't benchmarked on DeepSWE1.1, I can't keep track of which are in vogue at the moment but maybe it's thought to be saturated at this point, as they only have older GPT models as well

The recent DeepSeek release used Opus 5 as a reference point as well. I don't understand it. It's not like they're referencing superseded releases of other open models.

On Openrouter at least, all Mino models are painfully slow. TTFT is many seconds and the models spend a lot of time on its CoT, making effective output per second many times slower than DS Flash. On Flash 4.1, I constantly get ~180 tok/s and the real work got done fast, too. So even when the Mimo 2.6 are smart models, I use them much less recently. Not to mention their costs are also higher than DS.

For me, the DeepSeek 4.1 models are currently the ultimate cost-effective model the rest have to match with, especially if you are not on a subscription plan with heavy discount.

GPT Sol 6.1 is however, for me the most efficient model, both in terms of time to finish and token usage.

> Another day where the Pareto frontier is moved up and to the left!

It actually hasn't, it seems: https://news.ycombinator.com/item?id=50151775

Would place DS 4.1 1017 roughly equivalent to GLM 5.3 which means it's targeted to price rather than direct capability. Probably why they listed DS adjacent in the comparison table when according to their results more expensive models are beaten by it.

You do realize that there exist multiple Pareto frontiers, at least for every arbitrary combination of parameters for a given benchmark?

Obviously your marketing team is going to use the Pareto frontier where your model has improved the most and compares favorably against competitors.

Some reddit comments say it's not as good as DS 4.1 Flash. But I didn't test myself and I don't trust benchmarks.

> [Space Bunny] was an early version of MiniMax M3.1 Flash.

They mention that space bunny was an earlier checkpoints, so it may have improved (or may it could be benchmaxxed)

I find it interesting that so many models seem to be at almost exactly 74% specifically which makes me suspicious of the benchmark. I want to say 5.6-sol, Astra, deepseek flash 4.1, opus 5 (and 5.5?), Gemini 3.8-flash and a few others all ended up there?

there are some efforts to track it: https://github.com/ninjahawk/livenerf

This is thought to be conspiracy thinking but I don’t think it is

this comment was dead and I had to vouch for it

Apparently the domain is shadowbanned (https://news.ycombinator.com/from?site=minimax.io) which is why people kept submitting it (and normally they cannot be vouched), and presumably a moderator gave this one their blessing.

The solution is for @dang to unshadowban it because Minimax is now a newsworthy company.

Amazing haha. Why is it so quiet around hy3/hy4? Or Tripo? Or agitbot or z.ai? All incredibly capable

@dang maybe?

Friendly reminder: Submissions without discussion aren't considered dupes on HN.