Having tested K3, Qwen 3.8 max preview, Fable and Sol for the past few days, I partially agree. Don’t trust the benchmarks, and the Chinese models really are slow and token-inefficient.

However they do seem very close to SOTA : I’d say roughly equal to previous gen (Opus 4.8, GPT 5.5). It’s yet another silly benchmark, but compare them here: https://senko.net/vibecode-bench/

I also had K3, Qwen3.8 and Fable (using Kimi Code, Qwen Code and Claude Code harnesses respectively, and the official APIs) create a simple but far from trivial web app (zero shot, from a detailed spec). In user testing all three results looked/behaved more or less the same. I had Sol (via codex) do code reviews on all three and it concluded all three were solid, with some room for improvement. Fable was slightly ahead of the pack.

In my own work I still prefer Opus 4.8 (until Antrophic come to their senses and allow 100% Fable usage on Max plans) and Sol, but if I had to find an alternative, I could live with both K3 and Qwen3.8 just fine.

Fable works very well for me on a moderately large codebase. I have had to correct it a few times or point it on the right track, but given how much faster it is at programming than I am that's a very minor issue (and most of these errors are because I underspecified what I wanted in the prompt, I can only think of two cases where it was genuinely wrong... that's a lot better than me in my professional career). Code quality is equal to what I would come up with (and better in areas I'm not familiar with) and the overall software engineering bar is higher because it doesn't get bored when I tell it to do refactors or write integration/regression tests that I would otherwise put off. Also makes it easy to audit code for things like missing audit logging or error notifications that a human would get bored doing.

The product is a fairly standard Ruby on Rails webapp with postgres as the DB. Application complexity is probably a bit higher than average for a webapp. So it's nothing that pushes the boundaries of software engineering, but it is a real product. Token budget has not been an issue for me. I pay for the Max plan ($200/month) and it is well worth it.

Yes, they are all benchmaxxed, but the question is how benchmaxxed they are relative to each other.

We run an evaluation that only compares models in open-ended multi-agent environments where agents affect each other, primarily testing writing code. It's designed to be less vulnerable because there's no solution set, and it's been pretty effective and tends to rank Chinese models lower than their advertised model cards (relative to US models). Kimi K3 is a bit of an exception there -- it truly is a near-frontier model. But it's so slow. Ironically, Muse Spark 1.1 is one of the strongest models we've tested after Fable and Sol while also leading the cost efficiency curve. Big turnaround from Llama 4.

Data at https://gertlabs.com/rankings

I encountered the same issue. Kimi 2.7 looked impressive on paper, but in practice, the code was so riddled with errors that I ended up using GPT-5.5 to fix it.

didn't read the article, but just the title felt fishy because of course an inference provider wouldn't mind claiming to provide access to a mythos-class model. So far, GLM 5.2 is still my go-to, even at less than a third the parameter count compared to Kimi K3

So they aren’t as good as 2024 models? 2025?

Why is token efficiency a concern with free models?

You got baited by bad sampling settings.

It's exactly the opposite. Go turn on min_p once it's available post July 27th and most of the problems you describe will go away.

All you've said needs the qualifier - "for now!" Look at the trend line. It's clear that if they're not yet at the level of being "good enough" for coding, they will be soon. Sensationalist headlines aside, we all need to be preparing for a world where open models can do pretty much any software tasks you need them to.

I have been testing the various models, and I would not call Fable SOTA.

I can't actually get Fable to do anything. I only work on back-end code, and the moment Fable notices the jwt scope checks on the endpoints it's game over, it refuses to do anything because security is involved.

So for me, Fable is completely useless, the bar is very low, any llm that will actually attempt the task beats it every time.

> Oracle routing is a method for measuring the best theoretical performance by running the task through each model and then picking the cheapest correct option (the cost/performance ceiling).

Their "router" is an oracle reference point where they choose the lower cost model after running both and therefore knowing who passed the test. The cost savings part is only Fireworks theorizing what would happen if an equivalent predicting router exists. That's a big if.

There are a bunch of routers like this eg https://openrouter.ai/openrouter/auto

Funny. I prefer a model that does not attempt to talk like a human.

I strictly prefer when models ignore any human quirks in my responses. Claude trying to be your friend, saying LOL to your jokes is ridiculous and frankly, harmful

You don't even need to pay a 5% hit. Just paste Fable output into Gemini Flash and it will rewrite it in more accessible language.

Why? LLMs are not humans.

Yeah. I've found that Opus by default outputs something I call "Claude-lang." It consists of oversimplified, grammatically incomplete sentences that I find painful to read.

Maybe it is something that is easy for it to read and write, but definitely not for humans.

For example,

  Skim once now; refer back while reading Part II. \*Every bold technical term in Part II is defined here\* — treat these as a dictionary, not a reading assignment. The first table covers the vocabulary of the *deck*; the three that follow cover the *methodology* vocabulary introduced in Part II, grouped so you can find a term fast: \*(A)\* the logic of rules, \*(B)\* the neural-network & training machinery, \*(C)\* the method-design ideas.

  JMRL is the paper the thesis instantiates, so this is the one to know cold. Its pitch is \*end-to-end\*: earlier rule methods (LogicRE, MILR) bolt a rule learner *onto a frozen* extractor in a pipeline and suffer \*error propagation\*; JMRL trains the rule module *jointly* with the extractor.

  **Identity.** Conformal prediction for NER producing **finite-sample-valid prediction sets** at two granularities: **full-sequence** sets over entire label sequences (capturing contextual dependence — "if Sarah=PER then NYC likely LOC") and **subsequence-level** (per-span, **class-conditional**) sets; an **integrated** method filters full-sequence predictions with entity-level sets. Adds **covariate-stratified calibration** by **sentence length and language** for valid multilingual coverage, and studies **combined nonconformity scores** (Naive / Conditional / RAPS). **Read in this order.** Abstract → §1 contributions (full-sequence / subsequence / integrated / **covariate (length + language) calibration** / combined scores) → §2 CP recap (inductive split-CP) → §3 NER formulation (IOB2, CRF) → the subsequence / entity-level set construction + class-conditional coverage → the language-stratified calibration results. **Why it matters here.** The **span-level construction** for **Topic 11**'s per-triple score, and — crucially — its **language-stratified calibration is exactly the EN↔zh case**: it shows how to keep conformal coverage valid across languages of differing length/script. Complements PASC (pipeline-level joint coverage) with the *NER-internal* set construction. **Caveat.** A heavy statistics paper (44 pp., *Annals of Applied Statistics* submission) with CRF-based NER; the project needs only the **inductive split-CP + subsequence/entity-level sets + language-stratified calibration**, not the full-sequence machinery (likely overkill for triple-confidence). Assumes exchangeability — borderline under the EN→zh shift, which is precisely why the PASC/ConformalNER *shift* analyses matter.

Yeah, I didn't get in either.

It's (good) content marketing. They sell access to Kimi K3. They're one of the biggest model inference providers out there.

Fireworks is an inference provider that specializes in running open source models very fast and makes almost all of their margin on chinese models

So the economic incentive is literally their entire business model lol

Have you tested Qwen3.8-Max yet?

Lots of money in tech influencing - but most is from the big players (OpenAI buying tbpn, early access to select influencers etc).

Paid?

>Lin Qiao is the co-founder and CEO of Fireworks AI.

LLMs are the Space Race of the US-China cold war. Money is a very small thing; the incentive is proving ethnoracial and civilizational supremacy.

From https://platform.kimi.ai/docs/agreement/modeluse

> We may use Content to provide, maintain, develop, support, and improve the Services, comply with applicable law, enforce our terms and policies, and keep the Services safe and secure. Customer who requires restrictions on the use of Customer Content for training or improving Moonshot AI models may contact Moonshot AI to discuss available enterprise arrangements or separate written agreements. Unless otherwise expressly agreed in writing, Customer Content may be used for the foregoing purposes.

Notably, unlike Claude, there is not an opt-out option for the model training part. The TOS explicitly allows Kimi to train on your code.

> What's the data governance and privacy controls on using Kimi K3 if I subscribe to their coding plans?

If you can’t figure it out from their website, perhaps it is a red flag?

Need to wait until "western" providers start hosting it.

Simplest way is to signup to OpenRouter and filter out all non ZDR (zero data retention) providers. Paying "API rates" however can prove expensive compared to coding plans (for instance, MiniMax $20/mo coding plan allows 1.7b tokens; depending on input/output/cache ratio, it is worth $200 to $500+), except for Hy3, DeepSeek v4 Pro, and MiMo v2.5 Pro (whose API rates are cheap and/or discounted already).

If you're looking to not have to deal with Chinese providers, AtlasCode ($20/mo), OpenCode Go ($10/mo), and Cline Pass ($10/mo) provide 2x to 6x usage for some of the popular open weights (depending on the model).

Personally, I subscribe to Z.ai ($17/mo), and pay API rates for MiMo v2.5, Hy3, Qwen 3.7 Plus, & DeepSeek v4 to the original providers (Xiaomi, Tencent, Alibaba, & DeepSeek).

https://www.kimi.com/user/agreement/zh/userPrivacy

The article says Kimi is better at some things than Fable. That's probably not true of Sonnet.

One benefit of an open source one is that you can, as a large corporation, run it "locally" within your own data center. Even fine tune it.

They share their methodology and results. I learned things about the relative strengths and weaknesses of Kimi and Fable I hadn’t seen anywhere else. Should being in the model hosting business disqualify them from sharing?

The don't only host open weight models. Also, why not promote this. If Fireworks thinks this big news might convert some new business doesn't make it not true.

Anybody who's tried it knows that what they're saying is true though.

Yeah deepseek is my workhorse of choice these days, I mostly use pro because I do a lot of concurrent jobs so speed is less of a concern. When it falls over I change the model mid-chat to GPT5.5 and chuck a couple of tokens over to OpenAI and then once it's correctly found the problem I switch back to deepseek, keeping all the 5.5 analysis in context.

I have been a heavy user of k2.5/6 but they seem to have gotten slower and worse at their jobs (and the "engine overloaded" errors have increased ... ), and k3 spends a LOT of time thinking and doesn't produce noticeably better results. I do keep chucking some tasks over to moonshot every now and again to give it a chance. I figure they're under compute pressure and have probably quantised the models to cope temporarily.

But recently I've found myself mostly using deepseek v4 pro and gpt 5.5 when I need the big guns.

GLM 5.2 has been good at select tasks but really shit at others and given how sparingly I use GPT 5.5 I'm not that compelled to use it. I've yet to be impressed by any Google or Anthropic models!!

The best part is using harnesses like reasonix or whale make cache hit at a rate close to 98%, making requests converge to practically free. And that's with unsubsidized American providers like cloudflare or Digital Ocean.

Prepaid means someone can't sign up, use a bunch of inference, then cancel their card and disappear into the sunset. VPS is a more long term investment where it's harder to switch and it doesn't cost the provider much if a few users jump out without paying for a month.

As someone who is also eyeing Bifrost, I am curious to know what made you choose Bifrost at all / why you decided to use an LLM gateway. I also was considering OpenRouter, since it provides pretty much every model, with same-day releases for new models. One thing that is attractive to me about Bifrost is that if I decide to leave OpenRouter tomorrow, I can do so without touching any other part of my stack; I'm hoping self-hosting models becomes more viable, and then I can become less dependent on third party LLM providers like OpenAI/Anthropic/OpenRouter, and I would not need to worry about a model that I depend on suddenly being deprecated.

What don't you like about the service, besides the mark up?

Have you looked into OpenCode Zen or OpenCode Go?

https://opencode.ai/zen

I didn't realize that OpenRouter had a mark up. Is it a flat mark up across the board or depending on the model?

[flagged]

There's this https://github.com/code-yeongyu/oh-my-openagent which implements the OP article's oracle pattern across 11 roles. Each role has a whole ranking of recommended LLMs across many providers. For example "Sisyphus (claude-opus-4-8 / kimi-k3 / glm-5 ) is your main orchestrator."

So China is now only 4 months behind US frontier models now when the rule of thumb was 6 months just a little while ago.

It released in June..

If only to put some pressure on american labs to bring those costs down

If everyone took your position, then frontier labs can keep raising their price.

The apparent bias may be explainable as it’s not remarkable for OpenAI or Anthropic to be slightly ahead. It _is_ remarkable for an open weights model to be better than the closed models from the trillion dollar (allegedly) companies.

it appears that 3% is the threshold as 3.1% gets the nod the other way

There is too much business risk in running on an American company. They can pull the model back or lobotimize it. No Alex Karp fan, but he was right: companies are worried about hyperscalers stealing their alpha. The lack of guardrails, data security and control really give these open models the edge. If you factor in that they appear to cost less, the hyperscalers are in big trouble. I don't see how this works out. Software was always supposed to be deflationary and collapse down to 0 marginal cost but this isn't remotely the case. It's truly amazing to see the state of open weight models

My questions about strawberries got blocked as too dangerous! I’m not joking

From the blog post moonshot refers to it as open source but only mentions releasing the weights.

Or biology or chemistry.

Interestingly the reality of the open source release is they are opening up to full distillation by the closed source model providers at a deeper and more fundamental level. If anything the open sourcing will help Anthropic and open ai ladder up faster. Open source has always been about mutual cooperation towards a goal and has never closed the door to commercial success. All the hand wringing about open weight models putting closed providers at a disadvantage doesn’t get what working in the open actually does for commercial interests - it is like science in the open - it enables and lifts all boats. Likewise commercial success doesn’t close the opportunity for competition or more open source work - it’s the economy of activity and competition that matters overall. When things stagnate is when closer concerns turtle up and collude on not competing for each others turf.

The future is good and better for everyone the more work is in the open and the more work is in the commercial space. It’s good all around.

For regular chat users it's $19 while Claude is $20...

hmm almost like the exact race-to-the-bottom + arms-race dynamic all the doomers have been warning about

You can! It just might be a little bit outside your budget.

Until Kimi 4 comes out and you'd be like "if only you could run Kimi 4 locally"

Everyone wants the latest and greatest.

It would only be inconsistent if the router chose different models for the same task.

Considering that of 89 terminal tasks there were 11 that only Kimi got right, and 7 only Fable - having a router gives significantly more consistent results if you define consistent to mean correct.

In my experience, Fable is not even close to Sol 5.6 High (not even the max tier) for coding.

1) it's substantially slower.

2) it's substantially more expensive.

3) it's code is considerably worse.

It's a joke when you consider what you get for what you pay for.

Neither. They ran all the cases using both models, picked the winning result for each one, and then said "if you had a router that guessed with 100% accuracy, here's what it would have picked"

What you're seeing is the distilled (no pun intended) result of realpolitik at play. China's labs aren't as open as they are out of altruism, but rather to capitalize and undercut the monopoly held by U.S. competitors.

I'm reminded of space race, where US built centralized planning, while USSR did competition between construction buros

> Chinese are being very democratic with their models

They can afford to, assuming they distilled Fable, because that reduces their pre-training costs, doesn't it?

> The irony is the Chinese are being

There's no irony. The problem is these generalizations and labelling that's hurting everyone. It's the wrong way to view China and maybe many other places.

Just like the definition of AI, the definition of democratic or not evolved long ago.

I really like the idea behind OpenRouter's new auto-beta -- classify by task type, and then just follow what the market is using based on the last 7d. https://openrouter.ai/docs/guides/routing/routers/auto-route...

Maybe a continuous evaluation process, why is there anything to train?

It's all pipes, Jerry.

It's a series of tubes

It’s routers all the way down

And the weights are just in the routers now? Love that idea.

"Information Retrieval is where it's at" Buttle said to Tuttle.

Two that I use are:

* Openrouter.ai for a hosted router

* https://github.com/diegosouzapw/OmniRoute for a local router

Oracle routing is by definition not possible.

> My impression is that, if China surpasses the US in AI development, there will basically be nothing that the US does better than the rest of the world--except for military spending

This comment is stunningly out of touch. The US dominates finance (overwhelming lead in assets managed, stock market capitalization, trading volume, and basically every other metric); space launch (about 85% of all mass placed into orbit); medical devices (exports nearly 2x the second largest exporter); and pharmaceutical research (more active clinical trials than any other country, near-universally ranked as the top producer of major pharmaceutical innovations, world-leading survival rates on many cancer types). We are the world's leading producer of both oil and natural gas. We manufacture the world's most advanced jet engines, precision agricultural machinery, and measurement instruments. We design the world's most performant CPU's and GPU's. We film the most popular movies and record the most popular music. Despite having only 4% of the world's population, there is literally not a single major industry where the US is not a leading player.

There is no doubt that China is a serious threat to US global leadership, and they have already surpassed us in many areas. But that doesn't mean the US is behind overall. We still have a lot of strengths and we have every reason to hope for continued success in the 21st century.

Implying that AI is the last jewel of the USA is too simplistic and ignorant.

People are paying for tokens, that will likely continue even if open weight wins.

Agreed. The benchmark closest to my experience is FrontierMath Tier 4. Fable and Sol (90%) are very far ahead of Kimi K3 (not even 40%). Kimi is trained heavily to basic agentic tasks, like all the other open models right now.

Why do you think it's not SOTA?

Cerebras does not share the quantization of the models so you don't know if you're getting real K3 or k3 lite or something else.

> hyping an open model that isn’t open

Moonshot has committed to release it end of the month.

I'm not sure why you're so dismissive about this model. It isn't currently open, but it will be. It was slow when I used it so i'll give you that. There will be other providers with better performance.

My only gripe was that it takes a very long time to think. It's a good model otherwise.

It should be SotA.

SOTA police

An cross of SoT (source of truth) and SotA (state of the art). SotA looks weird, admittedly.

I suspect typo.

probably buckeye fans

“State of THE art.”