That’s only true if you think AI is the only reason to own a powerful and efficient server. Mine does plenty of traditional server stuff too.

how much performance (tok/s) can you expect from 128GB Strix Halo? assuming this model will be released with FP8

also can you use it for fine tuning?

Openrouter was pretty great before prompt caching became common. Now it is extremely expensive for most individual workflows, unless you spend a lot of work customizing router preferences, and then you still get a worse cache hit rate than using the provider directly. I only keep $5-$10 in OR for occasional testing.

The best solution to this for me is to self host litellm or a different router and use model aliases. For example I have a model called "coding" and when a new good model comes out I just switch the backend without needing to change the alias or the key in my projects (opencode, etc).

I have a few of them even a smart router called "agents" which will use local models but if it thinks the request might require higher reasoning it's routing to a different model

I'm confused? Can you just define some presets and call them instead? With preset you can pinpoint a lot of things, especially the providers

Have you tried FreeToken yourself? I was hoping to find some benchmarks on their github but took a quick pass at their research paper and it seems they're showing ~2x performance on qwen 3.6 35b when compared to llama.cpp - but llama.cpp is so sprawling and has so many options I find that a difficult comparison.

You can already run it locally its just not the same.

It is still slow, a lot slower than what you are used to with claude and co.

And as soon as you increase context size, your memory requirements jump.

Then when it runs for 30 minutes for something claude needs 5, your device will get hot.

And even a used 3090 is apparently now between 1-2k.

Yeah 27B is way too slow for the Strix Halo. Laguna was better but still slow when I tried it. Qwen3.6 35B is still the best today.

Just out of curiosity, why run "local-local" when you could just set up a Mini or Studio at home and query it over http? [edit] whole conversation about this in another thread https://news.ycombinator.com/item?id=49433413

I’m personally considering retiring my MBP for a Studio + 15" Air whenever this MBP ages out.

I have a 128GB M5 Max, and it sucks at this stage. 50-70 tok/s might be something...

IDK, prefill speed is a bigger concern for most wokflows, like agent coding, and I heard that this is quite low on macs?

They've said no moe for 3.8, and since they're already releasing a qwen4 early preview, they're probably focusing on that arch going forward.

So back in the Qwen 3.5 release, the 122B-A10B model scored slightly better than the 27B model. I'd expect this new 125B-A6B to perform similarly to the recently released 27B. Qwen3.8 27B is supposed to rival Sonnet/Opus 4.6.

Why not? It's not really competing in the same size class.

Besides, as they explicitly wrote here, the main goal for this release is not performance, rather to serve as a reference for inference runtimes about what to implement. So that later Qwen 4 can be released with zero day support.

Their "next" variants are usually undercooked, but useful for the community to verify support for inference stacks. This will likely be the same.

Sounds like a line from a fairy tale

You must have hibernated for a year. Most modern 27b models can outperform gpt-oss-120b.

gpt-oss is long dead though. It's been 8 month since Qwen 3.5 was released.

Have you tested Muse Glimmer in low reasoning strength?

Token generation is slow (and prefill is) but you will likely find it solves actual problems faster than Qwen 3.6 35B-A3B.

As a 5090 owner and local model enthusiast, I was hoping it would be 35B A3B so I could run it myself =(.

Only VRAM? Unlikely unless you can also load the whole model into regular RAM. The previous 3.5 release was 250gb at BF16, so FP8 would likely be around 125gb. Your best best is FP4/Q4.

This is what I copied from the en version of the modelscope page, right when they published it:

> Redisgned Multimodal MoE Model: 125B main model parameters, supplemented by an additional 51B N-gram embeddings,and 6B parameters activated per token.

> Efficient Training and Inference: Significantly reduces training and inference costs. At ~1/9th the training cost，Qwen3.8-Flash-Next achieves comparable capability against Qwen3.7-Plus, while being more capable in areas of coding and cowork.

There was another paragraph about a new attention, but I didn't copy that.

It was in description under the countdown initially, but was quickly removed.

It also said 51B of n-grams and new attention (IIRC it said "Qwen Sparse Attention").

edit: here's a random screenshot https://x.com/AiBattle_/status/2092210011858460819/photo/1

Well put and succinctly put. And if OxA is a flash model? it becomes: goodnight