If I read correctly, that's based on a 1-bit quantization, and can we really expect that to produce any useful output at all?

Why did you use 1-bit quantization vs 3-bit quantization?

It looks like the 3-bit requires 90 GB[1] which, I imagine, would fit within the DGX Spark's 128GB of unified memory.

[1] https://unsloth.ai/docs/models/qwen3.8-next

You should find an excuse to offer 3D printed extruded pelicans from various models as awards for something. I have no idea for what, but the idea captivates and I'd love to win one somehow. They'd be collector's items in a few decades

The spark can easily run UD-Q4_K_XL on this model... using IQ1_S doesn't make much sense.

Tried again with a different quant, UD-Q2_K_XL:

https://tools.simonwillison.net/markdown-svg-renderer#url=ht...

Doing this on a 1-bit quant is unfair

The Pelican Brief

It is 125B A6B. vLLM is already out with support, ngrams can be offloaded to RAM so you only need ~96GB VRAM for nvfp4 w/ full context.

Likely soon we'll see nvme offloading for ngrams as well. They're just an index, so that should be plenty fast for what it does. LLama.cpp support should come soon as well, and they might do some things with offloading first.

The N-gram parameters can be fetched from SSD, with maybe the hottest ones staying in memory.

I have nvfp4 quant fitting fine in 128GB on DGX Spark, but with paging (from nVME) of the n-gram table. Resident ~80GiB for weights & context.

On branch of https://github.com/rdaum/eider (for DGX Spark). ~12 tok/sec decode without speculative decoding (will come later)

Still actively working on this. Prefill currently sucks. Will merge to main by end of day.

EDIT: This has now landed on main. Still haven't done MTP speculative decoding boost, but:

80tok/sec prefill, 12 tok/sec decode. ~90GiB or so resident. n-grams paged from disk.

People in my server are running it on Strix Halo 128GB using RoCmFP4 and reporting 35tok/s, without much optimization, with proper MTP, better kernel, expecting about 50-60tok/s.

I'm running the UD-Q4_K_S on my 128GB M5 Max with 180K context, it uses around 100GB~

Gonna have to wait a few days to see what the wizards of the HF community come up with…

For comparison with hosted models, GPT 5.6 Luna scores 67% on DeepSWE, compared to 59% here for Qwen.

Luna is $0.20 / $1.20 vs $0.16 / $0.47 with Qwen.

Curious, how are you running it and what quantization are you using? I've mostly been using MTPLX; 125B sort of looks like it'd be right at the limits of my 128GB MacBook once you factor in KV cache and context window.. wondering if it's worth it compared to the 27B model which gives me a lot of headroom or even a 72B model.

It's a much bigger model, with a next-gen architecture. It's expected to be much better.

I don't like these comparisons. Sure it is impressive, but it does not have a world knowledge of larger models. It has most of theirs intelligence.

My AMD strix halo box (haven’t benchmarked yet) should also run it reasonably well. It was $1400 at launch, and is $4K now.

Your mac is < $2K in Biden-era dollars. Presumably the economy will eventually recover; maybe in one Moore’s law doubling if the midterms go outrageously well. That’ll be two doublings since the halo launched. I’d expect this model to run on a sub $1K box by then. $2K ought to get you a 512b parameter model at that point. If we have to wait out the rest of the term, the cost cliff will be even more pronounced when it hits.

>Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy.

How much memory does this translate to and what quantization (if any) were applied?

[flagged]

Quoting RGFusion from Reddit: LLMs run into an issue where the further you train a model, the more it overwrites facts with generalized concepts. You need the model to be able to do both. Intelligence arises from generalization, but without accurate information the model will hallucinate.

The engram table allows for a low-computational method of fact-recall. You can think of it like a better form of RAG, where the data doesn't take up any of your context window and it's injected deeper into the model's layers, freeing the lower layers to carry out abstraction. This results in better "focus" for the model, both in regards to its intelligence and context recall.

Basically, they've separated the specificity-critical portions of the models memory into a parameter space that doesn't need fast compute (you can run it on system RAM) and allows the model to be trained on higher volumes of data without ruining its knowledge-base.

https://www.reddit.com/r/LocalLLaMA/comments/1vy6smx/comment...

Ngram is compressing several layers of multiplication to a lookup which negates the need to have the same model depth and reduces the model size that must be loaded.

Using llama.cpp I one-shotted (2 hours) a reasonable asteroids clone on my strix halo/128 using the 1 bit quant, using my custom harness (which isn't anything exceptional).

It was tedious - a lot of second guessing itself, and quadruple checking things it fixed a couple of iterations back - but it got there and the result is a playable game.

Speed starts out strong, but definitely drops off as context grows. At the end (I think context about 70k) it was down to 12 output tps.

Mind a bit blown.

In my early testing it's way better both quality and speed on Strix Halo (posted recipe in sibling comment).

> I think that Longcat was the first one who used it

Wasn't it introduced by Gemma?

I only see a 1-bit quant posted on unsloth HF and it’s 72.5 GB. Is that what you mean? That’s much bigger than I expected. If you can’t run a 4 bit quant in on Strix Halo it becomes a lot less interesting. https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF

Download is available, but likely need to wait for an update, I get this which is understandable with the architectural change :

Original error: llama.cpp does not support this GGUF's model architecture ('qwen4exp')

Edit : Saw the pull request, should arrive soon enough https://github.com/ggml-org/llama.cpp/pull/27742

73GB for the 1 bit model...

[dead]

You’re absolutely right to be hopeful. Three honest possibilities, and I’ll be straight with you about each:

1. It overthinks — Just like the previous iteration. High confidence. 2. It doesn’t overthink — Improvement from the last model for your use case. Regression for others. 3. It sometimes overthinks — Best case all around. A feature, not an impairment.

One final thing worth mentioning: (I made myself irrationally angry writing this)

You might already know this, but a large part of test-time compute / 'overthinking' is just letting the model do more passes, and refine its activation residuals more.

For example, even if you make thinking tokens literally just '....' (absolutely meaningless; zero information), you still see significant performance improvements: https://arxiv.org/abs/2404.15758 and https://arxiv.org/abs/2607.22925 for some starters.

Treat thinking more like a "loading screen message" that's been RL'd to somewhat resemble its actual internal state; which happens in its activations, not tokens.

Did you observe the model overthinking on practical tasks? While 3.8 does think a lot on xhigh I've found that it really depends on the task. On one-shot prompts that are usually the first to be posted during new releases it will tend to spend a lot more time thinking than doing. In other words the more open ended a problem space becomes, the more Qwen will tend to second-guess itself.

Conversely I've found that it can be as succinct as Muse Glimmer when it has a clear path forward. This can be either through well defined requirements or through unambiguous steps to take based on its own reasoning. While I do think it's fair to call out how much smaller model overthinks especially on one-shot prompts, in practice it hasn't led to an overall increase in time to task completion at least for what I've been using it for.

That's low reasoning for a model, but max for a HN comment.

My stack is basically deer-flow with Qwen3.5-122B-A10B; this hopefully will be a speed and intelligence improvement. Running deer-flow overnight on any research topic or verify clear scoped programming issue is really neat.

Also, heating my home during the winter is nice.

Oh, also, I use llamacpp with --reasoning-budget; very simple way to move on.

You are absolutely right to push back on this. Let me think for a moment.

Since you're running through the trouble of setting that up, if its 125B params, but only 6B is activated, does that mean you mainly need to allocate enough VRAM for that much of the model? Or do you still need enough VRAM for the whole thing (and buffer for context window)? Or maybe anyone can inform me, this is one area I'm uninformed in.

It will be interesting to see the token efficiency analysis. This is my first question now with Chinese models; I take raw benchmark performance for granted.

What kind of machine do you have in your homelab that can run this model?!

Yep.. for 'general purpose' use I found qwen3.8:27b to be disappointing due to overthinking. It's brutal especially considering how slow it is compared to MoE variants. It often overthinks to the magnitude of ~10x the tokens vs a ~4x faster gemma4:26b-a3b.

As a result, qwen3.8 will churn over a prompt often for 5-10 minutes while gemma4 regularly finishes the same prompt in under 20 seconds, while giving a consistent and accurate response in my favorite test case. Qwen3.8, despite churning like that, often misses with an inaccurate answer.

Obviously, 'YMMV' depending on your use case... just sharing my two cents.

What about adding rtk proxy?

> Better... than 3.8 27B

How can a "6b active per token" MoE be better than a just released 27b in the same family? Possible maybe, but quite interesting and requiring some explanation.

Edit: ok, on second thought, probably because the small "experts" are really rich in specialized training compared to the dense model. Still raising questions about the details, e.g. the reasoning abilities (or all meta-skills) of a "6b per token" network compared to the dense capabilities...

We should still expect significant performance improvements.

I believe Unsloth’s branch (and ggml’s) don’t support MTP or the ngram embedding yet.

Haven't tried, would be surprised if it's any different.

It's new arch demo for future Qwen 4 family, but (as I understand) training recipe/data is same as any other 3.8 model.

If you've got a DGX Spark try my little engine: https://github.com/rdaum/eider/

NVFP4 quant

Where are you seeing that? At the bottom of this post from Qwen I see:

Qwen 3.8 flash: $0.16 / $0.47

Compared to

Deepseek 0723: $0.03 / $0.075

(units in USD/m tok)

(Edited: I thought Qwen3.8 Flash Next was smaller, but it's not, in bytes. Here's how they compare.)

DSV4 Flash 304B params, 167 GB download (at full size)

Qwen3.8 Flash Next 180B params, 360 GB download (at full size)

They are giving pointers to how to run it now using for example https://recipes.vllm.ai/Qwen/Qwen3.8-Flash-Next (and an especially provided vllm release).

Relevant PR: https://github.com/ggml-org/llama.cpp/pull/27742

This branch works now: https://github.com/unslothai/llama.cpp/tree/qwen4exp/qwen3.8...

  cmake -B build -DGGML_CUDA=ON

  cmake -B build -DGGML_METAL=ON

  cmake --build build --config Release -j --target llama-server llama-cli

Probably going to take a 1-3 days for support to land in llama.cpp and vllm.

I guess it was obscure before, but someone did "know" it: https://old.reddit.com/r/Qwen_AI/comments/1u7xnvq/how_big_is...

A single Spark alone is not worth the price. You are paying $1000 just for networking equipment you aren't using. At 2x it starts to maybe become worth it if you don't want to deal with Apple. Outside of the newest Macs, I can't think of anything else you can get 256 GB ~550 GB/s memory bandwith for $8200. Even at 3-4 Sparks it scales relatively well.

With 2x Sparks, I am getting 40 t/s. I'd guess that without MTP you'd get 12-15 on 1 Spark, maybe 20 with MTP?

Given the new architecture, speeds are harder to estimate. Max would be ~40 tok/s.

indeed, can't wait for it to be supported by llama.cpp (or other engines)?

I agreed

I suspect we will see optimizations where the various vectors of the n-gram you actually use are hot in vram, the rest are warm in system memory and then cold storage on nvme. Same with MoE. If your workflow is particularly same-y then you're looking at cache miss below 5% with NTP/MTP turned on and the right harness. Agentic "openclaw" type stuff cache miss might be below 1% in the right local llm setups. There's been zero exploitation of n-gram stuff yet, it will be very interesting as things progress.

It depends how you use the models. These small models work great for developers who prefer to stay more in the loop, and only task the model with things that can really only be interpreted in one way.

Not to mention, they’re great for self-hosting and getting yourself to not be dependent on some API that can go down or be altered at any time.

Big models seem to mostly be good for pushing ahead the frontier - the smaller models tend to gain the frontier’s capabilities after only a handful of months anyway. Many are perfectly content remaining a few months behind the bleeding edge.

If you have good feedback signals, like tests/benchmarks/etc, then it is potentially better to do multiple turns where model uses that to adjust code. Which might not need as smart a model.

If there’s news, then yes. This is a pretty great new release for those still stuck on Qwen3.6 35B A3B if they have enough memory but don’t have super powerful compute.

I wonder if I could get this running through vLLM on 6x Nvidia L4 - the 3.6 worked great on 4 cards but sadly TP6 just isn’t a thing and I don’t have 8 cards available, maybe it’s gonna be okay with like TP2 and MTP. I have no idea at this time, probably need to test out what even might be possible.

This particular release is interesting because it's a preview of qwen4 architecture. And, while benchmarks are iffy, this is a direct comparison, by the same team, with qwen3.8-27b that was pretty well received for a local model.

This "next" release adds a new concept, first public release with n-grams, I think. And it's in a MoE size that is likely to be very fast and cheap to serve (faster than 27b for sure). It's also well suited for inference on alternative compute (i.e. sparks, macs, etc) so it's relevant to local users.

I and presumably quite a few others with AMD AI or Apple Mac platforms are very impacted by this.

:)

It is very relevant and for a certain group of us, far more impactful to our work the next month(s) than any blog post could be.

this is a new architecture (foreshadowing qwen 4)

> trained at just 1/9 the cost of Qwen3.7-Plus, while outperforming it across the board

https://x.com/Alibaba_Qwen/status/2092591393424515114

There are many topics, personalities and politicians we hear about daily who have no merit.

Qwen's advances do (currently) have merit.

This actually is meaningful news, I think. Pretty wide audience appeal in the local LLM space too.

yes there’s no shortage of online real estate