Probably won't have to wait that long. Prism released Bonsai 27B (https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one.

It is debatable if we will actually need that many parameters though, since recursive nets like HRM (https://huggingface.co/sapientinc/HRM-Text-1B) don't need to parametrize as heavily.

That is awesome!

I am curious about the decision to not use GPU since this is for Apple Silicon.

Wouldn't the GPU potentially accelerate the DeltaNet/attention layers and matrix multiplication in general?

I mostly agree with the prediction though maybe a bit more pessimistic about the timeline. Also I'm not sure our current usage of parameter count would make sense in this scenario, such a feat would require compressing current parameters in a manner much different then something producing a bit count per parameter. A hypothetical example would be using a single seed parameter per layer which then passed into a noise function produces the functional weights for that layer, able to reduce per weight size to sub bit levels (256 bit seed, producing 16K weights).

> on a decent speed

But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.

> I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware.

This prediction alone isn’t useful at all without a bound on speed and maybe quantization.

You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.

If you’re saying that hardware will catch up by mid-2027, I disagree. The limitation is fast memory and that’s going to be expensive for a while. I have a 128GB unified memory machine that can technically run 200B MoE models with enough quantization, but it’s so slow that there’s no reason to do it. It’s going to be a few years before we have enough RAM and processing power in basic consumer hardware without spending as much as a used car to get it.

> This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.

Sorry, but 9 tokens per second with a slow prompt processing speed is not decent for anything other than getting short chat responses.

You’re also not running the full GPT4 quality model. I’m very familiar with that model from some other work and the 4-bit quants are just not as good as all of those KL divergence plots would have you believe.

You also have very short context. It’s basically useless for anything more than short chats where you’re okay watching the output come back at reading speed, skipping the reasoning part (which is important for calling it GPT4 level quality), and waiting a long time for the first token.

Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.

Downloading now just 'cause the repo name

How are the thermals? I noticed that running any serious workload locally heats system fast.

> I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat

It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that, seems there was another collaborator ;)

Show something you've built with the model+tooling instead, truly dogfood it. I'm sure you'll discover things along the way too!

By early 2028, major players like Intel, AMD, QC will ship accelerators in consumer laptops capable of running ~1T MoE models at ~100 tok/s

I tried Qwen3.6-35B-A3B, but it couldn't generate a 50-100 line Clojure file without having broken parens mismatches. I know Clojure isn't super popular, but the syntax is pretty simple and the frontier models do fine with it.

> I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second.

That is no where near decent at all.

I don't pay anywhere near 0.30usd in the US - I pay half that off peak and can buy 1000$ worth of batteries to load up on super off peak (0.11usd). Also the inference providers are fighting over market share with huge debt loads so they are definitely going to go up in price.

> Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally.

Maybe, but for how long? Prices keep going up, and every new model eats more and more tokens...

Of course efficiency matters, but a lot of people either have cheap electricity or efficient hardware. My AMD strix halo home server can serve Gemma4-26B at like 70 TPS (rough estimate, I don’t remember the exact speed buts its fast af) while only using 100W.

Not OP, but your math is a bit off - I have solar panels :)

don't care, and yeah i don't like to hear it. we don't run local because it's cheaper money wise. we do it for freedom, for privacy and having option makes it cheaper in the long run. if there was no local options, your cloud model would cost much more!

It would be more efficient if you had multiple users (or agents) making parallel requests to take advantage of batching, right?

You pay 3x as much for electricity as I do, so the math here is going to work out very differently depending on a lot of factors.

Locally I’m looking at about CAD $0.05 per kWh when off peak.

its 18k not 180k

It's the "Race-to-Idle" situation all over again. It consumes less power to complete a task faster, whereas using "low power" hardware that draws max TDP for 30 minutes isn't very power efficient.

The privacy nuts have a better leg to stand on, but even then it's hard to believe that they're using on-prem AI to replace SOTA model inference. As cool as local LLMs are, a lot of the stuff people run is a novelty.

Have you tried with a single CPU to get rid of the NUMA penalty? I understand this likely means halving the memory but I am interested in how much of a difference it makes

Thank you for sharing!

Thank you for sharing / linking!

But OP is using Q8 and you're using Q4?

Reasonable analysis, especially because this person seems to have an actual house. In my case, I rent and don't pay for electricity directly, so the cost effectiveness threshold is whenever the landlord starts complaining

I think, may be actually wrong, that most of us do not consider running a model locally a way to save money. It is a way not to spread personal info around.

It gets better in the cooler months when heating is running in a home :)

I bought a trashcan Mac Pro on a whim last week ($120 in eBay!) and did some reading about them—it turns out people recently started using the GPUs to run Qwen4 @ 20-30 tok/s.

Here's some of the resources I came across if you're interested in reading. I'm excited to get my mitts on it on Friday.

https://echalupa.com/blog/mac-pro-6-1-llama-cpp-firepro-d300...

https://matthewgribben.com/blog/mac-pro-6-1-llama-cpp-firepr...

What is it useful for at slow speed?

hey, I’m the author. That box has 384gb, but loading the model “only” uses about 80gb.

Gemma 4 26B is capable up to 256k or 262k, can't remember which.

Whether the writer's setup affects that choice I don't know.

This reads as pretty clearly AI-generated text, which is against HN guidelines.

If the local model is served via ollama, there's a default timeout of 10 minutes , which can be adjusted either per-call , or (as I did) in the systemd service environment variables

https://docs.ollama.com/faq#how-do-i-keep-a-model-loaded-in-...

You didn't specify what was serving your local model.

Not everyone is as confident in their writing as they are in their engineering