I don't care who King Charles is every single time

that's the trick and a multi-billion dollar question, how would an llm engine know that? it's an active research area how to cull the initial layer surface and do the optimal traversal path through the layers and it's a damn hard problem. It's definitely an area where a ton of performance is left on the table still.

> I don't care who King Charles is every single time

There is just a stupendous amount of everything packed into a 35B size or larger model. For instance Qwen 3.6 35B A3B (Q8) can do a fairly decent job translating English to Arabic, but it can also generate python code with a reasonable layout and commenting.

I try to remember that as a mental model, an epub copy of a truly gargantuan sized 1000+ page novel such as the unabridged/2nd revision of Steven King's The Stand is about 800KB, and we're talking about a GGUF file that's 37GB in size or something like that.

> I always wondered why we need to shove the entire model into memory, I don't care who King Charles is every single time.

That's kind of the problem, isn't it? How do you know which part of the model to put in memory? You have to make a per-parameter decision of whether or not it's worth it to have it in memory or whether the value should just be treated as zero. Then you have to "re-link" the layers of the model to the new positions of each of the weights. For billions of parameters, that's a lot of calculations. And it requires us to know what each parameter actually represents, which nobody does.

> I don't care who King Charles is every single time

Always curious when someone will figure out how we can elide most of the data from an LLM (but retain the logical ability). I don't actually need an LLM to have a very big internal knowledge base to be useful, so long as it can invoke a search tool...

> It always felt as though we already figured out how to break up large files and parse them efficiently with very little memory.

The A in 26B-A4B is the active weights.

The problem is that this is a per-token load/unload at best, not for the whole prompt.

The division happened until one of these can fit in a single GPU and they stopped scaling it down any more, because you can wire up 8 of them to do their share of the work.

You're kinda describing the MoE architecture; you can offload expert layers and stream them as-needed if the experts are small enough and the SSD is fast enough.

Dense LLMs typically perform better, but slow down much more than MoE models when you try offloading layers.

You're right, but the "just" in "just finetune" is doing _a lot_ of work here.

It's still early days and we "just" don't really know how to do it well.

The problem is that knowledge in a LLM is separated in the mathematical sense (vector direction) but not necessarily meaningfully grouped in the matrix (would be easier to split between disk/memory) if such.

I think a rough analog is that it would be difficult to organize the rows of a hash table of everybody in a country by their geographic location.

At this point, people are just thrilled LLMs can even function as they do…

Games used to render entire map in map in memory. Later they figured only the surroundings can we rendered. We can hopefully get same in AI

Indeed, it begs the question why we have "everything" models where instead we could have very efficient "something" models. Typical LLMs out there can generate code and translate between 60 different languages. Sometimes I only need the first part, sometimes the second. Two distinct models would be a lot smaller and run much faster (token-wise).

One key thing with MoE I am still not understanding is why we don't keep the same expert in memory for a larger number of tokens than 1. Why do we route to some other expert every token? Wouldn't it be more memory efficient to generate at least 2,3,4,8,12 or 32 tokens and then swap the experts?

Apple does something similar with their latest foundation model. They process input prompt and based on results they preload required experts. Quite neat solution for the edge devices

Putting the whole model in memory is far faster then swapping to disk.

You don't know whether King Charles or 42 is the answer to "the Question"

my 2c on King Charles vs 42.

I don't think the problem is that memory footprint can't be reduced. The problem is that the token generation rate is simply far too low.

OP suggests that the token rate of of their solution is ~5 per second. That's at least an order of magnitude slower than commercially available models.

> Frontier AI feels like its full of people who are brilliant at making models, but when it comes to scale and practicality, they just leave it to whoever sets up infrastructure to worry about.

The idea that people in 10+ frontier labs (OpenAI, Anthropic, Google, Alibaba, Z.ai, DeepSeek, xAI, Amazon, etc) in a trillion dollars industry are all dumb is frankly, hilarious.

Thank you! That’s useful. I might try lowering the minimum version later. The 2.4x prefill improvement will only work on the apple10 GPU family. The M1 uses apple7 as I remember

To build this (on macOS 15) I also had to do this (in Package.swift):

    @@ -4,8 +4,7 @@ import PackageDescription
     let package = Package(
         name: "TurboFieldfare",
         platforms: [
    -        .macOS(.v26),
    -        .iOS(.v26),
    +        .macOS(.v15)
         ],
         products: [
             .library(name: "TurboFieldfare", targets: ["TurboFieldfare"]),

Why are you still on 15?

Nice! Can you mention what kind of prefill numbers you’re seeing?

Thank you very much for sharing! Great results and useful info!

M4 Max is typically better than M5 Pro for inference IIRC.

playback in Resolve would probably just use hardware decoding and barely hit your CPU or GPU. RAM usage would also not be much.

at that much ram you can just load it outright without tricks. it will be much faster even if it ends up swapping.

My first version used plain `mmap`. On the 8 GB M2, a cold 3.36 MB expert took 10 ms with mmap and 2.8 ms with `pread`. The full simulation was 0.50tok/s for `mmap` vs 4 tok/s for `pread`

With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work

And common weights still use mmap for simplicity

So, I believe llama.cpp might run it under 2gb, but I assume it will be slower

Ya I'd be interested to see a comparison of using llamacpp with ssd offloading to compare real speeds.

Cool project! I looked into it recently and thought that running diffusion models locally doesn't really make sense: https://eamag.me/2026/why-parallel-diffusion-llms-are-slow-o...

What are your thoughts on this?

It is super cool! Diffusion Gemma was released around the middle of my project, and I seriously considered switching to it. But I decided to finish the project as it was.

I believe it would be a perfect match!

Feel free to use any parts of my project or drop me a message. There’s my LinkedIn link at the end of the readme. Or I will drop you a message later!

I tried out one of the NVidia diffusion models, and from memory it only worked on MLX but seemed to leave a lot of features out. Would your work support non-Gemma models too?

I am not native, my English is far away from perfect. I am using LLMs for checking my texts or grammar. I always trying to edit it properly, but sometimes I missing parts like that because I don't really have this "language feeling" as natives. Apologies for this

I grind my teeth when I see it. It's so pervasive that I worry I'll pick up the same ticks by reading so much Claudeslop.

"I ran more than 100 experiments. Most didn’t work. A few got me here."

And here.

I'm sure it was a chatgptism first, but I wouldn't accuse a western company of distillation.

In all fairness, maybe it's just that they let some post-2022 recipe blogs get into the training runs around ~4.6-4.8 time

Let my karma burn for saying this: Maybe it is time to let this go man. These comments are really the new incarnation of "grammar policing". (1)

They don't add anything of value, did the author use an LLM to fix his prose but no useless slop was added in the process: who cares ? Is the article useless slop: fine, downvote it to oblivion.

(1) For those not old enough to remember that wonderful practice please use your nearest LLM to find out or, you know, visit a library and do your own research.

In a way, the AI-text-policing is a golden age. Allasudden, folks actually give a shit about style??? Before AI was there _ever_ comments on HN "bro, the semicolons ... I just can't"?

Everyone's on high alert. Maybe writing will get better!

Okay, I’m the first to get annoyed at LLM-ese. But language requires the ability to say that something is A and not B. There are certain phrasings of that that are painfully Claude-esque, but ffs the one you quoted is the type of thing an actual human being is just as likely to say.

> The same MTP head could also be used to speculative prefetch the expert weight residing on the SSD. If the expert weight can be preloaded before the GPU actually need them, the speed penalty from VRAM cache miss will be quite reduced.

When using SSD streaming, the GPU is practically always waiting for the SSD to fetch the right expert, rather than the other way around. There is basically zero slack on the SSD side, so I'm not sure how "prefetching" is supposed to help. It would mostly hurt by fetching the wrong predicted experts, which already makes conventional MTP practically unhelpful for typical (not widely batched) SSD streamed inference.

Worth mentioning why this is harder than it looks.

There is a different set of experts at every layer, and each layer has a small router that decides which ones to use.

The router needs to look at the state produced by the experts below it.

Drafted tokens from the MTP head can be used to predict which experts the first layer will want, but not beyond that. To know what layer 10 experts needs, you have to run layers 1-9 which means loading their experts.

So, yes, instead of a next-token drafter like MTP, you'd want something trained to predict the expert activation across all layers at once.

The M5 SSD's performance uplift was fairly substantial, even when compared to the prior generation.

> In the Blackmagic Disk Speed Test, the SSD in the M5 MacBook Pro achieved read speeds of up to 6,323 MB/s, compared to just 2,031 MB/s on the M4 MacBook Pro. It's not like the M4 is "slow" in a vacuum, but the M5 SSD is over three times faster, which is a great generation uplift.

https://www.tomshardware.com/laptops/macbooks/m5-macbook-pro...

My suspicion is that this is simply due to the M5 having more memory, and the OS already having most of the file cached. The M2 has more memory pressure and would cache fewer of the SSD reads

If that's true, inference speed would be even lower if you have only 2GB total, including OS caches

I am relying quite heavily on system caching and pread. And yeah, M5 is a way faster and I can guess Mac can cache something, even if process stays under 2gb.

It was 83ms read per token for M2 and 12ms on M5 pro. Total is 163ms/tok vs 30ms/tok for M5. So yeah, there is a faster read and faster gpu processing

Not only is it older, so Pro:Pro it would have much slower SSD, but doesn't the Air also have slower SSD than the Pro in the same generation? And maybe narrower memory bandwidth?

The M5 MBP has 24GB of RAM, more context in RAM perhaps?

Thanks for sharing! SSD read speed is the biggest limiting factor here, unfortunately

Yeah! Check the Colibri and Flash-MoE projects. They’re already doing that.

https://github.com/danveloper/flash-moe https://github.com/JustVugg/colibri

How large?

With 64 GB of unified memory, you should be able to run a DeepSeek V4 Flash quantisation at 7–10 t/s, for example with: https://github.com/antirez/ds4 or https://github.com/steadfastgaze/MoEspresso (my engine).

The routed experts needed for the next tokens that are not already in memory need to be read from the SSD, so the speed becomes SSD reading bound and the larger the memory, the faster the inference.

Without fundamental model architecture improvements the practicality largely depends on how Apple increases memory bandwidth.

Memory bandwidths (* = rumored):

  M1:       68 GB/s
  M2:       100 GB/s
  M2 pro:   200 GB/s
  M2 max:   400 GB/s
  M2 ultra: 800 GB/s
  M5:       153 GB/s
  M5 pro:   307 GB/s
  M5 max:   460 GB/s
  M6:       200 GB/s*
  M7:       240 GB/s*
  
  Nvidia 4090 1008 GB/s
  Nvidia H100 3.35 TB/s

My assumption is that the difference is 90% from more memory. I'm making several assumptions because nothing here looks groundbreaking so I don't care to dig deeper, but the model + KV cache definitely cannot fit in memory on the 8GB machine, but probably can on the 24GB machine—or can at least get close. Assuming that this benchmark makes use of that, skipping SSD streaming will speed things up massively (I would have guessed much higher than the reported 6x speedup).

I have an M5 128GB. Being on the cusp of practical is a good description. It will run, but prefill and token gen are still slow relative to my consumer GPU box.

It also gets very hot. If you’ve never heard the fans on Apple Silicon really spin up, it could surprise you. Makes the full GPU setup feel quiet by comparison.

I think after the hardware market calms down the ticket is going to be a light laptop with a second dedicated inference server on the network.

I think it's M5 MacBook Pro, not M5 Pro, as he mentioned.

The speed of this perfectly correlates with the memory bandwidth of an M2 vs M5 Pro.

Thank you! Thankfully this is purely software engineering problem. And as usual there is no free lunch. You trade speed for lower memory usage

This approach will only work for MoE models. There is a Qwen 35b-a3b. You just need to do GPU stop after router and read the requested experts to ram. And it is possible to build similar engine for this model (or feel free to adopt my engine) Not sure about generic approach for now, but coding with ai agents is relatively cheap now, you can try it

What exact specs do you have? It might be because it's the 256 GB version. afaik, those versions have much slower memory bandwidth than the 512 GB models

My friend tried it on an M4 MacBook Pro and got 25–27 tok/s

There was an ai winter for very long time. The math for NNs was already here, but not enough compute/data

I saw a pretty cool project to run an llm on an esp32 device https://github.com/slvDev/esp32-ai

This comment is giving "100% tested virus free" on a free downloads site vibes.

In-prompt "security" is not reliable. You can not tell if the LLM/agent actually followed your instructions or whether it fell for a prompt injection.

Thanks! I wanted to add hugging face token field to speed up model download, but then I realised that people might not trust to give their tokens And yeah, local models are better for security, at least your conversation stay on the machine

seems like a great little chrome extension or tool we could use to just quickly validate stuff like that.

Please don't do this. I know you mean well, but if you think you're providing a service here, you're not. This is not the same as posting an archive.today link to a paywalled article. This is not actually contributing anything to the discussion. Anyone who wants an LLM review can do so themselves. You have no idea if this is good output or slop. Nobody else knows if you even actually sent this through an LLM or not.

"Review this project" is that how you use LLMs lmao

Just toss GBs of file structure: "AI, do your work baby!"

I for one break things down much smaller into very specific tasks involving very particular text. Maybe I'm overdoing it lol.

For me, an AI security review would still take hours or days, it would hardly be a 1-shot prompt like this.

Thank you very much!

I think it will throttle quite soon, but I haven't tried runs longer than 30minutes with this engine.

However, there is no constant load on ssd or gpu. i/o and gpu work are alternating and there is a brief idle periods for each i/o and gpu during inference (because gpu waits for i/o and after that i/o waits for gpu)

This is where MoEs shine though. You don't need all experts in memory at once. Diffusion inference doesn't have sparse inference.

The full route changes almost every token. The cache works through partial reuse, about 40% of experts repeat on the next token and 57% within two tokens, cutting I/O from 166 to 88 ms/token on M2 Mac.

The longest exact repeat we found was only two tokens. Coding tasks may have higher reuse if code related experts are selected repeatedly

I haven't tried it but it should work! You can try it and share your results, it would be really appreciated

I tried it on my wife's M1 MacBook Air 512GB and it gets 4–5 tok/s

Also, it must be easy to adjust for iPhones and iPads in theory

Yes, a Neo is roughly equivalent to an M1.

It heavily relies on M-series Mac unified memory architecture. And shaders are written using Metal, Apple's own gpu programming technology. It cannot be ported directly to classic architecture (ram+vram)

Thank you for testing and sharing results!

I think I might understand your use case. You ssh the Mac and want something like `ollama run` with an interactive chat in terminal. Am I right?

There is already experimental OpenAI-compatible server in this repo:

``` swift build -c release --product TurboFieldfareServer .build/release/TurboFieldfareServer \ --model scratch/gemma4.gturbo ```

After that a small terminal client can run inside the same ssh session and talk to `/v1/chat/completions`

The client needs to keep a messages array, add each user message, send the full array with `stream:true`, print SSE chunks until `[DONE]`, then add the response back to the array. `/reset` can clear it

There is a python example in the server docs. (https://github.com/drumih/turbo-fieldfare/blob/main/docs/OPE...)

It is non-streaming, but can be used as a starting point.

The server is still experimental and I am fixing some problems currently. But you can try to vibecode a simple terminal client around it.

If not, create an issue on Github and describe desired behaviour

It depends on the use case.

I measured this exact model with a 4k context on the mlx engine. It runs at 75 tok/s on my M5 Mac Pro and using 14 GB of RAM. For my engine the same model uses 2 GB of RAM and produces 31–35 tok/s.

The project is still experimental so performance may vary as it continues to improve. If you want to save around 12 GB of RAM for other tasks and you are ok with 35 tok/s (afaik it is roughly comparable to ChatGPT’s speed for basic responses) my engine may be a good fit.

If you need maximum speed and flexibility just use MLX

you could use mine ... github.com/0gsd/enough (it has other stuff too)

hm. just open repo, copy commands into your terminal and you will get app installed (if you have swift toolchain installed)

after that download 14gb of weights and enjoy offline inference (and a bit of Gemma4 intelligence) for your everyday tasks

multi turn chat is coming!

Uh, I’m afraid it is Apple only. It is written using Apple’s GPU language, Metal, and heavily relies on the Apples’s shared memory architecture

Windows PCs would require a completely different approach

Switching from mmap to parallel pread. From 0.5tok/sec to almost 4tok/sec. Running GPU work while reading missed experts also helped a lot, 4.4 -> 4.7

You’re right, Gemma isn’t the best model for coding (afaik more "everyday tasks" related). My first idea was to use Qwen, but its architecture was much more complex to implement in this stack. I chose Gemma so I wouldn’t spend all my time debugging custom kernels and could actually move the project forward with simpler approach

Gemma 4's tool calling was recently fixed; that was the main issue with agentic use in my experience.

Otherwise IMO it codes about as well as the Qwen MoE for PHP and SQL. It's a fully impressive model (though it is not as mindbendingly impressive as the 12B, which is outrageously good for its footprint)

afaik there is some research at this area. Also the new apple foundation model uses related idea. they process the whole prompt and based on prompt load required experts and use only these experts for generation. It doesn't require fitting full model into memory or per token ssd streaming

m5 device also uses the same approach. the same 16 cache slots. and experts are evicted from memory as needed. And activity monitor shows 2gb usage for m5 pro (24gb btw)

Yeah, I tried both rearranging experts on disk and predicting the next expert using statistical approach. Reordering helped on the test prompt, but failed on another prompt. Markov and cross layer prediction didn't work either

all the effort in this space is going into multiuser, datacenter workflows for high throughput inference. running in resource constrained environments is not where the money is. but it will be, especially if we look forward to a world where having 512gb unified ram is normal for "workstation" machines. The semiconductor space is slow enough to respond that it's likely it will take a few years before production capacity has ramped up enough to get us past the current supply crunch but it seems inevitable to me that we'll be able to run huge models like kimi 3 locally in the next few years (maybe 2029/2030 for it to really be affordable)

I think it depends on usage pattern. You trade speed for lower memory usage. Maybe engine specialisation and faster SSDs is the future for local inference, who knows

Or maybe it's just lazy programmers, wouldn't be the first time.

uh, it's a bit difficult to discuss the classical approach with vRAM and regular RAM. Not really familiar with optimisations and hacks, I always worked with apple platforms and shared memory. But description sounds cool, good luck with your project!

For Mac I would start from MLX engine. For exact model choice it is better to check bench results, and select model based on your need. A lot of good feedback about Qwen3.6, but I haven't used it in my tasks

I think I first saw Flash-MoE (https://github.com/danveloper/flash-moe) in April. Huge respect to them, it was a big inspiration for this project!

https://github.com/antirez/ds4 coming out at the same time I started a new job and they gave me an m5 max a few months ago was the lightbulb moment for me.

I double checked m2 logs. Cache hit rate is about 59-69%. 250-320MB went through `pread` per generated token. It is 3gb/s during this i/o phase.

Please don’t use Ollama.

https://sleepingrobots.com/dreams/stop-using-ollama/

Thank you!

afaik ollama relies on llama.cpp and mmap. mmap loads pages on demand and doesn't use the same explicit cache or parallel reads like my engine. Most likely ollama/llama.cpp will be way slower in this case

Have you tried the new chat template Google released recently? It’s supposed to address this and enable reasoning content preservation. I have not tried it myself but am hoping it does the trick, since Gemma is great model otherwise.

Yeah, Gemma is not the best for coding I guess. qwen must be better

Size reduction is mostly based on Experts size. And it is limited by SSD speed. Check for Colibri and Flash-Moe, they are doing similar things with bigger models, but tok/s is not high

not with this engine. Kimi is a very different model. You can try to check HN later, I believe someone will build engine for this model for edge devices

Yeah, it must be exactly the same. The same weights are used, nothing skipped or pruned. But it might have differ to MLX for greedy decode because of small floating-point nums difference

I think there is a limit based on MoE number of active parameters and quantisation, bytes count for active experts. But I believe we will see more project like this for different models.

Thank you! Under good conditions it achieves approx a 67% cache hit rate with 16 expert slots

Uh, maybe, who knows. I am pretty bad solving leetcode, btw

Not sure it will be really usable. Check for Flash-Moe and Colibri repos A lot of request for qwen3.6 moe, it might worth exploring

Yeah, must be possible. Not fast, but possible if you have enough ram. I think you can search online for projects, I think I saw something related

Gemma E2B QAT will run on a 4 gig RAM pi.

Reads don't wear out flash memory to any meaningful extent.

AFAIK it should not because it is only reading

Gemini uses MoE and context caching, which is a similar approach.

You are not really accessing the biggest frontier model every time, and you're not really doing an end-to-end LLM request on each prompt.

I would go so far to say frontier models have peaked and improvements from here come from clever (or very elaborate) harnessing. "LLLMHs" - Large Large Language Model Harnessing !

This is an odd comment: the project is right there for you to use, so just try it and see if it holds up to the claims? Then you can comment about the fact that it either doesn't hold up, with numbers to back that up, or on how awesome it is because it works =)

It is Apple platform only implementation because of Metal (and Swift). Other platforms would require CUDA or Vulkan and a complete rework

Apparently Kimi K3 has 104B parameters active at a time. So at 4 bits you'd need 52GB just to hold the active params.

That said, in theory this same technique should be able to run it on a 64GB Macbook, probably at <1 tps.

Yeah, the same ideas should work for qwen. You can try porting this engine to use Owen.

Owen 3.6-35b-a3b was my initial idea, but I switched to Gemma because of its simpler architecture and kernels

I'm curious too!

One obvious thing is that the memory requirements for this are substantially smaller than DwarfStar-- which AFAIK can only start to be used at 64GB ram and upwards. Another obvious thing is that antirez is pretty obsessed with making sure that DwarfStar passes all of DeepSeek V4 Flash's generating tests (loosely). I suspect that is also true of DwarfStar's implementation of GLM5.2, but I don't use that.

DS4 is designed to do real-work. Gemma 4 is not going to cut it.

uh, I don't think it is possible to compare them. DwarfStar4 is for high end macs and a lot of ram. this project is more targeted to low end devices and "general use" Gemma4 model

Thank you! Let me know how it goes and share your tok/s results

Uh, not really, unfortunately. Asahi uses Vulkan for gpu, but kernels for this project are metal.

Thank you! Share your tok/s results later

My friend just compared this project to running Cyberpunk on a very old machine at 16 FPS, huh

Thank you! If you can use it for your tasks I would be happy!