Qwen3.8-27B on the iPhone 18 Neural Engine at 15–70 tok/s (1.5 bpw)

30 pointsposted 12 hours ago
by anemll
(twitter.com)

13 Comments

analog_daddy

12 hours ago

Hi. Could anyone share any recommendations (hopefully with actual links) for App Store apps to try running such local models? I’m not really interested in side-loading since it currently seems (to me) more of an hassle than it’s worth.

App Store search is broken in a sense that there might be so many apps claiming to be local but once you read reviews, it’s just another scam where it’s it’s usually sending your query to remote server.

woadwarrior01

11 hours ago

I'm the dev behind the Private LLM app, which was the first on-device LLM app on the App Store. Don't download it yet. I'm currently finalizing a brand new v2 rewrite which can run models that the copycat app (locally) cannot run, like Bonsai 2 27B, Gemma 4 12B etc.

https://x.com/private_llm/status/2101769928252473488

analog_daddy

an hour ago

Thanks! All the best for your rewrite and definitely will check it out once it launches!

simonw

12 hours ago

My current favorite is this one: https://apps.apple.com/us/app/locally-ai-by-lm-studio/id6741...

It's free, not annoying, and has a bunch of model options.

jdgoesmarching

11 hours ago

Note that the model discussed in this article isn’t available in this app running on the phone. Still, it has some good local options like Gemma 4 and can link to the desktop LM Studio app which is cool.

createful

11 hours ago

Not Qwen, Bonsai. Although it is based on Qwen, and achieves similar scores, it is not the "full" Qwen.

Last I remember, when trying to run a model above 1-2 GB on iOS, it would refuse to do so, at least from Locally AI. iOS only allows apps to use about 1-2 GB of RAM before being killed. Though using their Neural APIs might bypass this.

spacecoder2

12 hours ago

Anyone know how hot an iPhone gets running the Neural Engine flat out for 10 minutes?

GeekyBear

11 hours ago

The neural engine is optimized for energy efficiency, and the GPU is optimized for performance.

You're going to get much better battery life and much less heat running on the neural engine.

anemll

12 hours ago

It's warm but not hot. On M6 it's about 50% less energy consumption in my tests vs GPU. I think iPhone should be similar