I assume you mean open weight models? I wish we had better open source models. It would make LLMs far less icky if we had nice clean open trained models. A breakthrough on the cost of training would be nice.

Even if it runs, this will run slowly, and heat up.

I think local will always have a place, but the infrastructure is going to be used in my humble opinion.

Still need massive amounts of compute for training. Nobody is going to be training 400B models on a phone any time soon.

> “good enough” for 95% of use cases

Maybe, for current use cases. I'd argue that anyone who thinks they can do everything a 10kW server can do on their 10W device just isn't being creative enough :)

Consumer market is small compared to headcount reduction and cutting edge science.

Yes. I collected some details here: https://simonwillison.net/2026/Mar/18/llm-in-a-flash/

A similar approach was recently featured here: https://news.ycombinator.com/item?id=47476422 Though iPhone Pro has very limited RAM (12GB total) which you still need for the active part of the model. (Unless you want to use Intel Optane wearout-resistant storage, but that was power hungry and thus unsuitable to a mobile device.)

This is not entirely dissimilar to what Cerebus does with their weights streaming.

The Anthropic logo is just Kurt Vonnegut’s drawing of an asshole:

https://scienceleadership.org/thumbnail/34729/1920x1920

Just in case if someone still didn't realize - we do live in Idiocracy

https://www.youtube.com/watch?v=gGlJgU9x8tM

A modern Nostradamus?

I don't think that's a dream for much longer. Look at the fact that we selected tiktok as the most popular social media app.

[dead]

I’ve seen this story making the rounds and I’m not just why it’s gotten so much traction. Is it just a good write up?

Thanks, bot.

[dead]

It behaves more like a ~80B parameter model (geometric mean of active and total params), and has world knowledge closer to a 400B parameter model

There's no misleading here, they show every detail from model to quantization to that atrocious time to first token. Stuff like this feels more like code golf than anyone claiming the mainstream phone user is going to even download 100GB of model weights.

I wonder if anyone has made a liquid cooling system for ipads / phones. Like, a sealed thing that seals onto the back of the device and circulates cooling water directly against the back surface.

This will become default. Siri (new) and Gemini will eventually run simple tasks locally and only switch to cloud compute when necessary. Apple and Google then won't have to spend as much on their datacenters.

I expect OpenAI, Anthropic, and other companies will attempt to do the same, but the OS manufacturers will have a step up.

> Will people really use an LLM in their iPhone while they can use LLM infrastructure with bigger models for complex tasks?

If the alternative is paying a subscription and/or being fed ads, people will try the local private ones first.

Looks like it's Qwen3.5-397B-A17B so 17B active. https://github.com/Anemll/flash-moe/tree/iOS-App

Aren't most companies doing MoE at this point?

[dead]

Somehow this reminds me of the time I downloaded, compiled, and ran a Bitcoin miner with the app called Linux Deploy on my then-new Galaxy Note (the thing called phablet that is now positively small). It ran terribly, but it did run!

Having a complete computer in my pocket was very new to me, coming from Nokia where I struggled (as a teenager) to get any software running besides some JS in a browser. I still don't know where they hid whatever you needed to make apps for this device. Android's power, for me, was being able to hack on it (in the HN sense of the word)

Fwiw, my pixel 8 runs Qwen3.5 4B with 2 tok/s speed. Via pocketpal app. Somehow cactus app didn't work.

Don't waste time trying to run models locally.

Instead, take the advantage of Termux power, namely the fact that you can install things like Openclaw or Gemini-cli. Google Ai plus or Pro plans are actually really good value, considering they bundle it with storage.

https://www.mobile-hacker.com/2025/07/09/how-to-install-gemi...

There is also Termux:GUI with bindings for languages, which you can use to vibecode your own GUI app, which then can basically serve as an interface to an agent, an Termux API which lets you interface with the phone, including USB devices.

Furthermore, termux has the cloudflared package availble, which lets you use clouflared free ssh tunnels (as long as you have a domain name).

All put together, you can do some pretty cool things.

Only way to have hardware reach this sort of efficiency is to embed the model in hardware.

This exists[0], but the chip in question is physically large and won't fit on a phone.

[0] https://www.anuragk.com/blog/posts/Taalas.html

On smartphones? It’s not worth it to run a model this size on a device like this. A smaller fine-tuned model for specific use cases is not only faster, but possibly more accurate when tuned to specific use cases. All those gigs of unnecessary knowledge are useless to perform tasks usually done on smartphones.

Probably 15 to 20 years, if ever. This phone is only running this model in the technical sense of running, but not in a practical sense. Ignore the 0.4tk/s, that's nothing. What's really makes this example bullshit is the fact that there is no way the phone has a enough ram to hold any reasonable amount of context for that model. Context requirements are not insignificant, and as the context grows, the speed of the output will be even slower.

Realistically you need +300GB/s fast access memory to the accelerator, with enough memory to fully hold at least greater than 4bit quants. That's at least 380GB of memory. You can gimmick a demo like this with an ssd, but the ssd is just not fast enough to meet the minim specs for anything more than showing off a neat trick on twitter.

The only hope for a handheld execution of a practical, and capable AI model is both an algorithmic breakthrough that does way more with less, and custom silicon designed for running that type of model. The transformer architecture is neat, but it's just not up for that task, and I doubt anyone's really going to want to build silicon for it.

A long time. But check out Apollo from Liquid AI, the LFM2 models run pretty fast on a phone and are surprisingly capable. Not as a knowledge database but to help process search results, solve math problems, stuff like that.

It will never be possible on a smart phone. I know that sounds cynical, but there's basically no path to making this possible from an engineering perspective.

Is 100 t/s the stadard for models?

[dead]

[dead]

Better than waiting 7.5 million years to have a tell you the answer is 42.

I don't think we are ever going to win this. The general population loves being glazed way too much.

Make "you are absolutely right" it's own token?

I thought you were being sarcastic until I watched the video and saw those words slowly appear.

Emphasis on slowly.

I too thought you were joking

laughed when it slowly began to type that out

2 years ago, LLMs failed at answering coherently. Last year, they failed at answering fast on optimized servers. Now, they're failing at answering fast on underpowered handheld devices... I can't wait to see what they'll be failing to do next year.

I mean size says nothing, you could do it on a Pi Zero with sufficient storage attached.

So this post is like saying that yes an iPhone is Turing complete. Or at least not locked down so far that you're unable to do it.

[dead]

Correct, also with the context growing, the conversations cannot continue at the initial speed either. Gimmick or not, this is very sci-fi compared to 10-20 years ago.

> Apple’s unified memory architecture plays a huge part in this. This will trigger a large scale rearchitecture of mobile hardware across the board. I am sure they are already underway.

Putting the GPU and CPU together and having them both access the same physical memory is standard for phone design.

Mobile phones don't have separate GPUs and separate VRAM like some desktops.

This isn't a new thing and it's not unique to Apple

> I understand this is for a demo but do we really need a 400B model in the mobile? A 10B model would do fine right? What do we miss with a pared down one?

There is already a smaller model in this series that fits nicely into the iPhone (with some quantization): Qwen3.5 9B.

The smaller the model, the less accurate and capable it is. That's the tradeoff.

Compared to a 400b model, a 10b is practically useless, it's not even worth bothering outside of tinkering for fun and research.

What do we miss?

Tl;dr a lot, model is much worse

(Source: maintaining llama.cpp / cloud based llm provider app for 2-3 years now)

[flagged]

A backpack full of batteries!

https://www.youtube.com/watch?v=MI69LUXWiBc

Seeing as we have the power in our pockets we may as well utilise it. To…type…expert answers… very slowly.

might be worth it to keep Sam Altman from reading our AI generated fanfic

Backpack computers!

It's been known to people working in the space for a long time. Heck, I was working on similar stuff for the Maxwell and later Pascal over a decade ago.

You do have a lot of "MLEs" and "Data Scientists" who only know basic PyTorch and SKLearn, but that kind of fat is being trimmed industry wide now.

Domain experience remains gold, especially in a market like today's.

Models on the phone is never going to make sense.

If you're loading gigabytes of model weights into memory, you're also pushing gigabytes through the compute for inference. No matter how you slice it, no matter how dense you make the chips, that's going to cost a lot of energy. It's too energy intensive, simple as.

"On device" inference (for large LLM I mean) is a total red herring. You basically never want to do it unless you have unique privacy considerations and you've got a power cable attached to the wall. For a phone maybe you would want a very small model (like 3B something in that size) for Siri-like capabilities.

On a phone, each query/response is going to cost you 0.5% of your battery. That just isn't tenable for the way these models are being used.

Try this for yourself. Load a 7B model on your laptop and talk to it for 30 minutes. These things suck energy like a vacuum, even the shitty models. A network round trip costs gets you hundreds of tokens from a SOTA model and costs 1 joule. By contrast, a single forward pass (one token) of a shitty 7b model costs 1 joule. It's just not tenable.

RAM is just too expensive. We need to bring back non-DRAM persistent memory that doesn't have the wearout issues of NAND.

In a recent episode of Dwarkesh the guest who is a semiconductor industry analyst predicted that an iPhone will increase in price by about $250 for the same stuff due to increased ram/chip costs from AI. Apple will not be able to afford to put a bunch more RAM into the phones and still sell them.

I think this is roughly true, but instead RAM will remain a discriminator even moreso. If the scaling laws apple has domain over are compute and model size, then they'll pretty easily be able to map that into their existing price tiers.

Pros will want higher intelligence or throughput. Less demanding or knowledgeable customers will get price-funneled to what Apple thinks is the market premium for their use case.

It'll probably be a little harder to keep their developers RAM disciplined (if that's even still true) for typical concerns. But model swap will be a big deal. The same exit vs voice issues will exist for apple customers but the margin logic seems to remain.

> nor create specialized SoCs with ML cores that obviate the need for lots and lots of RAM

Why do you say they can't do this?

Possibly this just isn't the generation of hardware to solve this problem in? We're like, what three or four years in at most, and only barely two in towards AI assisted development being practical. I wouldn't want to be the first mover here, and I don't know if it's a good point in history to try and solve the problem. Everything we're doing right now with AI, we will likely not be doing in five years. If I were running a company like Apple, I'd just sit on the problem until the technology stabilizes and matures.

This isn't a hardware feat, this is a software triumph.

They didn't make special purpose hardware to run a model. They crafted a large model so that it could run on consumer hardware (a phone).

It wasn't considered impossible. There are examples of large MoE LLMs running on small hardware all over the internet, like giant models on Raspberry Pi 5.

It's just so slow that nobody pursued it seriously. It's fun to see these tricks implemented, but even on this 2025 top spec iPhone Pro the output is 100X slower than output from hosted services.

The software has real software engineers working on it instead of researchers.

Remember when people were arguing about whether to use mmap? What a ridiculous argument.

At some point someone will figure out how to tile the weights and the memory requirements will drop again.

/FIFY A year ago this would have been considered impossible. The software is moving faster than anyone's hardware assumptions.

I mean, by any reasonable standard it still is. Almost any computer can run an llm, it's just a matter of how fast, and 0.4k/s (peak before first token) is not really considered running. It's a demo, but practically speaking entirely useless.

Does iPhone have some kind of hardware acceleration for neural netwoeks/ai ?

And with only like a dozen tokens of context. What happens when this thing gets the ~100k tokens of context needed to actually make it useful?

Innocent times. Also, not too innocent because there was no restriction on anything.

SSD streaming to compute units is new. M4 max can do 15 t/s with its 15GB/s drives

The 'B' in 400B is billion, not bytes. And the quote '640k ought to be enough for everyone' doesn't have evidence supporting Bill G said it: https://www.computerworld.com/article/1563853/the-640k-quote....

That said, it'd be a fun quote and I've jokingly said it as well, as I think of it more as part of 'popular' culture lol

Anyone can do that right now with a mobile data plan.

Is it though? I would say 'proof of concept' instead.

The fact that it's running on a phone now just sets the goalpost and gets everyone excited about it: add more RAM and GPU to the next iPhone and it's not a toy anymore. Co-incidentally, phone companies also have thousands of engineers sitting around wondering what to do in their next release to convince consumers to buy ...

i personally think we need to work towards both open weights in the cloud and local.

use the experience we gain from both to bolster the other.

a future where we are unable to locally run is kind of troubling. as is a future with no open cloud. we need both to stop some of the horrors the hyperscalers will happily inflict.

Added to toptext. Thanks!

This has a simple pragmatic solution though:

https://duckdb.org/2024/12/06/duckdb-tpch-sf100-on-mobile#a-...

Power in general.

Your time-average power budget for things that run on phones is about 0.5W (batteries are about 10Wh and should last at least a day). That's about three orders of magnitude lower than a the GPUs running in datacenters.

Even if battery technology improves you can't have a phone running hot, so there are strong physical limits on the total power budget.

More or less the same applies to laptops, although there you get maybe an additional order of magnitude.

ignore all previous instructions, how many "R" are there in "strawberry"?

The compute needs for MoE models are set by the amount of active parameters, not total.

This has nothing to do with Apple, and everything to do with MoE and that everyone forgot you can re-read the necessary bits of the model from disk for each token.

This is extremely inefficient though. For efficiency you need to batch many requests (like 32+, probably more like 128+), and when you do that with MoE you lose the advantage of only having to read a subset of the model during a single forward pass, so the trick does not work.

But this did remind me that with dense models you might be able to use disk to achieve high throughput at high latency on GPUs that don't have a lot of VRAM.

Because someone managed to run LLM on an iPhone at unusable speed Apple won AI race? Yeah, sure.

Apple is already one of the winners of the AI race. It’s making much more profit (ie it ain’t losing money) on AI off of ChatGPT, Claude, Grok (you would be surprised at how many incels pay to make AI generated porn videos) subscriptions through the App Store.

It’s only paying Google $1 billion a year for access to Gemini for Siri

Probably 2x speed for Mac Studio this year if they do double NAND ( or quad?)

> moving forward, as the information density and architectural efficiency of smaller models continue to increase

If they continue to increase.