It is absolutely not worth buying hardware to run models for purely (long term) cost reasons. For open weights models the economies of scale means the cloud beats local significantly and your payback time is like 10 years.

However there are other reasons (e.g. privacy) that might make it worth running locally for some people.

I have a Strix Halo and dual 32GB GPUs in my desktop, that sit idle right now, because the electricity to run them and to cool them in 110F weather Texas is currently experiencing pretty much nulls any savings I might see over getting better models from cloud providers. While I mostly use Claude or Codex with subscriptions for agentic work, for API use DeepSeek has usually been my go to, but now I guess it's GLM 5.3 or the Flash version. And, for security work that Anthropic or OpenAI models are likely to refuse, I've been using Kimi K3 (also via subscription, though their subscription is extremely stingy), but I guess GLM is now the one for that, too.

Anyway, yeah, even at the prices I spent on my local AI stuff (I bought before RAMpocalypse really kicked into gear, so I bought old server GPUs for about $350 each and the Strix Halo for a little over $2k) it was never going to pay for itself; I just like to tinker. But, I can't imagine spending today's prices for hardware for local AI.

When the memory shortage ends, I'll be down to the Apple Store (or, more likely, clicking refresh on the Apple outlet every few days). But, until then, there continues to be a glut of cheap and free models in the cloud that are better than anything I can run locally and they're faster, too.

So far I don’t regret buying an M1 Max device with 32Gb of RAM. The models available for it keep getting better (running just about okay for interactive use) and 400 GB/s of bandwidth is still considered a lot.

The models are currently improving much faster than the hardware and this doesn’t seem to have plateaued yet.

Jalapeno is matching or very near Vera Rubin at 1/4 the power. I would not buy hardware now.

That's basically the question I'm trying to answer.

If you're paying Anthropic or OpenAI to use their models, harness, governance, etc., I could see the local inference potentially coming out ahead. They're already starting to ratchet down what your money gets you on their platforms, and that can be expected to continue as the leaders of those companies continue to seek the road to the El Dorado that is being a trillionaire.*

If you're looking to get into the guts of AI development instead of having it handed to you by a provider, that's where it gets murky. I'm wanting to write some sort of agent that does things and get into making outputs consistent in the like, and I'm not sure whether to host something on GCP or buy an M5 Mac.

*Note: El Dorado is a mythical city and many people died trying to find it.

Tools vs services in my mind. There is no guarantee any provider will continue to do what they are doing for you at the price they are doing it.

The object permanence of not having to reinvent the world every time a model gets sunsetted has value.

Part of it is knowing that whatever sort of enshittification the cloud providers do, my local programming environment won’t ever be less effective than it is today locally. It’s the same reason my entire development stack from editor to compiler is open source. I don’t need to modify it today, but I always must retain the option to do so later.

There are several things I do in my life that only pay off in the event of a big disaster, like an extended internet outage, civil unrest, supply chain disruption, war, etc.

I like to be able to do the things I do even if offline for weeks.

I spent a lot of money for more flash in my iPad Pro so I can keep all of offline wikipedia and OSM in it, for example, along with tons of books. It’s sort of like being a digital prepper. (Being a prepper is a spectrum, from anyone who keeps food in their pantry to people building bunkers under their house - how much you invest is a personal prudence and threat modeling decision.)

Also, privacy. And when I got the Mac Studio the 512GB was only $15k, which is dirt cheap for that much VRAM.

You'd also likely spend far more in electricity than the API cost of processing the prompt(s)

> "dual xeon"

Does inference make full use of the memory bandwidth in a NUMA system?

It’s not unified ram? I.e VRAM so it will struggle

I’ll be very curious what you get with DDR4. I also almost went that way. I have an Epyc DDR 5 rig and the best I see is 10 tok/s. Caveat being that’s at Q8 and a 4090 doing pre fill so it could be pushed up.

The surprising thing for me is how much work you will need to cool the banks if you’re near your memory ceiling. My memory starts soft throttling at about 74C (dies may be hotter, that’s the bank temp) and will turn down speed to try to stay below 80.

Happy to send my llama.cpp config settings if you want it.

Depending on which Epyc you got it might be slower than 1/5 of the speed.

Curious about that price, if you don't mind sharing a ballpark

I have a dual epyc + 1TB RAM. I could push glm 5.2 to 7 tok/s CPU only.

Honestly I suspect neither of them will be performing terribly well but with DDR4 3200 RAM I wonder if you'll be counting tokens per second or seconds per token. I mean, you do at least get a lot of memory channels at least, compared to consumer PCs. I am curious to hear what performance you get, I feel there is not enough information out there on what different setups manage to eek out.

> If you wanted to ask a 2002 person what they thought about future historical events you can just ask them directly.

The weights arent the truth tho, maybe a timecapsule-vhs but i wouldnt trust llm weights more than more hardcore deterministic media that might get preserved to infer facts from an era.

The companies doing the training are becoming the "winners" that are "rewriting history" as they train their models.

Models are quite safe when they're useless, actually.

In the times of GPT-3 I'd scoff at the idea of an LLM doing any hacking; today, I'm running several AIs on my code before publishing, and they are finding (and demonstrating!) RCEs on my localhost server.

For example, one found a missing check in a third party JWT library which allowed full account takeover, which I'd have never even looked at.

Hence I don't believe a single word coming out of these people's mouths. Their "beliefs" are just marketing.

> According to the book The Thinking Game...

According to me, this is nonsense.

Maybe I'm reading too much between the lines, but I suspect the reason is to rub his nose in the duplicity or naivety depending on how generous you're feeling. Publishing the model would be a confession that he was wrong.

AI policy is being shaped somewhat by the things Sam and Dario say. So even if you're not feeling vindictive, it's probably good to keep a track record of the previous things they have said as a Bayesian prior. People who don't know better listen to these people, and maybe they shouldn't.

GPT-3 is a different model than gpt-oss and is therefore not an answer to the question.

I cannot stand using gpt-oss, but I miss some of the creative spark of GPT-3 davinci dearly.

At release, GPT-OSS was arguably a few generations behind the open frontier.

Yes, sorry 5.3 flash.

NVFP4 ~20-30tps (MTP + vision, no dflash2).

I hear the argument here, but isn't it possible it has dramatically more knowledge and when you get outside the common cases many of us use it for, it'll have completely different capabilities?

I feel like most benchmarks cluster on a reasonably limited area of human knowledge

It seems like there is tradeoff between model size and the need for tool use, which - in my mind - is quite costly in terms of time and tokens. More detailed world knowledge requires an exponential increase in model size, but most knowledge can be acquired ad hoc using search or database queries. This will fail for questions where the model lacks the knowledge to ask the right questions, but maybe this could be solved by a handful small inquiry models with different knowledge encoded in their weights?

You can't compare models released 6+ months apart. GLM 5.2 was same architecture as 5.3 and not nearly as good. Takes time to build frontier intelligence and distill down to smaller sizes.

I think the Western labs are burning through funding and compute to maintain the lead at any cost, efficiency be damned.

4.7 and 4.8 are smaller than 4.5 and 4.6

(And 4.5 was smaller than 4)

> that would prove rather embarrassing for Anthropic

Not really, in that you just work with different constraints.

Anthropic and US labs in general has maybe 100s to 1000s of GPUs per person to experiment. Zai and Chinese labs in general have 1-10.

The priorities are different.

My second favorite model by now is GLM 5.3 flash which is very capable of day to day task. I use it as the main model and GLM 5.3 for task that is more complex

Pretty much anything is better than Opus 5.

z.ai coder plan, both in opencode and direct API access. I use it for my side projects like https://github.com/scosman/Biscotti (on-device meeting transcription and summaries).

many of them (kimi k3, glm-5.3) have license requirements to sell them with model-as-a-service.

I have seen you advertise your website a few times. I like the idea of not having to trust the router, so I took some time out of my day to critique your website: https://files.catbox.moe/v68cf7.png

My visit to your website went like this:

1. Visit models page

2. Try to find GLM-5.3-Flash (which is among the ~5 models that 90% of people currently care about)

3. Give up scrolling (which would have taken OVER 50 SCROLLS!!!) and use Ctrl + F

4. Try to find input/output/cached price

5. Scroll all the way up to find out which column is what

6. Notice that output price is cut off

7. Notice that the scroll bar is over 100 scrolls further down the page

8. Use Shift + Wheel to scroll horizontally (most visitors probably won't know this trick)

9. Notice that cached price is missing

10. Conclude that this is probably not a serious offering and bounce

There are probably more issues later on, but this is how far I got.

I would suggest you to:

- Deslopify all pages that a user may visit before conversion

- List important models first (see OpenRouter rankings)

- Move the most important information (model name/input/output/cached price) to the left

- Disaggregate the prices per provider (maybe subtables per model? not sure)

- Measure cache hit rate and compute effective price per provider (see OpenRouter)

(- Optional: Fix the broken link on your HN profile page. Currently, the only way to get from this comment to your website is a search engine.)

How am I supposed to navigate around there? For example the pricing page is empty or is that how it was supposed to look? On models and providers pages there are lists but no way to filter or get any kind of meaningful info. Or is this a WIP/POC?

I tried using deepseek v4 flash with OpenRouter. It switches between providers too eagerly which resets the cache. Then, each provider begins to rate limit me for providing so many uncached tokens, so it just keeps on switching providers. I'm paying for every token... why rate limit me? It was unusable compared to just using the official Deepseek provider which has a much better cache rate.

I just checked because I was a bit paranoid, but I have a 96.6% cache hit rate for GPT-5.6 Luna and 96.8% for Opus 5.

> Amodeis wife was tight with JE

Apparently, Cami Clark was tight with Eric Schmidt. Per unsealed documents, she seems to have pursued Epstein to invest in her "luxury porn" businesses, after a divorce & going bankrupt? Wild: https://www.wsj.com/tech/ai/claude-dario-amodei-wife-anthrop... / https://archive.vn/MJI7q

It's interesting that OpenCode Go is treating it as 2x more expensive than DeepSeek Flash, even factoring in the 50% discount

z.ai is using all Chinese hardware for flash: https://thenewstack.io/glm-5-3-flash-chinese-chips/

There are other providers with much faster inference, like BaseTen at >100t/s: https://openrouter.ai/z-ai/glm-5.3-flash#performance

I think we don't have a good draft model for better speculative decoding yet (e.g. DFlash 2). Once we do, it will be faster.

GLM 5.3 Flash is equivalent to 5.2, which is still awesome considering it's 10x-17x cheaper

Yeah I used Ox alpha earlier this week when it was free and deepseek flash when it was free on opencode. Both were great. Z.ai’s subscription doesn’t look very good versus the others.

I burn through my current Google AI pro sub for the week in about 2 and a half days so wanted something extra to add to it, but don’t want to buy any expensive ultra plan. Flash models have got me about 98% of what I need, but quotas are still a bit low.

> but I haven't used it because their privacy policy was pretty buns last time I checked.

What did you find objectionable? I looked at it when I subscribed almost a year ago and I was fine with it (e.g. they don't train on your API inputs).