> “The source described leading AI labs or their allies approaching the administration every 3-5 months with an idea to ban open-source models.”

That's going to hit first amendment grounds pretty quick, the same way that software in general did.

The modern version of the decss flag will be a character that says "I think good weights are {...weights go here...}"

They could, however, ban any payment to a chinese entity, or any entity owned by a chinese entity for inference/ai services/etc

Aren’t the models just software ? You could ban them in say high assurance environments like to be FedRAMP certified you wouldn’t be allowed to use them.

There is no great firewall so banning their import via the Internet is impossible

It's simple, the US government will put any Chinese open model companies on the entity list which blocks any company which does business with the US from also doing business with the Chinese companies. This creates a chilling effect where even if it may be harder to tell, no US company will be able to provide or use any overt Chinese open model and won't even risk trying to go around as the punishments for trying to evade the ban are severe.

It’s not really feasible, in my opinion.

US Gov could make US companies comply, like have Huggingface take down models out of compliance.

But most likely, a foreign-to-US Huggingface replacement would be made and everyone would go there instead. Lose-lose for US.

> So, any solution to this “problem” must include ALL open-weight models.

What about the EU? Would they follow Uncle Sam's order to ban all open-weight models? Lately the EU hasn't been that cozy with american companies: there are EU companies and institutions moving to EU clouds, the EU just fine Google a cool billion, several are switching away from Windows to Linux, etc.

Or is it just the US that'd ban open-weights models, while, say, the EU and Japan would still allow them?

> Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them.

It is not possible to 100% ban open weight models getting released in the same way you cannot stop leaks.

Just ask Meta with the original Llama leak.

It would basically make America behind as every other country would use open, cheaper models for all tasks but the ones requiring frontier models.

And that list of tasks grows smaller every day

> Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them. I think it’s impossible simply because technically there is no such thing as a “Chinese model”. There is no way to tell apart an “American” model from a “Chinese” one by looking at their weights. Weights are just numbers and you can’t assign country of origin to numbers. One can find very easy workarounds to any naive attempt to ban them by origin.

Historically just asking it about tianment square or getting some random answers turn into chinese (as latest interation of online deepseek likes to do recently) is enough

> Now there are solutions for that latter problem. But they are all ugly and restrictive. Making a DRM-like license protection system mandatory can be a solution.

I am very worried that's where consumer hardware will go to. All so AI companies can license local use of their stuff, and once that happens, less of an incentive to even have model be open.

Possibly even have DRM that counts number of computation done per model in pay per use model

You need to ask what happened in Tienanmen square

> It's not very clear why using GPT-4 in early 2023 was so expensive and then six months later 20 bucks could get you a fair amount of GPT-4 inference.

The price is what the market is willing to bear for the available compute capacity and competitive landscape. You can only discover that price after trying different price points and seeing what happens.

Everyone is trying different pricing schemes and discounts as they test the market. The demand is fluctuating at the same time.

It’s probably very confusing if you’re primarily familiar with stable and mature markets. Price fluctuations are a common feature of new and evolving markets.

> if you really want Kimi K2 instead of K3 you can still use it.

I think this is a very important aspect, especially after the huge GPT-4o backlash when GTP-5 came out. Each model has certain quirks, and areas where the previous model might be better for some use cases than the latest and greatest, and the labs so far seem to have no desire to offer some kind of "LTS" release.

> It's not very clear why using GPT-4 in early 2023 was so expensive and then six months later 20 bucks could get you a fair amount of GPT-4 inference.

FlashAttention was a hell of a drug.

Why do prescription medications cost so much, and generics so little (comparatively)?

Artificial inflation to recoup R&D.

What is strange about GPT-4 being expensive in 2023? Supply and demand. Which other model choices did we have? Prices are related to supply and demand. We see it play out with the introduction of capable open weight models or even other closed cloud models.

Did they actually ever cut the price on gpt 4? The oldest versions of it in the api still seem stupidly expensive? There were definitely price cuts as they introduced the turbo models and stuff, and new versions of each model might have gotten pricey cuts, but just because they're both called "gpt-4 something" doesn't mean they're the same under the hood or that they didn't change a bunch of stuff under the hood to make it cheaper to serve

Its very clear: nobody wanted to pay for usage at that price point

Quantization also started picking up around then, as well as distillation into smaller models

It's only strange if you do the silly thing of presuming a "fair market" in which e.g. it's generally easy to get reliable information about how all of the things work.

There's just obvious and enormous incentive for the OpenAI's of the world, along with all of the other players, to confuse, misrepresent or just straight up lie a whole bunch about everything given how new and unknown the tech is.

Yeah I've been thinking about this and the analogy I came up with is that tokens are basically equivalent to an in-game currency in -free to play" mobile games. You're trading actual money for some notional "curency" or coins that can only be used for one thing but, unlike mobile game coins, you don't know how many coins something costs before you use them. It's kinda weird.

Market discovery

Why is this so difficult to understand?

1. the field was nascent and new efficiencies were discovered

2. supply and demand

3. its in the company's incentive to make their models more efficient to increase overall usage so that while the margin remains the same, the total revenue + profit increases

I genuinely don't know what puzzles everyone?

Because as per usual it's silicon valley misunderstanding economics. AI is HPC. And how the HPC market worked before:

If you're the best performing "computing cluster" (ie. whatever you call the entity that can complete a massive calculation), you get a blank check from Congress.

Why? Because you need those calculations to "pump" nuclear weapons. They are needed to calculate both the geometry to make fusion bombs possible at all and to calculate the effect of a given geometry. They are the reason US/Russia/China have the biggest and strongest weapons known to humanity. And of course, they were replicated worldwide for this reason. I mean not that anyone will admit this but we don't have the best possible solution, and we don't know either the upper or lower limits for fusion devices (plus the lower limit would be very useful for energy generation, which for the US would effectively mean almost literally unlimited large marine ships that never need refueling. And yes, the solution to that problem is almost literally a 3d shape. Not just that, but mostly)

Now it appears it does not work the same when you democratize computation. Humans want a particular amount of computation and are willing to pay a given price for that. But the accountants still saw the blank check from before and ... do what accountants do. Economics don't change because you make things bigger and accessible, do they? Oh ... wait a second ...

As someone put it recently though, we now have data. 2.3% of humans in the US are willing to pay $20 per month for the support of a model like GPT-5.5/Claude code. If that's true (and after years of having this model, why wouldn't it be?) ... it means AI startups are doomed (because it's not even 10% of what they need it to be to make economic sense).

It's basically supply and demand?

> There is a larger 120B that you can't realistically run on consumer hardware at reasonable tok/s too.

gpt-oss-120b runs at 30+ tps on Strix Halo and +75 tps on a MacBook Pro M5 Max 128GB.

> I wish OpenAI updated these models more frequently though.

I think the spiritual successor is the Nemotron 3 series, although they also are getting a bit long in the tooth: https://research.nvidia.com/labs/nemotron/Nemotron-3/

The Gemma 4 models are a bit more up-to-date: https://huggingface.co/collections/google/gemma-4

Or Qwen3.6: https://huggingface.co/collections/Qwen/qwen36

"We have been having extensive discussions around open source strategy. [..] one thing we'd like to do soon is to create a language model with the approximate capability of GPT-3 that can run locally [..]. In general, we think this helps discourage others from releasing similarly-powerful models, and makes it harder for new efforts to get funded." - Sam Altman emails OpenAI board, 2.5 years after GPT-3

Chinese release open models to drive the state of the art, OpenAI and crooked Sam do it to keep you down.

The OSS models are good, but weren't very competitive at release with alternatives like Qwen3 Coder A3B. If you were a startup, why would you build on OpenAI's first (and last, for quite some time) model versus Qwen3, which was updated with multiple open-weight point releases?

The article certainly has a point, the current release cadence is untenable for startups that want frontier-grade models from American labs. Unless America wants to cede that market entirely, OpenAI et. al. need to kick it into gear, fast.

Model-on-Chip is coming. GPU are for general computing but have a huge bottle neck for doing model inference.

Even not being able to significantly update a model that is burned on a chip the performance gains are immense. You also don't need the latest chip fabs to make them drastically reducing the cost.

>> do most things and it then is game over.

For the Hyperscalers...and Oracle...cant wait for the day...

Pre bubble prices (~= “we stop building data centers with subsidized credit / circular loans / hidden debt”), a 128GB halo strix ran for $1400, and 200-ish watts. Four of those in a cluster will run a 1T parameter frontier model:

https://www.amd.com/en/developer/resources/technical-article...

At 7 months of claude code subscription per node, the cluster pays for itself in 28 months. On a 5 year (60 month) depreciation schedule, you can buy two of those clusters for basically break even, so you get two concurrent request streams (each of which can batch, etc).

The next generation hardware has already been announced, and should ship roughly two Moore’s law doublings later. It’s likely its steady state price is <= $1400 USD (2024), and it is faster.

So, once the bubble pops (because the financial machinations eventually will come to an abrupt halt), and the labs stop buying hardware for data centers, local inference will be extremely practical and cheaper than a subscription.

My main question is, when that happens, will UNIX Surplus be selling inference servers for pennies on the dollar (like after the dotcom crash), or are the power requirements too exotic for home use?

> it really isn’t economical to run this stuff yourself

Quantised models running overnight go most of the way for non-coding tasks.

> Sadly until china scales production of hardware it really isn’t economical to run this stuff yourself.

I'm running this stuff at home on my desktop and using it through an app on my phone. 60-140TPS depending on model / use case.

It's more than fast enough to even maintain voice conversation.

> Honestly imo this is just proof apple will win in the end. Eventually a phone will be able to run a model good enough to do most things and it then is game over.

I don't see how these are related.

The accuracy and capabilities of your model are directly related to its size. You need a lot of memory for that.

It will be decades before we get enough useful memory in a phone form factor at a price point people can afford it before something like a frontier model now is useful on the phone.

Now, you can run some models on your phone today.

Either way, Apple is using Google today. That could change, but Google isn't exactly getting out of the TPU business and they've been doing it a long time.

Also, some of you live in a very weird Apple bubble. Apple is not so relevant outside the US.

I don't know what you mean by "economical", but it has been "economical" to run this stuff yourself for the last 3 years.

1. You must be willing to be resourceful. 2. Be willing to learn, do the hard things. 3. Accept the tradeoffs.

I don’t know if others would find this useful, but previous did have custom harnesses etc.. but tools have improved so much that I drastically simplified.

That said, even the foundational models fail at the hard parts of my code so I use it opportunistically.

I have reduced down to just using zed, will three locally hosted models.

Qwen 3.6 27b on 1x3090 llama.cpp with 128k context ~50tps

Qwen 3.6 35B-A3B on 1x titan v + 2x1080ti llama.cpp with full context ~30tps

GPT-OSS 120b on pure cpu (slow)

I just use zeds parallel agents, task switching, stopping and fixing the code when a model gets stuck.

This still lets me stay engaged, and to modify code to be maintainable etc…

It gets me 80% there and I use to keep a subscription but often times just using googles AI mode is just as good.

That said I have 30 years of experience and insist on knowing how my code works, so this gets me 80% of the short term benefits while not depending on a 3rd party to keep my code moving forward.

Your mileage will vary and 2*5060ti 16gb cards would get around 100/tps with Qwen 3.6 35B-A3B on cards that are widely available.

To be honest the more modern cloud models are using draft tokens etc… that while they are superior for common coding tasks are degrading with more domain specific tasks.

That is just the cost of the draft model being ~10-20% of the foundation models size, and even the biggest Blackwell GPU is limited to ~250/tps so MoE or draft models are required for scaling performance at the foundational level IMHO.

The hard part is my use case are the OOD or small examples in corpus level, the above hurts there.

A Lamborghini may be nice, but I personally need a minivan more.

I'm using qwen 3.6 35B unsloth 4 bit with my 5950x (128 gb memory) and a 3060 12 gb gpu with a self made harness.

At 10k context I get about 40 tps generation and 500 tps prefill. At 100k context I get about 25 tps generation and 400 tps prefill.

It works, but I often use gpt or claude to make a detailed enumerated plan of what I want to do first, then have qwen follow it.

I'm not sure if it is economical or not, but I have solar on the roof so the power use is not really an issue and I already have the hardware.

The biggest benefit for me is that it's all done locally, and I know the harness is not uploading anything or sending telemetry to someone else.

I'm using Kimi K3 + OpenCode. I pay their API pricing, costs about $5 / hour (and chews through ~10 million tokens / hour) during continuous use when I have one or two sessions running and doing their thing.

Can't comment on how it compares to plans (I really don't like the limitations and general shenanigans I see around plans, so I've never tried them).

It is notably slower than Fable / Opus / Gemini, but also vastly cheaper than their API pricing.

I am using GLM-5.2 via Ollama Cloud in the $20/month plan. With the same plan I can get many different API keys that I use to run my OpenWebUI server, my opencode and pi dev sessions. I am usually running 2 to 4 sessions concurrently, and I never hit quota limits. At work I get Claude, and I was getting reports that I was spending $75 per hour of work on Opus.

GLM 5.2 awq4 via Opencode, it was better than corporate's fave Sonnet 4.8 and a but worse than Opus 4.8. overall quite capable of a lot of what the dev team needed. Not as good as Sonnet 5 but also never runs out of tokens.

The flip side is that it takes four H200s to run, and that will only let you cache context for maybe three users.

Fingers crossed our Blackwells show up and Kimi 3 really releases weights, because at some point devs are spending a significant portion of their salary on tokens and it's somehow cheaper to buy these ridiculous DGX servers and rack and run them.

I use opencode or pi harness with the deepseek api for all my at home coding usages.

Deepseek is at least on par with Sonnet (ghcopilot at work)- I don’t use opus, too spendy and I don’t need that level of ability.

The cost is for me was $5/6 months of use. Not a big user I guess! It’s good though, fast enough and incredibly inexpensive.

Been testing Qwen 3.6 28B on a 5090, and it’s also quite good for “free”.

I mostly do small self serving embedded projects based on esp32, so not very complex.

> I’m asking because i keep hearing that open weight models are cheap and efficient - is that really the case in practice?

I think that's the case for people who compare it to proprietary models paid via API—which I think is irrelevant given the majority of people daily driving AI coding are doing on a subscription plan.

The better analysis then is not about AI coding, since there's no subscription plan for Kimi K3.

Instead, compare the cost of running some agentic _task_ that isn't coding which can only be done via API. Think of all the startups wrapping around ChatGPT and Claude to provide some additional set of tools, context, data and hoping to turn it into a profitable service.

To those companies, which are many, open models are the difference between the math working out today vs. praygeing they can scale fast enough to find profitability.

I'm using Qwen 3.6 27B on a macbook with Pi. It's alright, it runs fairly quick (40 tps for quality version, 80 for the fast). It doesn't tend to one shot things but I'm generally comfortable fixing the bugs myself afterwards or prodding it a little bit. I find the harness matters a lot. "Continue" (the vscode extension) worked horribly, OpenCode was ok but its vibecoded internals make me view it as a security nightmare so I'm hesitant to run it, so I've settled on Pi for now.

Claude and ChatGPT are good deals right now, with the subsidies. They produce things faster and better. I guess not cheaper, in that inferrence on my macbook is basically free, although the macbook itself definitely wasn't. My focus on running local is around three principles:

1. I don't want to support surveilance capitalism by giving these companies my data anymore, when I can avoid it. And LLM companies want to vacuum up every detail of your life.

2. I don't find these companies to be remotely trustworthy, and I find them hostile to a healthy society, so I want to avoid giving them money going forward

3. I think they're going to start charging a lot more

qwen3.6 27B q5, llama.cpp, RTX 3090, pi, cost: electricity bill

While I am grateful for open weights models I never found much use of them in the past, barring those I could run myself. This changed with deepseek 4 - it is staggeringly cheap, even if the performance definitely isn't near sota and it's not particularly fast either.

When I expect to need a lot of tokens and the task isn't too difficult I use sota to plan and create a thorough set of instructions and let deepseek chip away at it. With thorough instructions the quality tends to be satisfactory, and you pay something silly like $15 for 600m tokens.

GLM 5.2 seems like a decent price/perf and Kimi 3 has some real nice performance for an open weights model, but gpt 5.6 is unexpectedly affordable (especially if you don't automatically use Sol at max) so I don't think either is worth it atm. The exception is when you're working on something that US models get cold feet about, which seems like a constantly growing list. For me Fable is already too much of a headache in this regard, but chatgpt is still okay-ish. Hopefully it'll last. If not, there's Kimi.

tldr SOTA for most things because gpt 5.6 is token efficient. If I expect to burn a lot of tokens I use deepseek 4.

I use DeepSeek 4 with the VSCode CoPilot plugin. I pay about $10 month on the pay-as-you-go plan.

It's not as good as the frontier models I use at work, but it's plenty capable for the types of tasks I am using it for.

Kilo code with direct API payment to DeepSeek. It costs pennies per day event at max.

No, it’s consistent with what China is doing in other markets, which is dumping product to drive others out of business.

I had a shower thought on how to counteract this, specifically related to the AI dumping. If China is losing substantial money on every token, why wouldn’t an adversary try to maliciously increase consumption? This strategy is not really viable against physical goods dumping because demand is finite and there are large environmental costs. Software demand is infinite and the environmental costs are quite low compared to the financial cost to make it, even with ultra cheap Chinese tokens

What tools do we have to countermeasure the state sponsored bias in the Chinese models? Doesn’t seem like a smart plan if individuals can just compensate for the bias.

This is such an obvious conclusion. To take it a bit further...

Scale matters for these things. If we divide the available chips among 5 competing companies we end up with models that are trained on 1/5 of the resources that they otherwise could've been.

Let the companies take turns training on shared hardware, force them to publish results in the open, and then reward them based on how well the resulting model performs at democratically chosen benchmarks. Meritocracy not monopoly.

Make it about how well you wield the silicon, not how much silicon you wield, and make it a positive-sum game. If the people's data is going in, then the people should benefit from what comes out whether or not they have a subscription.

If capitalism as we know it can't complete, so much the worse for capitalism as we know it.

> - Unlike OSS where a single volunteer can keep a project going, training costs run into the $billions.

Just some thought: Wouldn't it make sense to build some kind of volunteer computing project to train the next-generation LLM by volunteers, similar to the BOINC [1] projects or Folding@home [2]?

N.B.: BOINC was particularly famous for SETI@home (completed), Einstein@Home, Rosetta@home and PrimeGrid.

I still remember the time when Einstein@Home was in its heyday, and many people who loved putting together fast PCs contributed sometimes even for the reason of showing off in the statistics [3].

---

[1] https://en.wikipedia.org/wiki/Berkeley_Open_Infrastructure_f...

[2] https://en.wikipedia.org/wiki/Folding@home

[3] https://einsteinathome.org/de/community/stats

China has no end of money to support these companies. The reason this equilibrium is unstable is that the autonomous agentic coding aspect of the models has been so successfully improved that it will soon be a threat to China state security.

I still don't know what it is tbh. Something for docker?

Recently my company bought another company, and we kept zero of the original engineers, we just had to run the ghost ship.

We walked in, and it was fine. Because it was all kubernetes and laid out like every other app for the most part.

The kube hate is just sad at this point. You need to know like 15 concepts that are all applied in the same way. It mostly just works.

It is natural to not see something that isn't there. The article isn' comapraing the two in application or functio

And that censorship will fail too, like sanctions, tariffs, and keeping down open source software or exchanging ideas across borders, ultimately it will fail, but that won’t stop governments across the world from trying.

Essentially doing this, https://health.clevelandclinic.org/catastrophizing while making other’s mental health worse..

Ridiculous. If investors wish to set their money on fire investing in over-valued companies, they are welcome to do so. Protectionism to preserve ROI is a dumb policy. Just look at the US car industry. We're building dinosaurs. On this trajectory we'll have 0% market share abroad in 10 years. Chinese EVs will probably get market share at a 100% tariff because US automakers fell so far behind.

Same situation in protectionism in AI. The rest of the world will simply lap us.

Another shill account I gues

How are weights stolen from the frontier model developers? What is that actual mechanism?

They don’t have a monopoly and if they do, it would only be because the EU and the United States let them or I should say they let all their decisions be made by private companies who scraped the public Internet and now want to fence it in for their up-and-coming IPO.

Hilariously bad take. Open weight models can be retrained of fine-tuned, that's the entire point. The idea that the "Chinese government controls your intelligence" is laughable in the case of open weight models. Once the weights are released you can do whatever you want with them. The idea that there's economic capture by the Chinese for products they're literally giving away is stupid to the point of inanity.

I can only assume this account is pure shilling for the closed-source AI labs.

Open Source is free as in speech

Open Weights is free as in beer

"Open source" open models can also survive if they are seen as a necessary cost of doing business. Same logic as tech companies working together on Linux. "We need an operating system. But an operating system is expensive to make on our own and does not really provide an edge. So let's just work on and use the open source one."