Companies making closed-weight models typically don't export them at all, but sell access to them as a service. (Which doesn't count as an export of the model.) So it's primarily companies releasing open weights (if they pass the thresholds for the restricted category), or that host their models in datacenters outside China that will need to apply for an export license.

No, what they've been doing so far is changing nothing.

What they are saying is that they won't be changing anything.

That could of course change overnight.

There are multiple aspects to this: China is cottoning on to the meaning of soft power, and besides that being a major supplier of AI related models, services and hardware further cements their place in the world economy. They may not be able to compete directly with the USA in this domain but they are able to undercut them.

Of course there are risks as well. Models produced and released into the wild for which you have not seen the training and conditioning data could have payloads that you'd rather not have. But the same thing goes for service providers in other countries as well. You can't really trust hosted inference, not to tell you the truth in so far as is was present in the training data (censorship, not inference errors) and not to leak your information.

“Mind people's actions over words.“ -Zed

“Right now, your government is doing things you think only other governments do.” -Van Halen, Right Now

Care to substantiate that? Not that I disagree, just curious.

My (not thorougly informed) take is that they are are taking measures to have the capability of restriction but are also adding provisions so that this capability does not add friction to the current practice of enabling export of open weight models.

So to me it's like they are keeping their options open rather than leaning on one versus the other.

I think you're contradicting yourself here.

What they will do remains to be seen. The document at hand is what they're saying.

I.e. less important if we follow your own assessment.

I would find it surprising if the chinese government wanted to allow Anthropic/OpenAI/Google levels of influence on the public opinion and regulatory environment without guardrails. OSS models make AI a replaceable commodity and reduce the potential for market/opinion capture by the frontier labs.

Jetson Nano(with Nintendo Switch SoC) used to require a signed form with restrictions like that for US export control compliance even in a democratic country. It's the same device as the one that was always there, except you were only familiar with its operating end being around yourself.

But China is, famously, much more open when it comes to AI. For this comparison to make sense, China should be the hypocritical one, declaring openness when not really being open.

There’s no need to be so sarcastic about OpenAI.

You trust other countries to behave like this out of the goodness of their hearts or something?

If China’s selfish behavior = more open LLM tools and greater ability to run them at home, I’ll gladly take it over the US frontier labs’ druthers, which is a restricted/closed shop we rent from.

This isn’t some BDSM gathering; there’s no need to keep bringing up "safe words."

I'd argue that they're still open-source because inference code is fully public. In operation, a user can crack open the model and examine its internal state, and we also see that many people produce various modifications or fine-tunings of public models for their own goals.

In my mind, this meets the conceptual and technical bars of an 'open source' designation. We can also imagine the opposite, where inference itself has to pass via an inscrutable binary blob that keeps the model's technology stack fully proprietary.

You're correct that we're missing the training data, the training recipe/curriculum, and any secret sauce that went into optimization of same, but I don't think that's an essential feature of an open source designation.

Instead, I think that the model weights are a kind of rendered artifact, and it should be evaluated on that basis. We don't treat permissively licensed images as something lesser, even if the photographer or illustrator supplies only the final image rather than the full camera raw file and/or Photoshop layer stack.

The difference, of course, is that the rendered artifact (weight blob) in a language model is so much more important than a typical photograph.

I mean all of open source software (as we traditionally defined it) relies on an ISA, which is actually executed by microcode that isn’t available to the public?

I don’t see how this is any different. It’s like saying Intel has to publish their transistor physical library and microcode to be “truly” open source. Like I don’t have the internal notes about the most recent microcode changes (which is akin to “training process” in the analogy).

There are risks in both cases, but they are very different.

If you're using a closed model, the model provider gets all your data, and can if they want to send back actively hostile tool calls and rely on your harness to catch them without any effort on their side to train that behaviour into their models. The only things you can do about that is to place careful restrictions

If you're using an open model against your own system or a third party provider, the origin of that model could have trained in "triggers" that'd cause the model to produce actively hostile tool calls as well, but they'd have no direct control and would need to train in that behaviour "blind" to how you'd actually use the model.

It doesn't mean you can be complacent about trusting the model, but it does change the likelyhood they'd be able to do any damage, not least because you'd be able to run evals against the model without any risk that the model or provider API could just randomly change on you after a while.

So it takes far more trust to run a closed model than to run an open model even on a hosted third party provider, and certainly far more than to run an open model on your own hardware.

You can run Chinese models locally without Internet, while with US models you need to have Internet connection. Guess where it is easier to hide a backdoor.

I think it is more about avoiding being boxed in by the rest of the world. The US would of course ban the use of Chinese hosted models and have already taken steps to ban the use of open Chinese models. There's nothing that could have been done about that. But the rest of the world would likely have adopted American hosted models eventually, the same way they adopted American mobile phones OSes, computer OSes, Office, etc. If the choice was between American hosted AI and Chinese hosted AI they'd choose American if they couldn't develop their own.

But if there's an open model available then they might choose that instead of either of them. Even if the Chinese models aren't used they will bring down profit margin just by existing. They wouldn't have the control the hosted American models give but that wasn't even on the table. But denying America control is a worthwhile strategic goal in and of itself. And they don't even need to expend extra resources to do it, they needed to develop their own models for security reasons anyway.

Probably the same reason that Meta gave: it's less important to control it, and more important to prevent someone else from controlling it entirely.

Soft Power, both in the obvious and the less obvious sense.

A model is never neutral. A model is always also a way of disseminating an ideology, and it doesn't need to be lying for that to work.

Consider the question "what caused the French revolution?"

There's many ways to answer this while being technically correct. What gets prioritized, what gets omitted, that's where the ideology lives.

Its because the triangle of good, cheap, and open favored them. Just being good isn't enough unless you're number 1 but likewise neither is just being open (ish). It was important to release the wieghts and the research because otherwise good wouldn't be good enough to outweigh the risks.

> they didn't take off because they are not as good as the rest.

This may be true now but probably won't be true in the future. My guess is that part of the reason for this is that the major American technology companies are acquiring talent to build out models and related capabilities. China doesn't (or at least didn't) have that same dynamic.

Does anyone know about how you could go about training this type of thing into the model? Surely it is quite tricky to get it to do such things without it doing them more than this one incident? It should have happened lots if it really was trying to obtain secret info and build backdoors?

You know people are saying that it’s not open source without regard to intent. The intent is 100 percent open source.

The reason why it’s not done is because it takes a huge amount of resources to not only record the training process but to release it in a readable format.

What you’re asking for is almost like surveillance documentation for a model that’s likely put together in a way that’s not easily peaceable into clean step by step procedures.

It's actually extremely smart.

Latest and greatest China is able to make is reserved for the use of businesses and entities in the Chinese market.

Everyone else gets access to models of more limited capabilities.

It's basically the US strategy. Obvious difference being only that the Chinese allow on prem hosting of the lesser models. So plus one to them I guess.

If we want to move China or the US on from the whole, "super intelligence models for me, more limited intelligence models for everyone else" strategy, then "everyone else" will have to come up with some models competitive with the latest and greatest from the US and China. Absent that, there's just no pressure to bring the US and China to the "share the wealth" table.

Unfortunately, "everyone else" coming up with a competitive model is not gonna happen anytime soon.

Yea, many here keep trying to treat LLMs like software, or at least conventional software. They are not. To me they are at least the modern equivalent to ballistics charts that governments used to keep secret. No one is going to hand them out to their enemy to use against them.

Hell, I expect before long hardware is going to have all kinds of lockdowns occurring.

In my mind, they’re putting huge downward pressure on the major US AI companies’ ability to become platforms.

Think about the history of AWS. Amazon made a very specific point of building, specifically, a “platform”. When they developed a major internal service, they would sell it. Basically anyone who wanted to use it could pay AWS (often an outrageous amount, but it was often worth it), and AWS would provide the service, and AWS was consistently neutral and trustworthy. They didn’t really tell you what you could or couldn’t do, they didn’t even ask what you wanted it for, and they never used your private data for their own purposes. AWS pulled this off quite well despite the fact that competing products and even open source products could and still can, with effort, do most of what AWS offered.

Compare to OpenAI and Anthropic. They want to sell you their front-ends (chatgpt.com, claude.ai, their apps and harnesses), but the big money is selling enterprise contracts with per-token pricing. The latter is the platform.

But their platforms suck in comparison to their open competition. You are paying for inference on a series of models, but:

- You need to negotiate with them if you don’t want them to retain your data.

- They might not even be willing to not retain your data.

- Part of the model output gets encrypted before you see it.

- You are not allowed to give the model arbitrary input.

- They run classifiers on the token stream to try to restrict what you may or may not do.

- You cannot see the model’s internal states.

- You are extremely restricted in your ability to modify the model and you may not be allowed to modify it at all.

- Oh, and maybe the price is above market rates.

The last point is not really a big deal: AWS has demonstrated quite effectively that customers will pay many, many billions of dollars above market rates for a high quality platform.

But I think the rest are a big deal, and the open-weight models allow anyone to use a provider other than Anthropic and OpenAI and avoid all the other problems in the list. This means that, if I’m a startup building a product that uses AI, I get to choose between OpenAI/Anthropic-like terms and a far better alternative. And this makes it very hard for OpenAI or Anthropic to sell me on their platform even at equivalent pricing.

Then again I should probably have checked the front page before saying this.