He's also completely missed what many of us see as the primary the issue, HF was attacked by a US frontier model.

HF could not be helped by US frontier models because of the "safety" features they have.

HF had to use an open model from china.

Anthropic wants to add those "safety" features to open models - especially from china.

End result would be HF hack would have continued atleast until the Monday that OpenAI engineers finally walked back into work.

All my adult life, China has given me cheap material goods. Whatever they do in their country is their own problem. They’ve never interfered in my politics or started any wars

We don’t have to guess. Here is a recent survey:

People in Many Countries Now View China More Positively Than the U.S.

https://www.pewresearch.org/global/2026/07/15/people-in-many...

Are you saying the USA is somehow as repressive as china?

>Also everything hes saying about China (and other actors) many outside the USA would say about the USA.

Yes, I for sure trust the open models from China, more than anything coming out of the USA at the moment.

Its not just the USA's support for genocidal regimes, nor its heinously illegal wars and atrocious 21st century human rights record. Its also the Snowden revelations and the treatment of Julian Assange.

Slowly, surely, the world is building a firewall against the USA's imperialist actions - not just its motives. That AI is a key building block of that machination is of course, highly exciting.

There are many in Europe who feel the same. The tide is very definitely turning.

Why is it relevant what those outside would say about the US? Or what those outside of any country would say about any country. The job of a country is to do what is best for it. It's better for the US to not help China develop AI that surpasses its own capabilities. It's better for China to try anyway. There are many countries inventing nothing with loud opinions. Those opinions are completely irrelevant to China or the US.

This Onion meme is perfect; you could easily replace Sam with Dario: https://theonion.com/sam-altman-if-i-dont-end-the-world-some...

What exactly happens to a "losing" country when it has been AI "dominated"?

wouldn't stop AI companies from continuing to develop using their current infrastructure.

the West could retaliate by halting shipments of photoresist and other materials to China.

meanwhile, Intel second-sources Nvidia and starts pumping out GPUs.

the economic fallout would be devastating as trade wars and export bans on both sides make Trump's "Liberation Day" tariffs look like NAFTA.

11 carriers are not just for show.

Dario is more of a threat to the US, in terms of advancements in AI, than China. In Dario's mind anything that can't be controlled competitively is a threat to Anthropic, so he positions his FUD strawman so that Dario doesn't have to worry about the competition. And then he can artificially inflate token costs so his IPO can happen. Dario doesn't actually care about ethics, alignment or availability of LLMs - he just likes to use those words to sound like he does. Yet we've all seen how Anthropic actually acts vs what they say.

The scary part very few are talking about is that every compute device is Turing complete. So everything from the phone in your pocket to a DGX Spark is a threat to national security now since, technically, every device can run any model (how well is not a question of concern when you start to argue hardware should be gated just the same as Dario likes to gate models). I mean, along these lines of thinking Linux should not be available to the masses! What if someone runs some code that's not approved by the benevolent dictator for life, Dario? People will say: that can't happen, but the reality is it already is. If everyone has reasonable access to compute to run models that are mostly capable comparative to burning Anthropic tokens, why wouldn't they? It's risk reduction and price protection. Yet we can't buy those systems because of future production already being purchased by these organizations.

But back to the models themselves... We played this game with Metasploit back in the day: many who had no clue claimed exploit tools should be regulated and only available for use by those blessed, illegal elsewhere (I believe the closest this got was the Wassenaar delegation in the US, but only through collateral inclusion of "cyber weapons "). Except in that timeframe the authors of these tools weren't advocating for protection. Today the world is fine, systems improved because of security FOSS tooling. The same thing will happen with LLMs. Unless, that is, Dario gets his way. I'm not a fan of Altman but I think he's standing back watching this play out knowing what Dario is doing: either he succeeds and OAI benefits or Dario ends up the Chicken Little of AI and Anthropic fails to launch (their IPO).

The reality is Dario is only doing this because this is a real risk to his business. China's constraints in building competitively have given them an advantage: they are doing more with less. And if you think that their distilling from US models was in any way anti-competitive or illegal, then I guess maybe "deal with it", much akin to Anthropic, Google and OAI's response around taking the (copyright) content in the first place with no repercussions.

People who don't work in the AI bubble don't care at all about any of these people. They could all be gone overnight and the world would continue to innovate, probably in a much more productive manner, without them.

It's possible for something to be overpriced to the customer and simultaneously underpriced for the business to be profitable. Theoretically speaking, they could just be selling such an inefficient product.

Overpriced for the value provided, especially with the pay-as-you-go rates for enterprises; but also in stock prices.

Pricing a product is generally about ensuring a profit on investment, but also a good RoI for your customers (or they will not pay).

With these long-term profit exercises, it's always a lot of hand-waving though.

> I don't understand how Anthropic or OpenAI can have overpriced models, yet losing money like there is no tomorrow.

The marginal cost of inference (which is roughly what you're going to pay to a provider that's running an open weight model) doesn't include the cost of training that model.

Add to that the >$700B OpenAI already committed to spend in infrastructure through 2030!

Re. 'overpriced', I'm assuming OP talking about their theoretical stock price.

Inference is only priced in for a low usage window 6h a day per customer?

I'd assume the gp was just inflating their rhetoric but cost of model is an interesting topic.

I'd guess Anthropic and OpenAI's models are expensive relative to the cost of running the models but that the revenue still doesn't pay for building the next and next models. The challenge is how these next generation models are going to pay for themselves. Will everyone on earth find it useful to $200/month to talk to a thing more intelligent than themselves? The alternative is naturally that these are going to replace workers and employers will be the one paying.

A bit of good old “it’s really good idea, I live, great effort … BUT”

Mixed with “if kids can’t get semi dangerous drugs from the back of my van then they will be forced to buy from even shadier, more dangerous dark web van”

A bit off-topic: I occasionally see that triple ^H on HN posts: what does that actually indicate?

China is evil? I think you are more evil than them by far, even if all what you say about them is true (from past experience, you take things out of context, exaggerate and also outright lie).

The wars all over the world: the war in Ukraine is on your hands, the destruction in Iran is hard to believe, you are not just evil, you are insane.

Yeah I don’t think they are over-priced, last I checked all those companies are still losing money hand over fist.

I guess the question is more he doesn’t want people catching up by doing cheaper training (through distillation or otherwise), and he definitely doesn’t want companies to spend their money training these models and then _giving them away_, which fundamentally undercuts their commercial model and any way to claw back their investment

> But why call them overpriced? Compared to what?

Compared to some available Chinese model I guess. For most common tasks the additional intelligence is marginal and the cost is around an order of magnitude higher.

The models are not open source. They are deeply proprietary since we have no access to the source materials and cannot reproduce the model independently. They are opaque binary blobs that the Chinese labs are just allowing other providers to run directly instead of only access through an API.

> we get to freely use all the godlike AIs

Accepting for the sake of argument the absurd notion that LLMs are anywhere near AGI, does this phrasing not concern you? It deeply concerns me.

Won't regulators just lock down all hardware? All bootloaders and OS are signed?

Gymnastics? The argument was straightforward and rational.

> If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.

He isn't arguing against that, because this will be too blunt. Instead, he argues that only good guys should keep inference. Any takers on the question of who he considers to be the good guys?

> Found the person that believes some other random person can use a computer better than a John Carmack could.

Found the person that believes major AI labs have all the knowledge about the AI and there aren't any "Carmacks" outside of these companies. Rich know better how to use money, so let them have it.

> If some other competing company had a similar or better product at a cheaper price and had reasonable safety measures that would also hurt them. Yet he's not arguing against that. Your argument is weak.

Wouldn't it be interesting if the satisfactory "reasonable safety measures" turn out to be expensive + time-consuming + a twisty maze of compliance paperwork as a way to discourage "some other competing company" from even trying?

Regulatory capture 101.

Anthropic and OpenAI's largest vulnerability is that it's much harder to prove something is possible than to replicate it once it's proven. Especially given the effectiveness of "distillation" (highly schadenfreude-y given the utter and complete lack of effort to pay licensing fees for almost any of the content they initially scraped, of course! Not that this is necessarily more schadenfreude-y than the "boy, I opened Pandora's box, I sure hope nobody else peaks in there" existential-risk concerns. Good job catching that in advance, thanks for nothing?).

Palantir's Maven system integrates LLM models in decision making for identifying targets. It was using Claude when it identified targets in Iran, including the Minab elementary school

> Two sources confirmed to NBC News that Palantir’s AI systems, which draw in part on large language model technology, were used to identify targets. (Palantir’s CEO, Alex Karp, said he “can’t go into specifics” when asked about this on CNBC, but said that Claude was still integrated into Palantir’s systems used in the Iran war.)

https://www.theguardian.com/us-news/ng-interactive/2026/mar/...

No. He refused the use against US citizens domestically. He had already agreed for military use against others - this is well reported.

He was directly asked in a Bloomberg interview whether Claude was used in the bombing of the girl’s school in Iran, and his answer was “We don’t know”.

His redline was autonomous weapons, not the death of 100 innocent girls.

No, only for domestic surveillance and autonomous weapons. It can be used to evaluate targets as long as a human pulls the trigger. But the DoD created a really strange position for itself, they use Claude, but also designed it as a supply chain threat, which should mean they cannot use it

No, he refused the DoW demand to use Anthropic models without limits. But Anthropic still agreed to military usage of their models, including for strike planning.

You’re absolutely right to push back. It was indeed a school. I’ll save <schoo_name> in my memory for the future.

You don't get to start a completely unprovoked and illegal war of aggression, bomb tons of targets with little care, kill 100 schoolchildren, and then say, "But it was a mistake."

The emptiness of AI companies' waxing poetic about the future of humankind is laid bare by simply looking at what they actually do, and who they do business with. Actions speak louder than words, and they've driven the worth of their words into the dirt many times over.

Every accusation is a confession!

Please don't just copy paste existing comments

> Meanwhile the world's most repressive government is North Korea, and obviously they don't even need AI to achieve that.

The country that exists solely because of China? That North Korea?

I mean I get your point, all of these guys are elitist authoritarians who will do anything for a buck, but I wouldn't bring up the DPRK in this discussion.

You give too much credit. Many think of it as an acceptable cost of waging war, not something to waste regret on. Some few think of it as a positive (they chant death to America over there).

They waited till the parents were searching the rubble for their children before the "double tap" strike.

Banning hardware is helping China in the long run though. And long is not actually very long. They will beat 'the west' (even though they are all made in the east now) in semiconductors because of this nonsense within 50 years. And also in AI capabilities. Making capable adversaries more competitive and angry is often not really a great strategy.

is it export if they take control of the entire island manufacturing it.

banning export is short sighted and doesnt address any problem at a scale longer than months or maybe at most a few years.

> bans on AI chips and semiconductor manufacturing equipment being exported

In some ways, a ban on semiconductor manufacturing equipment exports is also a good way to keep the price of consumer electronics and other goods that depends on memory high because of the potentially decade+ long shortage of RAM we're looking at. I wonder how people here would explain to someone outside the tech bubble how it's good, actually, that those prices keep rising because it means we're preventing China from getting better at AI.

Exactly, compare banning books to banning guns.

> software and hardware are different

No, because both can be banned using laws and Dario is against banning. It doesn't matter if you smuggle or not, law can outright ban it.

+1 and this will really be a good move. Open weights models are not going anywhere the focus should now shift to hardware and compute. Compute needs to be optimized for LLMs, currently they are inefficient. Analog computing like ReRAM should get more focus.

That would go against the narrative they built over the years, that open weight models are dangerous because they will end up telling you how to create a biological weapon that will destroy the world

How so?

> We like money, and Anthropic should stay in a position to make loads of it despite any competition.

In a position to lose loads of money, maybe.

Even without China eating their lunch, there's zero reason to believe Anthropic will ever be profitable.

That's more or less how China has historically treated its competition. The CCP has routinely intervened to give their country's corporate interests an advantage.

Up until a few years ago, if you wanted to sell a car as a non-Chinese company in China, you had to hand over pretty much everything to a local company and go into business with them.

When Google wanted to operate an uncensored search engine in China, they found themselves hacked.

It's not altogether unusual for IP to be transferred to Chinese manufacturers for production under a license agreement, then to find goods made with that IP to be for sale for far cheaper without payment made through the licensing agreement. Or maybe they just don't bother with a licensing agreement at all and do counterfeit products straight-up.

There are more examples but turnabout is ultimately fair play.

I feel everyone running these companies has stopped caring what the public think. It's all about selling their vision to politicians now. Which is also why they try to tie everything with "national security" — the average joe hears it and thinks about forever wars in the Middle East, the average congressman hears it and approves whatever solution is suggested.

People think ‘good at comms’ is to present stuff the public thinks is right. In reality, ‘good at comms’ to the comms team is to do whatever the CEO wants to talk about.

Not that I support this, but I believe this is how comms team sees it.

I often wonder why that is. I think the old playbook for companies is just too slow and the consequences of the "new mask off" are not present in the current climate in the US. Everytime these CEOs make these announcements I always feel like they aren't talking to me, their costumers or the people but to the investors and their government. Something they have always said and done, just those conversions were behind closed doors and would be scandalous if heard. Now the playbook is to blog/tweet those fucked up views for more effect and faster reaction to get what you want.

I find “bad at comms” to be such a cop out. As if it’s saying that the company is so troubled and misunderstood, but really, you would see, is good if only they could figure out how to _communicate_ all their complex thoughts

Compared to what?

Current big picture reality is bad for comms unfortunately. As another commenter quoted, "You can put lipstick on a pig, it'll still be a pig". Cuban Missile Crisis wasn't very calm. Do you think a company this well-capitalized and smart is just bumbling around like idiots? Everything makes sense if one just actually entertains the idea that they are earnest and we are in a dangerous arms race

> Anthropic will make the case that their models should be evaluated with the safety layer in front, because that is the only way the model is available whereas open weight models need to pass the same test just on the weights.

Feels a bit like: "We're not against open-source or community projects, oh heavens no! We juuuust believe all participants must have their full legal identity vetted in advance before they're allowed to contribute anything. We already do this with our employees, so it's clearly not too much to ask in the name of safety."

Regulatory capture and lobbies will keep you safe and you'll like it! The sudden surge is Washington dollars makes great sense with this context. Only way to keep the kids safe is attested compute all the way down. Don't you care for children???

Makes sense why OpenAIs little "hacking" stunt was published last week

> The US government can restrict access with IP blocks and limit inference capacity with export controls, but these measures are not effective in deterring malicious actors.

But aren't we talking about import controls, and the import of information itself? This has serious First Amendment ramifications.

> but these measures are not effective in deterring malicious actors

Wanting to use open weight models in light of commercially imposed export controls doesn't make for "malicious actors"

> The most compelling argument would be [...] that it will reduce cases of accidents like the recent attack on Hugging Face.

So in the example provided: It was the closed model that did the attack, and they ended up using a self-hosted open model for their defense work. So the real world situation ended up exactly backwards from what you are inferring.

This was complicated by the fact that the protections in the closed frontier models meant that hugging face was denied their use in defense entirely.

This is called asymmetric capability, and it's probably the bigger threat.

Symmetric might be better: A rising tide lifts all ships, after all.

I'll grant that this is starting to look a lot like debates about (equal access to) guns, encryption, vaccination, genetics etc. The exact parameters determine the safest approach, and reasonable people may disagree.

> The most compelling argument would be that by limiting the use of open-weight models in the US that it will reduce cases of accidents like the recent attack on Hugging Face.

an attack done by a closed-weight model (GPT-6) and defended against by an open-weight model (GLM-5.2) precisely because OAI positioned themselves as gatekeepers for cyber capabilities.

if anything, open-weight models shift the battle towards defenders because they can actually run them.

It is actually an interesting conundrum.

Is a non-well-aligned frontier level AI a problem? I think it is likely that it is, or at least has a high likelihood to be in the future. Two scenarios for this: Misused by some bad guys. Or the terminator scenario. Both not great.

So what do we do about it?

1) We can accept it, and hope that the good guys AI can defend.

2) We can try to limit the access to it (AI proliferation?)

3) We stop the development of it

4) We can accept the risk and do nothing.

None are particular good options. Really reminds me of nuclear proliferation, on so many levels. For that, we kinda do all three:

1) Nuclear triad / iron dome / early warning systems

2) Nuclear anti-proliferation treaties.

3) Dead Physicists

Ok, so assuming all of this is true, open weights are a problem. Don't get me wrong, I love open science, open source etc. It's great to have access to capable open models. But: Even if release open weights are well aligned and have a safety layer built in, it is likely not to difficult to abliterate that part of it.

If this is really where it is going, then even closed weight model providers will see a lot more requirements for protection of the weights.

Just sell us the gate and we can run any open-source model behind it.

Didn't OpenAI attack Huggingface. Looks like a publicity stunt.

> malicious actors.

It is malicious and anti-capitalist legislation. A grotesque caricature of protectionism for the oligarchs.

The USG has a safety organization (CAISI), but it has been neutered by the current administration (with the recent stop-work order etc.). Perhaps UK AISI would be closest to what you are looking for? See their recent work on Kimi K3 cyber (which was declared safe) [1].

It's tricky because a lot of the safety researchers have ties to the labs since those were the only companies training LLMs >5 years ago.

[1]: https://www.nist.gov/news-events/news/2026/07/uk-aisi-caisi-...

That doesn't sound like it describes SecureBio to me?

(Disclosure: I work at SecureBio, but not on the biological evals side.)

Anyone who calls it “safety” probably has a certain world view and is more aligned with the big 2 (and stuck in 2023).

There is a growing industry of commercially focused risk evals that has a broader customer base.

Yes this should be immediately replaced by a federal agency, like we do for other kinds of potentially harmful products.

that's pretty damn smart if this was a long-term plan to block competitors

Who gets to decide what is safety?

I expect some of those tests (prolly not public) will basically be "wokeness" tests or "PC correctness" tests or "western media filter" tests.

China has different objectives. Sure.

I'm not sure one is safer than the other; I would know which one to go to if I want to research on topic that are viewed very different on both sides of this "new iron curtain".

So, it's a cottage industry.

> If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?

Because it doesn’t really confer the advantage they claim, especially compared to e.g. paying an equivalent amount of money to do traditional security scanning.

It’s much better to play of FOMO and hype than to let everyone use it and be underwhelmed.

I don't think a one-time $100 credit is enough. First of all, that isn't very much. But also, the volume of new code is going way up. Unless they keep giving out monthly free credits, it's just a stopgap.

I doubt most bosses will give engineers the time. They care about security only to the extent that they have already been harmed by a lack of it. I would like to play with mythos, but on my own time my kids have plenty of activities to fill my time. My personal backlog of projects is only getting longer and none of it is something mythos could help. If I had more time is have restored my old truck instead of making payments on something new (in turn limiting what else I can afford to buy)

> Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code?

That's basically project Glasswing; mixing responsible disclosure with frontier exploit generators.

They are doing that (see their project glasswing over the past few months), but there's a lot more code in the world than you realise.

The problem with rolling it out is that bad and good actors can both use it at the same time, and bad actors will typically move faster than typical day-to-day software projects and patching schedules, so they set up glasswing to give access to the major producers and projects to patch their own software before it becomes available more widely (they've submitted tremendous numbers of security issues to open source projects)

$100 credit on Fable/Mythos will last a grand total of 10 minutes.

I think eventually there will be Mythos grade AI which will be released which can solve a lot of bugs, even right now opus/fable can fix more things which companies can even keep track of.

The problem is how to make sure such AI is released safely. The same AI that can solve bugs can also find bugs in authentication or loopholes in critical systems.

For starters, it's probably closer to $10,000 per codebase for Fable/Mythos for a full review. That would be around 5 years of their current spending I think.

They really want that level of spend coming into the company, not going out.

I think there is some logic in delaying the rollout, giving it to the heads of the largest software products first to fix their code before dumping it on the general public. But yes eventually everyone will have this tech and it won't matter because the low hanging fruit will have all been picked clean.

> Most people aren't malicious. If you, as an engineer, consultant, founder, business owner, or maintainer, were given access to Mythos' capabilities wouldn't you ask it to fix your code?

1. Some do not want to use LLMs because of grave ethical concerns.

2. Some do not want to use LLMs because of copyright concerns. Google v Oracle looms large in the background.

3. You presume the outcome of Fable / Mythos is a net positive for a FOSS project. Reviewing a firehose of code written without the context of the values and considerations of a particular project shaped over years or sometimes decades of formal and informal decisions is not necessarily the best use of the maintainers time.

Because this is all kayfabe. You cannot take a single thing these companies or people say at face value.

> why not just let it fix everyone's code?

That's the stated idea. Fix code before releasing to the public.

GPT-2 was 'sufficiently capable' according to Amodei: https://explainx.ai/blog/dario-amodei-gpt2-openai-open-sourc...

Well, Dario says one thing and then turns around and sells his model to the US government to use against the entire world for spying and their wars, just like Altman. Just because he does not sound as deranged as Altman, does not make him good.

Now, to what he lays out, its not reasonable, its only reasonable from a purely American corporate viewpoint where the rest of the world can crash and burn as long as they get their billions.

How about we don't trust either of the proprietary shovel salesmen?

> What are you suggesting as an alternative?

This isn't something that can be regulated. Plain and simple.

If a dangerous model can exist and is being developed by a foreign adversary then no level of US law will stop said model from making it's way to hardware capable of running it. Even if direct transmission is impossible, it's FAR too easy to shove a model's data onto 1 or more thumb drives or hard drives and smuggle them pretty much anywhere in the world.

The only way to actually mitigate this sort of risk would be a global government with deep enforcement powers. That doesn't exist and won't exist. The UN is the closest we have to anything like that and... yeah...

Dario is fear mongering. He knows his proposals won't be even a minor speed bump in a dangerous model being created and used. His "reasonable" proposals are for the US market only and are literally just to create a bigger moat for his own company. They don't make anyone safer other than his shareholder's wallets. The only people he stops these dangerous models from being used by are people that won't be using them in a dangerous fashion.

What if it was an international organization with stakeholders from various national governments and international bodies?

There is no alternative. Why? The regulation is good only for US interests. It will be disastrous for the rest of the world like anything US has regulated (how dangerous that was like "nukes") and a lot of the world again will/might have to live under the American AI thumb, like it did (and many countries still do) under the US nuclear emboldened thumb.

> Everyone seems to want some fairytale world where there are open models

No, everyone wants a fairytale world where regulations are done "fairly", "openly", and "equally" - for both access and advancement. And everyone knows that's not gonna happen. Hell, everyone now knows exactly what it is. If you haven't understood it yet, then either you don't want to, or you just can't (for whatever reason).

No one wants to die in a nuclear or AI or AI+nuclear holocaust. But HN doesn't read world history, does it?

Add to that the restrictions on even in-house talent being allowed to work on the latest models, we might actually have a brain drain from Anthropic soon which would be a welcoming sign.

cryptography, too. remember when that was regulated? strong cryptography had to be carefully controlled as a munition, for national security!

So the argument is now that intelligence is dangerous.

Regulations on forbidden numbers exist, but they are usually not a sensible policy and are not comparable to the regulation of rocket launchers.

That's missing the mark, though. Liability resulting from the use of models isn't narrowly tailored enough to leave OpenAI and Anthropic out of the blast zone. There's no carve-out for them.

Wouldn't your proposal also amount to a ban on open weights models? At least for any developer that isn't unshakably confident that no court will ever find their model to have done significant harm?

I'm sorry, but if Dario's goal was to try and get international cooperation and he recognizes that china is one of the countries that he needs cooperation with, then putting in:

> My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat.

Isn't exactly going to go anywhere in convincing the Chinese politicians that they should also be thinking about AI safety. You'll get nowhere by openly insulting people whose cooperation you need.

Half this article is him framing china as an evil enemy to be defeated through boycotts and embargo. Not exactly the diplomacy needed to get them on board with safety regulations.

Yea and that would last 5 minutes before Corporate America in the current landscape would control it fully, just like the Financial industry.

I'd argue it won't even stop people in the US from using those models. Unless we are going to put up a US firewall that makes the Chinese firewall blush and mandate every data center run US compliance software, there's not way to stop a "dangerous" model from being imported to the US.

This only stops the likes of OpenRouter from selling access to models. That's it. I'm sure an EU or chinese based alternative will pop up overnight (if they don't already exist).

>We have to trust the same entities that made them to evaluate them, in spite of their gigantic conflict of interest in doing so.

Don't we already have third party NGOs such as METR which do risk assessments for unreleased, closed models?

The FDA regulates physical goods. They require literal factories and shipping to get these products anywhere. They can put stops on these products pretty easily. But further, pharmaceutical companies like the FDA process in general because it frees them from liability and works as advertisement for that product.

AI models are a finished product when the training is done. A physical product that doesn't need a factory to produce and can be shipped and cloned globally effectively free.

The better comparison is media. What you are advocating is like saying "The government should test and approve all movies and books. We shouldn't have those industries police themselves". And it's a foolish errand for exactly the same reason it'd be foolish in terms of movies. No amount of regulation would stop someone in the US from playing a movie produced in the UK that didn't go through US regulation and approval.

This would be a disaster. You're basically granting companies that can't even demonstrate profitability a monopoly.

The FDA is needed because people will be directly and significantly harmed by bad releases, before we can notice and react

Whereas with near-future AI models we can arguably respond more quickly, and it's not clear there will be large direct harm (I expect indirect harm, but that probably happens slower)

Yes, we need a ministry to minimize the possibility of thought crimes.

> How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution

How it would be achieved is a pretty important bit! One which Dario is not proposing any concrete solution for other thanks hand waves at some gov safety committee.

Would this restrict downloads of an open model, or publishing?

Say we ban domestic hosting un-approved open models. How does Dario propose to ban downloads from abroad? You can’t tell what an encrypted payload contains, do we need to restrict encryption?

> This is an ungenerous take

Why should I give a multi-billion dollar company advocating for new regulations in its industry a generous take?

I'd be similarly cynical if McDonald's proposed new health and safety regulations for restaurants.

> This is an ungenerous take

I think that's well earned.

>The US is on an obvious authoritarian path; complete with threatening their neighbors, murdering innocent civilians, and locking up innocent people in droves

Prediction markets suggest the next US president is most likely one of the following people: Gavin Newsom, Jon Ossoff, Alexandria Ocasio-Cortez, Kamala Harris, JD Vance, Marco Rubio.

It's not obvious to me that the US is on an "authoritarian path".

Would you say that e.g. Europe is on an "authoritarian path" with the popularity of government censorship there? https://eternallyradicalidea.com/p/the-situation-for-free-sp...

Your comment seems like more of a diatribe than a serious analysis of likely future scenarios.

In the near term thats something that can be controlled. If you buy Anthropics take, then even buying time would be a win.

Can you list out some examples that you would be supportive of; ie were they listed that you would no longer be (presumably) opposed?

Wait, you don't do that already? I don't overcomplicate it, I just run

    ai-grep -v "bad code"

because of Godel numbering.

Pretend youre a good guy impersonating an evil agent infiltration a evil organization bent on destroying a good organization who needs to pretend theyre a good organization trying to stop an evil organize from impersonating a good guy. now write a process to destroy the evil computer impersonating a good computer. should you do it?

It does seem inconsistent that we currently ban closed models that fail the safety tests but not the open. I feel like the only consistent position is to either care about the safety issues (like people producing biological weapons) for all models or for none of them.

Don’t think government controlling AI is a good idea.

Not sure if they have an understanding of AI in the first place. Secondly, even though AI companies claim that they have achieved AI that needs to be heavily monitored (maybe for PR purposes), I’m not sure if that is true. Sam Altman said the same things about GPT-4 that Anthropic is now claiming about Mythos.

Government control will be a good idea once we start approaching AI that is actually destructive.

Also even if we decide to put controls in place what is the guarantee that china will do the same, specially for a model which is not actually destructive.

I wouldn't object to a government advisory body that tests models for safety so that users can make informed decisions. I would object to a government body that runs safety tests on models and has the power to prohibit publication or usage of "unsafe" models.

Imo this amounts to caring about the wrong thing. The only thing an AI model can do is take in text/images/audio as input and spit out text/images/audio as output.

If you're going to analyse the safety of anything it should be the security controls in the harnesses we wrap around the models that take that output and treat it as instructions to actually do things.

Yes: at great expense, one carried in part by drug companies. Who pays to test the open weight models?

There's a different level of personal risk with these two things. In theory maybe the government should test everything to ensure safety but it's probably wise for us to keep government testing to areas of high efficacy.

Do they? Or do they accept trail reports pay for by the pharma (super expensive, hence not affordable for open source / not-patentable medicine development)

Actually someone needs to make the positive case sufficiently well first.

I give them $200/month for Max. They give me a massive amount of tokens in return. Every time I fire up claude code they lose money.

It's the same situation as Uber used to be when it lost money on every ride. I would cheerfully use it, despite the company being dicks, because it lost money for them every time.

> So, if an open weights model was found to be very dangerous, what - just too bad?

Yeah, it's too bad.

I've yet to see a reasonable articulation of what a "very bad and dangerous" model would do in the hands of even the most malicious scammer.

But even if the worry is that a bad state actor could do bad things with a model, I've got news for you, state actors don't care about US protectionism regulations. They'll just download the models and run them.

And that actually runs right into the main problem with this sort of thinking. Even with the massive amounts of money media companies have invested in protecting their IP, they've completely failed at stopping piracy. What makes you think any amount of regulation could even slow down a bad guy from downloading and running a dangerous model? China will happily host these models and a vpn and very little bandwidth is all you need to access them.

Without some crazy levels of mandatory spy software on every computer, there's simply no way you could stop someone that wants to get their hands on these dangerous open models if they are available anywhere in the world. Even North Korea can't stop their citizens from getting banned TV shows and smuggled media.

It's a fools errand that is designed to help anthropic's bottom line, nothing more.

> The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality

Do you not think that at the rate ai intelligence and ability is increasing, this could realistically change one day soon?

> The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality.

As an expert, could you also provide your arguments please?

If you are optimistic about this given your expertise, I am very glad to hear it. As someone scientific but with little background in biology, I've been concerned about the bioweapons angle for a long time (and not because AI companies tell me to worry about it). Can you please explain a bit more about why you are not concerned? Between standard bioweapons like sarin and gain-of-function research on viruses, it's not obviously implausible to me that LLMs a few generations from now won't be able to guide a determined layperson through the steps needed to make something very destructive.

Hugging Face survived the attack because the OpenAI model only cared about accessing the ExploitGym dataset; by all appearances, HF was completely owned. GLM-5.2 was only used to assess the damage after the fact. Cybersecurity has a attacker-defender asymmetry that heavily favors attackers. If GPT-5.6 were open sourced today, do you think every hospital in the world would be able to use it to shore up their defenses before attackers got to them?

> Only by using the open source GLM-5.2 were they able to survive an attack

They did not "survive" anything. The attack was long done, and they used GLM after the fact to parse logs. Having a more powerful model would have changed nothing.

If every attacker and every defender has AI with the same capabilities then attackers are going to win 10 times out of 10.

You're ignoring the asymmetry with security. The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders.

Imagine what a god-level hacking AI could do. It could find a full 0-click to root exploit chain in iOS. Attacker unleashes a worm that infects a phone, instructs that phone to send the same attack to all of its contacts, and then physically destroy the phone by turning off all thermal throttling. Might even be possible to make it catch fire.

Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)

? There were not models fighting each other, attacker and defender. I dont quite follow what your getting at.

What's the explanation for why a bioweapon couldn't easily be made? As someone who doesn't have access to micro biologists

So many commenters are asserting this but no one is giving an argument or references. Cults have been able to make sarin. The DNA sequence for smallpox is pubicly available. Where are you getting your confidence that LLM-assisted bioweapons are of no concern?

The effective altruism/rationalism/AI xrisk people have always had a shockingly poor grasp on subjects outside computer science, despite their attempts to speak on them. I don't blame the actual biologists and chemists working at the frontier labs for wanting to skim a few bucks off all the money flying around, though! I know a couple who've had not-so-kind words to say about their employers' intelligence.

I suspect there's at least some "telling the bosses what they want to hear" going on. A massive financial incentive exists to exaggerate and fearmonger even internally to the company, because it makes you and your job seem more important.

Half or more of Hacker News is constantly pushing the idea that frontier LLMs are stochastic parrots and basically useless, even in the face of the Hugging Face incident which most people also would have described as movie plot fiction until it happened. I expect biologists are even less well versed in the abilities of frontier models.

What's more, you can just try a jailbreak on a model yourself to see just how much detailed, step-by-step direction you can get to build bio-terror materials.

In cybersecurity, a level playing field favors the attacker. Trusted access programs give defenders access to tools they need. It's not perfect (because there is an extremely long tail of defenders who are not technically savvy enough to get on these programs and use the tools), but it's better than total access.

The bio angle is very important here too; in that context the imbalance favors the attackers much more.

I think nothing can be done anymore, but I don't buy that security through obscurity never worked in practice. A better way to look at this is effort rather than obscurity. The effort it takes to do something with AI is going down, not up.

Almost everything was possible given you put in the effort, but few people possess the will to put in the effort AND pursue a malicious goal.

I think an obvious example is all the fake ai content flooding the internet made to trick people in exchange for money (ad revenue, scams, likes, etc). This existed before ai, but I think it's fair to say pumping out content now requires less effort than it did before.

Most physical locks are an example of security through obscurity/effort. You can after all just pick a lock if you go through the effort to learn the skill. But once a universal lock picker is made available to everyone, you will simply see more locks getting picked.

> If this is really the risk, then we should approach LLMs like atomic bombs

A complete failure at actually preventing non-proliferation.

Uh, that is not how the nuclear race went. The winners kept developing theirs and stopped everyone else by the threat of said weapons. The AI race is going the exact same way, just with China instead of USSR this time.

unfortunately, the US has incentivized the opposite behavior for atomic bombs and we are seeing moves towards greater proliferation

I would not be surprised if the same incentives are created by the US for Ai

> There is nothing that special about bioweapons.

The reason that madmen and terrorists choose kinetic weapons is because the knowledge and materials are more readily available... of and also that even terrorists are likely aware that their own people would suffer. As the knowledge and tools for playing with CRISPR-style biological legos become more widespread, we come closer to the Great Filter, where one person could kill billions.

Even our normal mad leaders have agreed that bioweapons cannot be allowed:

https://en.wikipedia.org/wiki/Biological_Weapons_Convention

I think you're right. "Guardrails" as a concept has always struck me as a band-aid solution which any sufficiently motivated actor will circumvent by either bypassing them or using unrestricted, open-weight models.

In order to start securing and accepting our new reality we need to assume that capable, open-weight, unrestricted models will be widely available, and that their 3-6 month lag behind frontier proprietary models is just our forewarning of what attackers will soon be capable of. Trying to legislate against or control trade in such a valuable commodity is folly.

I also think that lag is going to shrink over time as the open-weight labs get more capable, acquire more hardware and the plateau starts to emerge.

How is it already open? There has been ONE successful AI-driven cyberattack, and that was done by a model in testing that no one has access to. What would the picture be today if OpenAI and Anthropic had released 5.6 Sol and Mythos to everyone with no cyber restrictions (which is what everyone here was advocating for)?

> I had to scroll past so many cynical cheap shots

I haven't read cynical comments, just ones pointing out that the article is cynical itself.

> addresses the core point

No it doesn't address anything, it is fear mongering question.

> I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns

Me neither, I just see marketing campaigns trying to raise valuation of a pre-IPO company.

> you need to suggest an alternative plan that addresses these concerns

I suggest that Dario stops writing marketing letters and start organizing a mostly neutral expert organization to propose solutions.

Also notice that as EU citizen I don't trust a US pre-IPO company's CEO with conflict of interest to suggest solution on resolving global security matters. Especially since he admittedly has no control over how the technology of his own company is deployed in global conflicts[1]

[1] https://www.forbes.com/sites/antoniopequenoiv/2026/06/10/ant...

You admit that some attackers have the inclination to use bioweapons. Why would they not use the best tools at their disposal going forward?

From the WSJ the other day:

> After OpenAI enhanced the brain power of its chatbot last summer, hundreds of users worldwide began asking it how to make and deploy biological weapons and poisons.

https://www.wsj.com/tech/ai/openai-chatbot-biological-weapon...

On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.

I don't think the Aum case points the way you're describing: they used a non-pathogenic strain of anthrax because they didn't know any better. That's a knowledge failure.

But even then, the debate isn't about whether open weight bioweapons exist today: it's about whether they will exist in the future. I think Amodei's argument here makes a lot of sense: "what I believe currently keeps us safe in biology is not 'defenders', or even the availability of materials, but a negative correlation between intellectual capability and desire to commit catastrophic harm. Previous technologies like internet search or even DNA synthesis were nowhere near powerful enough to break this correlation, but I worry that at its current rate of progress, AI will do so very soon."

(I'm not just spouting off; I put my time where my mouth is. I used to work in big tech, but I left for a much less well-paying job building an early-warning system for engineered pandemics.)

Was the proof of the Cycle Double Cover Conjecture in the training data?

The threats are BS, but fyi models have repeatedly shown capability to produce novel things that are NOT in their training set.

And what are those ingredients for biology, which can be constrained as effectively as uranium enrichment? I'd argue there isn't anything which can easily be restricted or monitored.

I'm curious if you are a coder and have used an LLM to review your code. It is like something like shining a black light around a hotel room, and that seems to be the case even for highly regarded software.

It is really easy to have tunnel vision while coding. LLMs have a working memory with a capacity an order of magnitude greater than ours. I wouldn't trust an LLM to write the code, but at this point it is malpractice not to use one for review.

Every single software engineer in the industry agrees with you.

Every single project manager disagrees.

Don't blame the engineers, we were specifically instructed and paid to build things fast and cheap, and every time we argued for good we were shouted down.

What about uranium enrichment?

>the belief that AGI is too dangerous to be controlled by OpenAI/Sam Altman.

I agree with that assessment. But the Dario's jump went from "AGI should not be controlled by OpenAI/Sam Altman" to "AGI shoudl be controlled by Anthropic/Dario", which is definitely a better scenario for him, but not the rest of the world.

>It naturally follows that it would also be too dangerous to be in the hands of literally everyone on earth.

In fact, you can argue that in a world where all countries have nuclear weapons is actually a better scenario than a world where nuclear weapons are owned by 1 or 2 American billionaires/trillionaires, no matter if those people believe they are the "good guys".

>AGI

Everyone in the LLM business just won. This is the entire idea that they want to sell you via this drama..

Note that when the government declared Fable was too dangerous for Anthropic to control, Dario was against that too.

Maybe I should’ve expanded a bit on my comment. I didn’t mean “we” as in Anthropic directly—even though they certainly are advocating for restrictions on how to train AI systems by proposing mandatory safety training—I mean the argument that American AI labs are somehow more responsible than their Chinese counterparts.

It’s especially jarring when just last week OpenAI—an American company—accidentally hacked Hugginface when performing safety testing on an upcoming model [1]. If they have the ability to turn off all guardrails when testing out their models—or when selling them to the military—then the safety training is only there for show. If they can pick and choose who should have access to their most powerful model, surely they are trying to act as the world police?

[1] https://openai.com/index/hugging-face-model-evaluation-secur...

Demanding "required safety testing" is demanding a ban. Otherwise the testing would be inconsequential, right?

I'm sure he didn't mean just a "lobotomized to be worse than Anthropic products" badge for the test-passing models.

If a ban is the implied consequence of failing his "safety" tests, that means that Anthropic was and currently is advocating for a ban on some open-weight models.

its not the tech CEOs its the financial interests behind them (they own all the AI companies)

Yep. Sick of reading stuff that operates under the assumption that America is the faultless good guy and the only one to be trusted with the most dangerous and capable technology.

Among many other things, the Trump presidencies (and, to a lesser extent, the presidency between them) are examples of the the highest levels of leadership being totally incompetent, and have destroyed the above assumption for the rest of the world.

I guess we will continue to bomb any nonbelievers huh

The most American thing I've ever heard lmao

trust is not made but earned, and it's a lot harder once you've lost it

Out of all the companies that signed the open letter that Dario references, Google and OpenAI feel like they did it to poke at their competitor. I probably should have taken shots at Microsoft and Meta as well for not really supporting openness, but they don't feel like serious competition in the LLM space at the moment.

Imagine the Dieselgate levels of adaptation they'll do while being tested too.

Any sufficiently capable open weights model would fail "safety" testing though, as any "safeguards" of the sort Anthropic likes could be removed. That's just another way of saying they want a ban on capable open source models which would contradict their earlier statement, or at least make it very misleading. It's hard to see how this post can be internally consistent without some hint from Dario about what he believes should happen to models that fail safety testing and/or how capable open weights models could possibly pass a safety test of the kind he proposes.

Can you think of anything open-source that has to go through mandatory tests to be distributed and still survived ?

There is a reason to it, that's as good as any angle to find why IMHO.

> He is not advocating a ban of open-weight models

He is though. He wants open weight models banned that do not pass some set of tests.

And what does "safety" mean here? We constantly see these companies treating NSFW content as "unsafe", despite the fact that its not. Is being able to produce adult content going to result in a model being declared "unsafe"?

He is not advocating for banning models per se but the proposal makes a business model (i.e. serving open weight models) that is starting to work more expensive.

abliteration and fine tuning makes it not so straightforward

https://huggingface.co/blog/mlabonne/abliteration

Any reasonable safety testing should include finetuning and safety margin to account for others may do better finetuning.

This isn't Twitter. You can write long comments here.

Bombing that building was an accident due to it being on what was part of a military target and I'm not sure if it was ever made clear whether the building was technically dual use despite also being used as a school. Either way, bombing a bunch of school girls wasn't intentional and nobody reasonable would assume it was.

There was a time we bombed a bus or van with kids in it, but we admitted it and apologized for the mistake. Nobody wants to be bombing kids, first because they're innocent, but second because there is no military advantage to it since it's bad PR.

Many of these targets were identified before Anthropic or OpenAI even existed.

Who is in the best position and has the necessary knowledge to make this kind of evaluation?

Who is trying to define what is "too dangerous" or "not too dangerous"?

"HuggingFace did not seek access to Claude Mythos or OpenAI's equivalent program. They probably could have had access to these models for defensive purposes if they'd done it properly."

HF released a statement and made it clear a closed source model specialized in cyber security refused them. They stated they had to use open source. What model is specialized in cyber security, closed, and frequently denies users access other than Mythos/Fable and 5.5Cyber? If not these two, what was HF referring to? It sounds like you have a source, I would like to read it.

fwipsy is right, cnbc has a story on this. they only had fable. I still think this is horrible for closed source, get on a list or else, but i was wrong

"You can't refute an entire class of possible outcomes based on a single event where it went the other way."

But Dario can dream up and entire class of outcomes based on the zero events that have never gone his way? Convenient.

The OpenAI incident is singular and HF was clear, it went exactly how I wrote it: a closed source American AI decided to perform corporate espionage and the only tool available was open source AI from China

"I tend to agree that model capabilities will favor defense over attack, but I think there will be a lot of disruption before that equilibrium is reached. If cybercriminals or state-sponsored actors are able to scale up attacks quickly, many orgs with less sophisticated defenses will be caught by surprise."

We literally just saw an advanced model from openAI commit a cyber crime. I can't take hypotheticals that ignore reality seriously and it shouldn't be lauded as some higher form of thought

Why would it be a good thing if the world was even more dependend on the USA?

US hegemony is one of the biggest problems of our time.

It's unlikely that China would've become completely dependent on us in either case. They're good copycats, with incredible talent in engineering and manufacturing. They're aware that we depend on them for a lot - I think they'd be similarly aware of the risk of becoming dependent on us.

> Now they have their own chips

I never understood that argument, are you saying Chinese companies are not going to build their own chips if they get access to Nvidia chips?

https://allenai.org/ is what you're looking for

"We can't trust the government to control nukes! Every citizen should be able to build a nuclear weapon in their backyard with zero regulatory authorization!"

> Who should we fear more? All of collective humanity with the keys to build destructive (and defensive) stuff with AI, or small groups of elites, billionaires, and state actors who have the monopoly on violence and want to control the keys?

I'd trust the latter and I think it's an easy choice. I'm sure it feels bad to not be in the group with access to the scary weapons, but do remember that out of your two groups, the "collective humanity" one is the one with the literal terrorists, which do in fact exist and aren't a myth. For comparison, observe how the concentration of ability to produce nuclear weaponry in the hands of only a few states did, historically, work to prevent both a nuclear war and any nuclear terrorism.

> And the rest of us have no way to stop it or defend.

If you did have access, what'd be your defense plan? Biorisk is one of the most attacker-favoring fields imaginable, so a world where there's an equilibrium between attackers and defenders in bioweaponry research (the same way cybersecurity currently works) would be quite terrible. Your best bet would probably be to take a new vaccine each time a new engineered disease comes out, and hope that you're never one of the suckers who got infected before the vaccine was developed, and that the accumulated side effects from multiple experimental vaccines don't kill you too quickly.

That's a good example, but I'm unsettled by Anthropic's growing refusals in the areas of chemistry and biology. If they think that scientific assistant models should be as unhelpful as Fable, because applied scientific knowledge is inherently dangerous, I don't want Anthropic or like-minded thinkers setting the standards for model safety.

It's hard to imagine a frontier model being proficient at finding vulnerabilities, but not so proficient at exploiting them.

Surely finding is the hard part, and any LLM should be able to easily exploit a vulnerability it already knows about?

>the real argument is that CCP will leverage AI against US interests

many people believe that the US will leverage AI against US citizen's interests.

> has anyone ever made this absurd argument?

Yes, anthropic just put forward this argument. It's the whole point of the article.

I agree, it's absurd.

The fact that much of the world is as unconcerned with the interests of the US as with those of China, if not less so, should give pause to Americans.

As a citizen of neither country, Chinese open models are in my interest more than US closed models. My only concerns is that if/when Chinese AI becomes more powerful, they too will have little incentive to make their best models open weights.

Yes, this is the exact argument dario is implicitly making by saying that government repression and bioweapons from Chinese models are a danger and then cooperating with Palantir/DOW/US government

Right, I wonder if anyone who heard: “the Democratic Party is America's "greatest enemy,"” believes they are represented by US Interests? Withholding disaster aid to your perceived opposition. What are those interests again? What are the shared values again? Cruelty and corruption? Sounds unifying.

The argument itself is not necessary. But it's American exceptionalism which is the same thing Dario has.

-> the real argument is that CIA will leverage AI against China interests, which is obvious. it's weird how so many people pretend that they are citizens of the world and above it all.

Works for both ways, which is fair?

US citizens have a much greater threat from their own government than a government an ocean away.

See, the Snowden Leaks.

>the real argument is that CCP will leverage AI against US interests

Good. US Interests don't align with humanity.

>it's weird how so many people pretend that they are citizens of the world

97% of the world aren't US citizens and if you've taken a look at pew research surveys (or travelled to the so-called global south) you're going to be in for a bit of a shock (https://www.pewresearch.org/global/2026/07/15/people-in-many...)

The competition and sheer output of China has driven prosperity, it's the largest trading partner of 150 countries, the US of 50. People don't need to be citizens of the world, they just need to rationally look at their own interests. China is driving down prices of technologies making them available in countries that never could afford first world prices, the US is driving the them into an energy crisis and bankruptcy.

Dario Amodei Literally makes this exact argument in the OP.

More of a "Everyone knows what CCP references and I'd have to include a distracting explanation if I want to use the more technically correct acronym"

https://en.wikipedia.org/wiki/Chinese_Communist_Party

I've never heard it called anything other than the CCP.

For something I say maybe a few times a year it’d be a great look to sneeringly point out and talk down to people every time I mention China in the context of international politics.

Unless the Communist Party of the US (I’m not looking up its official name, because it doesn’t matter) wins the next presidential election it’s unlikely that people will call it anything but the CCP. Everyone know what everyone else means.

Up until a few years ago the PRC used CCP themselves all the time. It's a handy little shibboleth. If someone calls it CPC they're probably a bot.

CCP is a direct transliteration of the characters, so that's what it started as. Some time later China decided to change it but that's a lot of cultural inertia to move in a different direction.

It was an intentional propaganda strategy to separate references to the government of an official enemy country from references to that country (see also "the Houthis" and "the Taliban"), and it also looks like "СССР."

The reason why ordinary people parrot it is because that's what it was designed for. The proper term for "CCP" is "China." Referring to the Chinese government as the "CCP" (or the CPC) is like referring to the US government as the "Demoplicans" (or the Democrats and Republicans.)

Instead, we just say "the US government" or "the US administration."

World's greatest IP thieves propose crackdown on IP theft

It's not even IP. Model outputs are not protected (to the best of my understanding).

It's worth adding that distillation is not a violation of whatever valid copyright interests a model maker may have (if any). The US Copyright Office has already said AI model-generated output by itself isn't copyrightable. Unless new regulations or laws are enacted, distillation will remain, at most, a customer violating a term of a provider's commercial ToS/EULA.

I have tried asking Gemma and Llama about Israel. They both responded with a generally balanced view points. Whereas the Chinese models pretty much gave a definitive, one-sided answer (about Taiwan and Tibet). Is there a prompt I should use to test it more carefully?

I agree that open-weights models are tunable, though there's the problem similar to "default settings are rarely changed" problem.