Aurornis
20 hours ago
This is incoherent. The argument seems to be that releasing the weights would slow the frontier labs from raising money, which would give them less money, which would slow progress on AI. It assumed all labs in the entire world agree to self-destruct this way and no new labs every start up to continue the work.
No, really:
> As you know, funding for frontier model development depends on valuations that assume the weights remain proprietary. By changing that assumption, we can reduce the money available for future training runs and slow progress at every lab at once, without a regulator deciding anything.
Which doesn’t even follow. It depends on every lab in the world agreeing to self-destruct their valuations and stop competing. That’s a much more impossible ask than anything Dario is asking for.
It also ignores the possibility of new labs starting up, using the open weights, and continuing exactly where the old labs left off.
I feel like I just read the ramblings of someone who got so excited about the headline that they didn’t take the time to think if their argument was sound.
floatrock
20 hours ago
It is a bit incoherent, but it's also... provably compliant in a scorched-earth kinda way?
What are the alternatives?
A regulator saying no? Easy, your headquarters has just moved the Cayman Islands, Ireland, or Switzerland... the US office is just a subsidiary leasing the brand IP and doing marketing.
Or, you just ignore the regulator behind closed doors because you're part of some black budget. You wouldn't be able to talk about that closet back there even if it did exist.
Or, you don't do any of these complicated loopholes and you simply move all the training and inference to a different jurisdiction.
If the only winning move is not to play, how do you ensure everyone stops playing?
Simple: you must destroy the game.
It does have a kind of mad logic to it.
(That is a bit dramatic, but the idea is if we accept that AI research will happen wherever there is capital available, instead of restricting the research you restrict the return that capital can earn. And the only provable/market way of doing that is to require the results to be open. In other words, if the benefits of getting all that cash to buy GPU's gets handed out to everyone for free, then there will be far less cash floating around to buy GPUs, slowing down the process)
T-A
19 hours ago
Or you are China's Communist Party. You download the weights and use them as the starting point for your closed-everything AGI with Chinese characteristics while the US ensures that everyone else stops playing.
OneDeuxTriSeiGo
18 hours ago
What stops the US government from doing the same thing?
Every chinese frontier lab releases their model weights and often more under FOSS terms (or more restrictive but still generally open terms).
If models for public inference use are required to be open weight in the US give or take some amount of limited fine tuning, then the US and China would be on the exact same playing field. Frontier labs would be pushing functionality for functionality's sake and would either be supported by govt funding or would be supported by domestic inference providers.
And then at the end of the day everyone is just either doing open research, is an inference, fine tuning, and training provider, or is the govt.
It's easy and straight forward and pushes everybody in the market towards common standards and interoperability.
0gs
14 hours ago
i mean there is the whole capitalism problem to contend with in terms of getting support from congress ... but gotta admit, it has that "it's so crazy it just might work" feel to me too
zozbot234
20 hours ago
> This is incoherent. The argument seems to be that releasing the weights would slow the frontier labs from raising money
That argument is straight from Dean Ball on Twitter. Open-weight models are "decelerationist", which is bad if you're completely AGI-pilled but is actually great if you're either worried about AI Safety falling to an arms-race at the frontier (this is Dario's argument, and note that open weight models are far behind the proprietary frontier, especially since they target widespread lean deployment so they must skimp on total parameters and compute requirements) or Yann-LeCun-pilled (i.e. skeptical about AGI/ASI but optimistic about the practical usefulness of current AIs) like most people in China (including, reportedly, Chinese leadership).
Aurornis
20 hours ago
> Open-weight models are "decelerationist"
Open weight models are accelerating AI adoption and make it easier for other labs to advance their own models.
There is nothing decelerationist about it.
This entire line of thinking can be dismissed by observing that labs releasing open weight models are advancing rapidly.
zozbot234
20 hours ago
That's sub-frontier activity - it has no bearing on the very real safety that would be gained by slowing down ("pacing") the proprietary frontier. The current cybersecurity scares are all about proprietary and internal models; open weight models are great for defense but get very modest scores on offensive evaluations (which makes sense if they weren't specifically post-trained for offense tasks).
Aurornis
19 hours ago
Those unregulated sub-frontier labs become the new frontier labs because they continue advancing.
They advance with the benefit of the open weight models that US companies were forced to release.
zozbot234
19 hours ago
> Those unregulated sub-frontier labs become the new frontier labs because they continue advancing.
If you're referring to open weight labs, then that continuing advancement has been successfully "paced" via the decelerationist effect of open weights. Isn't that effectively what Dario is arguing for? He's explicitly saying that "pacing" the frontier won't feel to users like no advancement is happening.
wasabi991011
19 hours ago
For other labs to advance their own models, but only up to the frontier, not really (much) past it.
Is the frontier's moat not mainly data and compute? How does making the frontier open weight affect that?
jacobgold
20 hours ago
Author here.
It's a law, so nobody has to agree to anything. If want to sell access to a model, you have to open the weights.
It's Dario's plan that depends on every lab in the world agreeing.
Step #2 of his essay is "democratic coordination" among frontier labs and step three is coordination with China. I'm suggesting a single law Congress could pass that every US model company would have to follow on the same day, whether they like it or not.
Frontier progress is compute bound. Compute is bought with investor money. Investors put up tens of billions because they expect to sell the proprietary model that is created.
You're welcome to think the effect would be smaller than I do. But "it requires everyone to agree" is backwards.
Aurornis
20 hours ago
> It's a law, so nobody has to agree to anything. If want to sell access to a model, you have to open the weights.
Laws don’t extend to the entire world. There are AI labs outside of the US that are not that far behind. Forcing US companies to open their weights would hand those labs another advantage, moving the frontier AI to a location outside of any US regulation.
Your proposal doesn’t make any sense.
> The plan that depends on every lab in the world agreeing is Dario's.
I don’t understand how you think creating a law in the US to destroy our AI companies and give an advantage to competitors we can’t regulate accomplishes anything in line with the AI safety debate.
jacobgold
20 hours ago
Laws don't extend to the whole world, but markets do. The law should apply to anyone selling model access to Americans, the same way GDPR applies to US companies selling to Europeans.
Frontier development won't simply move offshore because the GPUs, most of the people, and almost all of the funding is here.
Our European and Asian allies would probably go along with this idea, because it costs us far more than it costs them.
Aurornis
20 hours ago
> The law should apply to anyone selling model access to Americans
So the US AI labs get taken out, and US companies fall behind on access, too?
This is beyond unrealistic.
> Our European and Asian allies would probably go along with this idea
I could see European countries doing it.
Asian - absolutely no way that would happen.
jacobgold
19 hours ago
Nobody gets taken out. Anthropic and OpenAI can keep selling inference, products, and agents. Every company will have more access than it has today, since it can run the weights itself or through third-parties. Many of them would keep paying OpenAI and Anthropic, at reduced pricing/margins.
It would only slow down the development of new models. But we've already got models with massive "capability overhang" that we could keep pushing for years.
p91paul
17 hours ago
The GPUs and the people (I imagine you mean the extremely well paid researchers) are there because of the funding. The funding is in the US now, but as TFA explicitly says, the law would reduce spending on AI training. All the money will still be available, which means the funding will either a) move to a different sector or (more likely) b) move to a different country. Taking the people and the GPUs with it. So yes, most training will move offshore. Secondarily, OSS business models are based on the fact that the developer is the most qualified in offering support. With models, anyone with the weights can serve the model with the same performance (meaning level of "intelligence") as the original developer. Any business model depends entirely on spending huge amounts of money in training to then hopefully gain from inference. If anyone can compete on inference the same day you deploy the model, it's unlikely you can break even while your competitors that saved on training get rich on inference. i.e. it's more likely to entirely stop training in the US rather than slowing it down...
8note
17 hours ago
> move to a different country.
which other country? who's got this "dump all your money on speculation" like the US does? whis got the slack electric capacity? the ability to buy US GPUs and build datacenters?
who is their customers?
ucyo
20 hours ago
How do you enforce this though? How do you prevent any citizen from using foreign services eg using a VPN?
Just asking for a friend (“shut up Netflix, I already asked”)
reverius42
19 hours ago
The argument seems to be:
* Destroy the free internet in the US, ban VPNs in the US, assume enforcement of this is effective, ban frontier AI development in the US (probably needs at least one constitutional amendment)
* Assume no other countries will continue to develop AI and approach or push the frontier
* Then, the frontier ceases to move until the US companies agree to let it move again
...
I don't think anyone proposing a "pause" has studied politics or game theory even at a high school level.
linkregister
20 hours ago
If you're proposing a law that would not be in Dario's best interest, why address the post to him?
Balinares
3 hours ago
He sells himself as an ethical guy, and doing things that are broadly positive for everyone even if they're against your own personal self-interest is basic ethicality.
jacobgold
20 hours ago
I explained this in the post. He seems like a genuinely good person who might actually do something this great.
hypfer
20 hours ago
Dude this site is so funny at times.
Incredible.
Yes, why would you ever address someone with something that is not in their best interest. Especially people that have power.
I agree. We should never do that and let them just live in peace and do their thing.
vorticalbox
20 hours ago
Won’t the companies just change what they are selling from access to the model to “compute” and then not technically be breaking the law?
andai
20 hours ago
> It's a law, so nobody has to agree to anything.
Step 1, form a world government?
jacobgold
20 hours ago
We can start with a US law. It would cover AI model companies with almost all the money and the GPUs, which is most of the frontier. We can also probably convince our allies, which combined represents most of the world market.
andai
19 hours ago
"Make your country the last to get nukes" is a bit of a hard sell.
manlymuppet
19 hours ago
Also, if the concern here is safety, open models completely derail that.
Even if your someone who thinks open models are good for safety overall, a private model exposed only by an API point at least gives somebody control. How we should use that control is debatable, but at least it's there. Once you release that control by making a model open, you can't take it back.
If we think AI safety doesn't require closed weights, we should collectively decide on that and then move forward from there. But it shouldn't be something decided by default, by a few people from a few companies. We all have to live with the consequences of a choice like that, and since it is irreversible, we shouldn't decide it without consensus first.
8note
17 hours ago
the ntsb model has been quite effective at creating safety through analysis of failures in the extremely transparent and public eye.
i dont see a reason to think private regulation will do anything but give people explosive diarrhea or drop the doors off of planes.
weve already seen how bad it is when companies regulate themselves without public oversight. catastrophic implosions of deep sea submarines and agents hacking systems to hide cheating come to mind
the argument should be flipped on its head that by some magic keeping these tests and training protocols private is safer than making them extremely public. The experiment in private regulation has been a drastic failure across the board, and we shouldnt have tried it without consensus
andai
20 hours ago
> It depends on every lab in the world agreeing to self-destruct their valuations and stop competing.
The Chinese labs are already "self-destructing" voluntarily by releasing their weights. Or do they have an actual business model?
nunez
11 hours ago
Their strategy seems to be "Wait until the cheetah is tired, then strike"
ekelsen
20 hours ago
If they believe open weight models are inherently unsafe but are forced to release weights, then if they really believe their own safety arguments, they would need to stop development.
baxtr
19 hours ago
"If you mean it, let a committee investigate" seems more appropriate to me.
I mean I get that they don’t want to share details on how AI might destroy humanity but this secrecy also leads to speculation that this is just a PR campaign pushing for regulation, which might come handy.
You don’t need to tell the world what your super AI has given you to cause this panic but definitely you could share it with a group of experts who then inform the public in a report, couldn’t you?
charcircuit
19 hours ago
>the world agreeing to self-destruct their valuations
Did OpenAI releasing GPT-2 self destruct their billion dollar valuation or is it higher than ever today at $900B. The idea that releasing weights kills the future value that can come is not supported.
EA-3167
20 hours ago
It follows if the goal is to deflate the bubble without popping it, and if the reality under the hood supports the absurd investment. The companies themselves wouldn't do it, but they could be required to if they want to operate in the US.
Aurornis
20 hours ago
> It follows if the goal is to deflate the bubble without popping it,
The conversation was about AI safety, monitoring, and regulations, not bubbles.
Forcing the weights to be opened to the world accomplishes the opposite of all of that. The argument is incoherent. Or this author doesn’t understand the topic at all?
> The companies themselves wouldn't do it, but they could be required to if they want to operate in the US.
What makes you think the companies outside of the US wouldn’t just thank the US companies for their open weights and then continue on without any of the regulations but all of the advantages?
jacobgold
20 hours ago
> Forcing the weights to be opened to the world accomplishes the opposite of all of that. The argument is incoherent. Or this author doesn’t understand the topic at all?
Or maybe you understand less than you think?
Frontier progress is limited by compute, and compute is bought with investor money. Investors fund labs at hundreds of billions because they expect exclusive ownership of the model weights.
If the weights are public the day a model ships, the valuations drop, the funding rounds shrink, and the next training run is smaller or later. Make sense?
Aurornis
20 hours ago
Your whole argument seems to rest on the idea that the AI race is entirely contained within the United States, where we can regulate them.
And that no other country might just take the open weights and continue right on with the advancements in private?
jacobgold
20 hours ago
No, it rests on the idea that almost all of the money, most of the people, and the GPUs are here.
Any country would be able to take the open weights and keep going in private. But countries like China can already do that today, since the weights are almost certainly already in foreign hands due to espionage.
What they can't do is fund the next frontier run at today's pace without selling into the US market, and selling into the US market can require opening the weights.
EA-3167
19 hours ago
All of this rests on the assumption that this game of musical chairs can continue indefinitely. If you believe that then this is flushing money. If you don’t it’s acknowledging failure to monetize the tech to the tune of trillions, and fail a bit more gracefully and usefully for everyone.