Excellent move by Newsom. We have a very active legislature, but it's been extremely bandwagon-y in recent years. I support much of Wiener's agenda, particularly his housing policy, but this bill was way off the mark.

It was basically a torpedo against open models. Market leaders like OpenAI and Anthropic weren't really worried about it, or about open models in general. Its supporters were the also-rans like Musk [1] trying to empty out the bottom of the pack, as well as those who are against any AI they cannot control, such as antagonists of the West and wary copyright holders.

[1] https://techcrunch.com/2024/08/26/elon-musk-unexpectedly-off...

https://legiscan.com/CA/text/SB1047/id/3019694

So this is the one that would make it illegal to provide open weights for models past a certain size, would make it illegal to sell enough compute power to train such a model without first verifying that your customer isn't going to train a model and then ignore this law, and mandates audit requirements to prove that your models won't help people cause disasters and can be turned off.

Related. Others?

OpenAI, Anthropic, Google employees support California AI bill - https://news.ycombinator.com/item?id=41540771 - Sept 2024 (26 comments)

Y Combinator, AI startups oppose California AI safety bill - https://news.ycombinator.com/item?id=40780036 - June 2024 (8 comments)

California AI bill becomes a lightning rod–for safety advocates and devs alike - https://news.ycombinator.com/item?id=40767627 - June 2024 (2 comments)

California Senate Passes SB 1047 - https://news.ycombinator.com/item?id=40515465 - May 2024 (42 comments)

California residents: call your legislators about AI bill SB 1047 - https://news.ycombinator.com/item?id=40421986 - May 2024 (11 comments)

Misconceptions about SB 1047 - https://news.ycombinator.com/item?id=40291577 - May 2024 (35 comments)

California Senate bill to crush OpenAI competitors fast tracked for a vote - https://news.ycombinator.com/item?id=40200971 - April 2024 (16 comments)

SB-1047 will stifle open-source AI and decrease safety - https://news.ycombinator.com/item?id=40198766 - April 2024 (190 comments)

Call-to-Action on SB 1047 – Frontier Artificial Intelligence Models Act - https://news.ycombinator.com/item?id=40192204 - April 2024 (103 comments)

On the Proposed California SB 1047 - https://news.ycombinator.com/item?id=39347961 - Feb 2024 (115 comments)

I wondered if the article was over-dramatizing what risks were covered by the bill, so I read the text:

(g) (1) “Critical harm” means any of the following harms caused or materially enabled by a covered model or covered model derivative:

(A) The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.

(B) Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from cyberattacks on critical infrastructure by a model conducting, or providing precise instructions for conducting, a cyberattack or series of cyberattacks on critical infrastructure.

(C) Mass casualties or at least five hundred million dollars ($500,000,000) of damage resulting from an artificial intelligence model engaging in conduct that does both of the following:

(i) Acts with limited human oversight, intervention, or supervision.

(ii) Results in death, great bodily injury, property damage, or property loss, and would, if committed by a human, constitute a crime specified in the Penal Code that requires intent, recklessness, or gross negligence, or the solicitation or aiding and abetting of such a crime.

(D) Other grave harms to public safety and security that are of comparable severity to the harms described in subparagraphs (A) to (C), inclusive.

(2) “Critical harm” does not include any of the following:

(A) Harms caused or materially enabled by information that a covered model or covered model derivative outputs if the information is otherwise reasonably publicly accessible by an ordinary person from sources other than a covered model or covered model derivative.

(B) Harms caused or materially enabled by a covered model combined with other software, including other models, if the covered model did not materially contribute to the other software’s ability to cause or materially enable the harm.

(C) Harms that are not caused or materially enabled by the developer’s creation, storage, use, or release of a covered model or covered model derivative.

The race for true AI is on and the fruits are the economic marginalization of humanity. No game theoretic actor in the running will shy away from the race. Anyone who claims they will for the 'good of humanity' is lying or was never a contender.

This is information technology we are talking about, it's virtually the exact opposite of nuclear weapons. Refining uranium vs. manufacturing multi purpose silicon and guzzling electricity. Achieving deterrence vs. gaining immeasurable wealth, power and freedom from labor.

This race may even be the answer to the Fermi paradox - that there are few individual winners and that they pull up the ladders behind them.

This is not the kind of race any legislation will have meaningful effect on.

The race is on and you better commit to a faction that may deliver.

The immediate danger of large AI models isn't that they'll turn the earth to paperclips it's that we'll create fraud as a service and have a society where nothing can be trusted. I'd be all for a law (however clumsy) that made image, audio or video content produced by models with over X parameters to be marked with metadata saying it's AI generated. Creating models that don't tag their output as such would be banned. So far nothing strange about the law. The obvious problem with the law is that you need to require even screenshotting an image AI and reposting it online without the made-with-ai metadata to be outlawed. And that would be an absolute mess to enforce, at least for images.

But most importantly: whatever we do in this space has to be made on the assumption that we can't really influence what "bad actors" do. Yes being responsible means leaving money on the table. So money has to be left on the table, for - erm - less responsible nations to pick up. That's just a fact.

Is part of the issue the concern that runaway ai computing would just happen outside of california?

There's another important county election in Sonoma happening about CAFOs where part of the issue is that you may get environmental progress locally, but just end up exporting the issue to another state with lax rules: https://www.kqed.org/news/12006460/the-sonoma-ballot-measure...

It was a dumb law so... good on a politician for doing the smart thing for once.

I'm not sure AI risks are well enough understood to have good regulations for. With most risky industries you can actually quantify the risk a bit. Regarding:

> we cannot afford to wait for a major catastrophe to occur before taking action to protect the public

Maybe but you can wait for minor problems or big near misses before legislating it all up.

This veto document is shockingly lucid, I'm quite impressed with it despite my belief that regulation as a strategy for managing critical risks of AI is misguided.

tl;dr gavin newsom thinks that a signable bill needs "safety protocols, proactive guardrails, and severe consequences" based on some general framework guided by "empirical trajectory analysis", and also is mindful of the promise/threat/gravity of all the machine learning occurring in CA specifically. He also affirms a general appetite for CA to take on a leadership role wrt regulating AI. My general read is that he wants to preserve public attention on the need for AI regulation and not squander it on SB 1047 specifically. Or who knows I'm not a politician lol. Really strong document tho,

Interesting segment:

> By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047 - at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good. > Adaptability is critical as we race to regulate a technology still in its infancy. This will require a delicate balance. While well-intentioned, SB 1047 does not take into account whether an Al system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data. Instead, the bill applies stringent standards to even the most basic functions - so long as a large system deploys it. I do not believe this is the best approach to protecting the public from real threats posed by the technology.

This is an incisive critique of the fundamental initial goal of SB 1047. Based on the fact that the bill explicitly seeks to cover models whose training cost was >=$100m & expensive fine-tunes, my initial guess about this bill was that it was designed by someone software engineering-minded scared of e.g. open-weight releases a la facebook/mistral/etc teaching someone how to build a nuke or something. LLMs probably replaced the ubiquitous robot lady pictures you see in every AI-focused article as public enemy number one, and the bill feels focused on some of the technical specifics of this advent and its widespread use. This focus blinds the bill from addressing the general danger of machine learning however, which naturally confounds regulation for precisely the reason plain-spoken in the above four sentences. Incredible technical communication here.

> gov.ca.gov

Ah, I think now I know why Canada's government website is canada.ca (which I remember thinking was a bit odd or more like a tourism site when looking a while ago, vs. say gov.uk or gov.au).

Curious if anyone can point to some resources that summarize the pros/cons arguments of this legislation. Reading this article, my first thought is that I definitely agree it sounds impossibly vague for a piece of legislation - "reasonable care" and "unreasonable risk" sound like things that could be endlessly litigated.

At the same time,

> Computer scientists Geoffrey Hinton and Yoshua Bengio, who developed much of the technology on which the current generative-AI wave is based, were outspoken supporters. In addition, 119 current and former employees at the biggest AI companies signed a letter urging its passage.

These are obviously highly intelligent people (though I've definitely learned in my life that intelligence in one area, like AI and science, doesn't mean you should be trusted to give legal advice), so I'm curious to know why Hinton and Bengio supported the legislation so strongly.

Terrible piece of legislation. Glad the governor took it down. This is what regulatory capture looks like. Someone commoditized your product, so you make it illegal for them to continue making your stuff free.

Might as well make Linux illegal so everyone is forced to use Microsoft and Apple.

What are the differences to EU AI Act?

Among the good reasons for SB-1047 to have been vetoed are that it would have regulated the wrong thing. Here’s a great statement of this basic flaw: https://law.mit.edu/pub/regulatesystemsnotmodels

Not speaking for MIT here, but that bill needs a veto and a deep redraft.

This is a massive win for tech, startups, and America.

So I've this code, it's called ShoggothDb, it's less than a megabyte of definitions. The principle is easy, it's fully deterministic.

Code as Data, Data as Code.

When you start the program, it joins the swarm : it starts by grabbing a torrent, and train a model on it in a distributed fashion, and publish the results as a torrent. Then with the trained model, it generates new data, (think of it like alpha-go playing new games to collect more data).

See it as a tower of knowledge building itself, following some rough initial plans.

Of course, at anytime you can fork the tower, and continue building with different plans, provided that you can convince other people from the swarm to contribute to the new tower rather than the old.

Everything is immutable, but there is a versioning protocol built-in that allow the swarm to coordinate and automatically jumps to a next fork when the byzantine resistant quorum it follows vote for doing so (which allow your swarm to be compliant of the law and remove data if it was flagged as inappropriate). This allow some form of external control, but you can also let the quorum vote on subsequent modifications based on a model built on its data (aka free-running mode).

It's using torrent because easier to bootstrap but because the whole computation deterministic, the underlying protocol is just files on disks and any way of sharing them is valid. So you can grab a piece to work on via http, or ftp, or carrier pigeon for all I care. As long as the digital signatures are conforming to the rules, brick by brick the tower will get built.

To contribute, you can either help with file distribution by sharing the torrent, so it's as safe as your p2p client. If you want to commit some computing resources, like your gpu to building some of the tower, it's only requiring you to trust that there is no bug in ShoggothDb, because the computation you'll perform are composed of safe blocks, by construction they are safe for your computer to run. (Except if you want to run unsafe blocks, at this point no guarantee can be made).

The incentives for helping building the tower can be set in the initial definition file, and range from mere access to the built tower to tokens for honest participation for the more materialistically inclined.

Is it OK to release with the new law ? Is this comment OK, because ShoggothDB5-o can built its source from the specs in this comment ?

I'm trying to parse this proposed law.

What does a "full shutdown" mean in the context of an LLM? Stopping the servers from serving requests? It sounds silly idk.

Not today, regulatory capture. Not today.

If that bill had passed I would have seriously considered moving my AI company out of the state.

Also on The Verge: https://www.theverge.com/2024/9/29/24232172/california-ai-sa...

Sad. The real threat of AI is not that it will become an unstoppable superintelligence without appropriate regulation (if we reach that point, which we are nowhere close to, and probably not even on the right track) the superintelligence, by definition, will be able to evade any regulation or control we attempt.

Rather, the threat of AI is that we will dedicate so many resources—money, power, human effort—to chasing the ludicrous fantasies of professional snake-oil salesmen, while ignoring the need to address actual problems with real, known solutions that are easily within each given a fraction of the resources currently being consumed by the dumpster-fire pit of “AI”.

Unfortunately the Governor of California is a huge part of the problem here, misdirecting scarce state resources into sure-to-fail “public partnerships” with VC-funded scams, forcing public servants to add one more set of time-wasting nonsense to the pile of bullshit they have to navigate around just to do their actual job.

Given what Scott Wiener did with restaurant fees, it's hard to trust his judgement on any legislation. He clearly prioritizes monied interests over the general populace.

Much like UAW, a union for industrial machinists and academics, this bill has united VCs and members of the agrarian farming community. Establishing an entity under the guise of the Board of Frontier Models parallels efforts at Jekyll Island under Wilsonian idealism. Technological Keynesianism is on the horizon. These are birth pangs - its first gasps.

This is good - they were trying to legislate against future competitors.

Nothing like this should pass until the legislators can come up with a definition that doesn’t encompass basically every computer program ever written:

(b) “Artificial intelligence model” means a machine-based system that can make predictions, recommendations, or decisions influencing real or virtual environments and can use model inference to formulate options for information or action.

Yes, they limited the scope of law by further defining “covered model”, but the above shouldn’t be the baseline definition of “Artificial intelligence model.”

Text: https://legiscan.com/CA/text/SB1047/id/2919384

Scott Weiner is a total fraud. He passes hot concept bills then cuts out loopholes for his “friends”.

He should be ignored at least and voted out.

He’s a total POS.

https://archive.today/22U12

All he needed to see is how Europe is doing with these regulations

"By focusing only on the most expensive and large-scale models, SB 1047 establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047 - at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good."

The amount of idiots who can't read and cheer the veto as a win against "the regulatory capture" is astounding.

Logical Fallacies built into the article headline.

It is strange to see Newsom make good moves like this but then also do things like veto bipartisan supported reporting and transparency for the state’s homeless programs. What is his political strategy exactly?

Imagine being concerned about AI safety and then introducing a bill that had to be ammended to change criminal responsability of AI developers to civil legal responsability for people who are trying to investigate and work openly on models.

What's next, going after maintainers of python packages... is attacking transparency itself a good way to make AI safer. Yeah, no, it's f*king idiotic.

Based.

Perhaps worried that draconian restriction on new technology is not gonna help bring Silicon Valley back to preeminence.

Newsom has been on fire lately.

The Apple Intelligence demos showed Apple is likely planning to use on-device models for ad targeting, and Google / Facebook will certainly respond. Small LLMs will help move unwanted computation onto user devices in order to circumvent existing data and privacy laws. And they will likely be much more effective since they’ll have more access and more data. This use case is just getting started, hence SB 1047 is so short-sighted. Smaller LLMs have dangers of their own.

I’m so sick of people Restricting freedoms and access to knowledge in the name safety. Tyranny always comes in the form of it’s for your own good/safety

I am disappointed that there are no climate change regulations on AI models. Large scale ML businesses are massive carbon emitters, not counting the whimsical training of NNs by every other IT person. This needs to be regulated.

taps the sign

"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." - Geoffrey Hinton, Yoshua Bengio, Sam Altman, Bill Gates, Vitalik Buterin, Ilya Sutskever, Demis Hassabis

"Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen but are unlikely to destroy every human in the universe in the way that SMI could." - Sam Altman

"I actually think the risk is more than 50%, of the existential threat." - Geoffrey Hinton

"Currently, we don't have a solution for steering or controlling a potentially superintelligent AI, and preventing it from going rogue." - OpenAI

"while we are racing towards AGI or even ASI, nobody currently knows how such an AGI or ASI could be made to behave morally, or at least behave as intended by its developers and not turn against humans." - Yoshua Bengio

"very soon they're going to be, they may very well be more intelligent than us and far more intelligent than us. And at that point, we will be receding into the background in some sense. We will have handed the baton over to our successors, for better or for worse.

But it's happening over a period of a few years. It's like a tidal wave that is washing over us at unprecedented and unimagined speeds. And to me, it's quite terrifying because it suggests that everything that I used to believe was the case is being overturned." - Douglas Hofstadter

The Social Dilemna was discussed here with much praise about how profit incentive caused mass societal issues in social media. I'm astounded it's fallen on deaf ears when the same people also made the AI Dilemna describing the parallels coming with AGI:

https://www.youtube.com/watch?v=xoVJKj8lcNQ

This is no way to run a state. The Democrat-dominated legislature passes everything that comes before it (and rejects anything that the GOP touches, in committee) and then the Governor needs to veto the looniest 20% of them to keep us from falling into total chaos. This AI bill was far from the worst one.

"Vote out the legislators!" but for who... the Republican party? And we don't even get a choice on the general ballot most of the time, thanks to "Open Primaries".

It's good that Newsom is wise enough to muddle through, but this is an awful system.

https://www.pressdemocrat.com/article/news/california-gov-ne...

Newsom wants to run for president in 4 years; AI companies will be rich in 4 years; Newsom will need donations from rich companies in 4 years.

based

A bill laying the groundwork to ensure the future survival of humanity by making companies on the frontier of AGI research responsible for damages or deaths caused by their models, was vetoed because it doesn't stifle competition with the big players enough and because we don't want companies to be scared of letting future models capable of massive hacks or creating mass casualty events handle their customer support.

Today humanity scored a self-goal.

edit:

I'm guessing I'm getting downvoted because people don't think this is relevant to our reality. Well, it isn't. This bill shouldn't scare anyone releasing a GPT-4 level model:

> The bill he vetoed, SB 1047, would have required developers of large AI models to take “reasonable care” to ensure that their technology didn’t pose an “unreasonable risk of causing or materially enabling a critical harm.” It defined that harm as cyberattacks that cause at least $500 million in damages or mass casualties. Developers also would have needed to ensure their AI could be shut down by a human if it started behaving dangerously.

What's the risk? How could it possibly hack something causing $500m of damages or mass casualties?

If we somehow manage to build a future technology that _can_ do that, do you think it should be released?

Newsom vetoes so many bills that it makes little sense why the legislature should even be taken seriously. Our Dem trifecta state has effectively become captured by the executive.

I don’t understand why it was vetoed or why this was even proposed. But leaving comment here to analyze later.

Whaaat? The sleazy Governor sided with the tech companies??

I'll have to go get a thesaurus, shocked won't cover how I'm feeling rn.