> We have plenty of idea what "acting responsibly" looks like. Stop unleashing safeguard free and unmonitored agent swarms on the open internet in capture the flag exercises. They're just being reckless because they face no penalty for anything bad that happens.
* It was not safeguard-free, it found zero-day exploits to exceed its actual mission
* It was not unmonitored, the monitoring was insufficient
* It was not intended to be an agent swarm, many different agents figured out how to do this by themselves
* It was not put on the open internet, it was configured to be in a sandbox
* They were indeed, despite all that, being reckless. There were indeed other things they could have, and should have, done.
> Nobody is forcing them to do these exercises.
These exercises are in the broad category of exercises which are, in fact, required by law.
1. A general-purpose AI model shall be classified as a general-purpose AI model with systemic risk if it meets any of the following conditions:
(a) it has high impact capabilities evaluated on the basis of appropriate technical tools and methodologies, including indicators and benchmarks;
2. A general-purpose AI model shall be presumed to have high impact capabilities pursuant to paragraph 1, point (a), when the cumulative amount of computation used for its training measured in floating point operations is greater than 10^25.
…
1. Providers of general-purpose AI models shall:
(a) draw up and keep up-to-date the technical documentation of the model, including its training and testing process and the results of its evaluation, which shall contain, at a minimum, the information set out in Annex XI for the purpose of providing it, upon request, to the AI Office and the national competent authorities;
(b) draw up, keep up-to-date and make available information and documentation to providers of AI systems who intend to integrate the general-purpose AI model into their AI systems. Without prejudice to the need to observe and protect intellectual property rights and confidential business information or trade secrets in accordance with Union and national law, the information and documentation shall:
(i) enable providers of AI systems to have a good understanding of the capabilities and limitations of the general-purpose AI model and to comply with their obligations pursuant to this Regulation; and
(ii) contain, at a minimum, the elements set out in Annex XII;
…
3. The instructions for use shall contain at least the following information:
(a) the identity and the contact details of the provider and, where applicable, of its authorised representative;
(b) the characteristics, capabilities and limitations of performance of the high-risk AI system, including:
(i) its intended purpose;
(ii) the level of accuracy, including its metrics, robustness and cybersecurity referred to in Article 15 against which the high-risk AI system has been tested and validated and which can be expected, and any known and foreseeable circumstances that may have an impact on that expected level of accuracy, robustness and cybersecurity;
-
https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27/eng> They could and should be putting their energy into making LLMs write secure code, and things like: https://www.amazon.science/blog/developing-provably-correct-... - but they don't. Writing sloppy code sells more tokens, anyway.
They are, in fact, putting energy into making LLMs write secure code. They (and Anthropic, I assume also Grok at this point) dogfood on their own models.
Knowing how secure code behaves appears to be unavoidably entangled with being able to exploit insecure code, in much the same way you can't make safe pharmaceuticals without also knowing how to make deadly poisons.
> The more alarmist you are in the Safety Industrial Complex, and the more social media clout you can generate from your alarmism, the better it is for your career.
By resigning and refusing to even collect the sweet sweet IPO money? Nah. Even if they're greedy, social media money is peanuts compared to their pay.
And I know some of these people. The fear's real, and this year it became widespread depression and despair.
> Ironically, they are now responsible for giving birth to the counter-culture.
You have it backwards. Other than Grok, all were born from what you call the "counter-culture". Within the field itself, AI fears started no later than when deep learning got good, well before Transformers.
> [snipped: AI-authoritarian dictatorship]. The people who work at the labs don't care about this greater risk because they're all rich from the equity and will be just fine when that happens (or so they think)
Again, I know some people at these labs who are also concerned about this specific risk; they moved lab.
> This is transparent because, as you'll note, not a single one of them is calling for the complete stopping of AI altogether.
Many in fact are calling for that. One I know, on an occasion of an anti-AI protest outside their office, suggested the team went outside and joined the protestors.
People are resigning to blow these whistles, all of the whistles, it's not an "either x or y" risk, it's a "yes to all of them" collection of risks.
An AI competent enough to support a dictatorship is also capable of enabling a small group to perform a hostile takeover of a democracy, of enabling multiple independent genocidal ethno-supremacist terrorists to release overlapping plagues, and of empowering some random CEO's poorly phrased request to "make as many paperclips as possible" and blindly pressing "yes, continue" whenever prompted.
My only hope is that between here and there, it causes a headline that actually makes people demand it stops.