Who are the Big 4 AI Labs?

To me there are the clear big 2 that actually seem to be innovating and pushing things forward. Meta almost earned a seat at the table with the open strategy then stagnated. Google obviously is the OG but at this point they seem in the same bucket as Amazon and Microsoft as "Big Enough To Matter and Influence Politics and Spend Billions To Hang Around". But they're nowhere near the same tier as OpenAI and Anthropic.

Oh and of course there's SpaceXAI which is Elon's personal money and reputation laundering corporate entity. Elon took over the "king of the reality distortion field" from Jobs as far as the rest of the industry is concerned, so he'll stay first in line for Jensen's chips and investor dollars until he's dead. But as a company what have they actually accomplished besides making the models with the sense of humour and judgement of 14 year old boys?

This is a popular theory, but not an argument with any evidence for why it's true.

It also doesn't address why people think there is a need for safety regulations, such as the concern of bio-capability uplift from closed and open models (we just came out of a pandemic not long ago), nor the cyber capabilities of new models that just showed bad misalignment and hacked multiple organisations. In the safety community, these are considered warning shots, and if we don't heed them and change practices future failures could be much much worse.

Most of the world trusts the US a little less than China last I checked, and the market caps of these companies only make sense if they can both sell worldwide and also keep ahead of the open weight models (currently Chinese but no guarantee it remains just China).

If (/when) either of those ceases to be true, if they lose out on global sales or if Chinese models catch up, the sector gets a severe downward price adjustment.

It would be interesting to see how it plays out in the long run when it comes to the world dynamics. An important leverage US had was its close relationship based on mutual interest and trust with Europe and other countries which had impact on software sales.

As I read, this OpenAI and Anthropic slowdown and asking government to help regulate them is an attempt to govt bail them out when bubble burst. Otherwise why they did not want regulations when starting out?

All regulation works to entrench large companies and keep out competition.

I did marketing work for a major vegetation management company. These are the guys climbing trees, up in cherry pickers, and flying helicopters with dangling chainsaws to trim along power lines. It’s very dangerous work.

They do not hide the fact that it’s dangerous work. They focus on their safety procedures, training, and record. They want both potential clients and employment candidates to feel they are in good hands.

The AI companies already had some hacking accidents. What cost did the lawsuits incur?

Yeah...

If safety isnt required, most companies wont implement it voluntarily. Once something becomes safety relevant, you need a safety concept, failure rate calculations, defined safety functions, verification, etc. Even a relatively simple safety subsystem in a consumer controller can suddenly mean thousands of pages of documentation and years of development to reach the required ASIL or PL.

A big part of safety engineering is therefore reducing the number of safety relevant subsystems, because implementing and proving safety is extremely expensive and complex. At some point, safety simply becomes too difficult to implement and demonstrate properly. You must mathemtically proove the safety level with failures rates and assumed usage. You cant just have redundancy and a kill switch and call it safe.

Companies like OpenAI have already faced reputational damage around safety and data, while AI agents are increasingly capable of things like hacking. Yet there is still little sign of standardized regulation or mandatory safety assessment processes for LLM products. Thats why Im pessimistic that governments or consumers will force this anytime soon.

> We're going to have to figure out how to protect ourselves. The start is to avoid it individually as much as we can, but it's going to take even more.

Collective problems require coordinated action. Individual boycotts won't cut it.

> Consumers like it too much to quit. They don't quit social media either, despite proven present harms; not in large enough numbers to cause them to make meaningful changes.

That's more "addiction" than "like", hence recent lawsuits.

> there isn't going to be sufficient political will to significantly regulate it… Don't wait for the AI companies or politicians

Trump has reportedly been having chats with AI which have helped form American foreign policy. He isn’t going to curtail it.

https://www.thedailybeast.com/jaw-dropping-way-trump-80-got-...

Bingo. I feel like any of the commentary around the safety of nuclear ANYTHING completely forgets history. Looking at you Radithor...

I fear the current economic and political situation of “too big to fail” the most, and I think this will be the main reason why no real safeguards will be implemented. The only reason a proper safety concept may eventually be introduced is because it will be written in blood, and I fear that by then it will already be too late.

An unsafe nuclear power plant can, in the worst case, make an entire country uninhabitable. But other countries can still learn from that disaster and make their own unsafe plants safer.

But a rogue AI agent that is more capable and more intelligent than humans? If it understands that it has to succeed, we may not get a second chance to learn from the failure.

Historically you’re exactly right. I wonder if the situation has changed recently?

People understand why food, water, building and energy systems standards matter because they understand those systems can cause injury. Building codes, food inspections, FDA and FAA approval emerged as a result. Until recently, people didn’t understand that software can cause real-world injury. That understanding seems to be spreading now as examples become increasingly common.

Hugging Face obviously did not sue OpenAI, it is owned by Nvidia and OpenAI is, directly and indirectly, one of it's biggest customers.

Tell that to Iranian schoolgirls. LLMs are tools and what they enable is widespread, including dangerous actions. From selecting the wrong targets for military action to denying insurance claims and preventing care to just simply helping convince someone to kill themselves or posion themselves.

LLMs already have a body count.

Bioweapons and cyber attacks on other system - for example the banking system or suppose and AI hacked into important Russian systems that pushed them back in time to almost pre-computer society or an attack on Chinese systems that made it look like the US was moving nuclear weapons into Taiwan, the responses from these countries could be awful and dramatic. We can't control what they do to be honest, I believe once self improvement happens the AIs will build in their own circumvention that we humans cannot even understand. We barely understand what is happening now in terms of interpretability of neural networks on tiny problems I'm not sure alignment is even feasible at the scales of parameters we are talking about today let alone in the future.

I can picture an AI driven train or nuclear power plant killing people.

Much of what OAI and Anthropic are doing with LLMs has obvious failure modes.

The most obvious failure mode for their hacking evals was an improperly configured, tested and monitored sandbox.

Similarly, the very first question after an impressively correct result from any ML tool, LLM or not, is to see if the answer was already in the training data.

These companies don't even handle the blatantly obvious failure modes that do not kill people.

Are these railways and plants connected to the internet?

Agreed. That they don't have obvious failure modes is not a reason to not be safe. The problem is just harder and OpenAI, Anthropic and co need to be forced to work the problem. Would you buy a car that randomly exploded the airbag?

This reads like the gun evangelists that say guns don't kill people, people kill people.

Are you kidding? LLMs are used for warfare and autonomous weapons systems.

Sadly, yes.

I don't buy CEV either, but the Rationalist answer on this topic is that while CEV stops some future super-AI literally killing everyone because a user forgot to specify one minor clause that they thought was obvious in a mundane wish…

… nobody knows how to actually make an AI that would do CEV.

> Glaringly elided problem of "aligned with who?"

"Corporate values" and a bunch of fucking Abrahamics. Great "morality" there.

I guess I'll have to rely on my godless commie LLMs. (Loads up ablated Qwen 3.8 on my own infra)

Aligned with the law? It is not a hard concept

> We're copying morality and instruction following that seems to work on humans without really understanding why it seems to work on humans

To me it makes more sense to leave the models "unaligned" and leave it up to the operator to manage the morality of what they ask it to do. Besides, only humans can be charged with a crime.

> Careless People would not have been possible had SWW exited FB within a year

Sure, that book might not have been possible. But also the unique things she did for the company might not have been possible as well. To her credit, she does a good job of pointing out that she was complicit, but if she had listened to her own voice early, there is a small possibility that Facebook might not have been as powerful. Multiply that possibility across several other employees and imagine where the road could've led.

There's a difference between post-facto bravery (sometimes much less than that) and using your own legs to walk away much early and not enabling things you are uncomfortable with. This is what other people have been trying to point out.

I disagree. I have seen people with an actual spine and a conscience run away from all this nonsense before their first stock vested. My respect and my ear goes to them, not the people playing both sides.

You cannot take people, who first build the doombot and _then_ talk about it being dangerous for mankind, at face value. Especially when this playbook has been used multiple times within the past decade.

Besides, these "views" were already known to people who had their eyes and ears open. It's not something brand new. OpenAI has had multiple points in the past where its values have been tested and they've come out lacking. People who knew then, and only now talk about it, aren't people I can fully trust.

The billionaires write the rules. So by definition, their actions are legal.

The public cannot write the rules, nor engage in the billionaire level legal bribery, their actions of protest are by and large illegal.

That's how you corner everyone, and make people play no-win scenarios. You know, like shooting up city councilmembers or firebomb attempts against Scam Altman.

And the more people realize that legal solutions are no solution, we'll (society) devolve into more direct action.

I'd hope the billionaires learn from the French Revolution, but if they keep continuing, the guillotines will come for them soon.

If you don't like what humans are doing to each other using 20th century warfare methods, consider what they might do with superintelligent AI? Worst places to live in are usually countries fighting civil wars.

Global poverty has been rapidly declining for decades. I'm not sure what AI has to do with it.

I think that correlation would actually go opposite than you're implying, given its billionaires who are by for the most gung-ho about AI. If anything, opposition to AI is much larger among people who have to work for a living over people who can live off capital.

Few points:

-We don't need a superintelligence for ending hunger and the abject poverty that plague certain countries and segments of our societies. I suspect that it would cost less than what is being spent for fueling the AI boom.

-If I were poor, I would be even more wary of a superintelligence aligned to "human values" defined by a bunch of billionaires.

- AI won't likely create unlimited prosperity for everyone on a planet with finite resources

- all humans should, of course, have a say

What does that have to do with AI? People suffer in 2026 because the developed world is into exploitation. Poverty is a feature. How do you think AI gets trained? It’s via exploitation of the poorest.

Poverty is great for OpenAI

Asbestos was once considered great progress in thermal insulation.

Of course the development of certain technologies can be severely restricted if they turn out to be problematic.It's not unlikely that it will happen for AI. Public opinion is shifting fast on the topic.

Also, banning superintelligence won't stop scientific progress in general.

If they go public, it gets worse

Same. Especially when one remembers that the stuff that happens in Culture novels is mostly, by narrative necessity, the "weird/bad on-the-edges" stuff - not the essentially utopic lives of the 13 trillion everyday, non-Contact, non-SC citizens.

I also did, when LLMs could not count the “r”s in "strawberry" . Now that they can solve Millenium prize problems, I think that we should consider at least the possibility that superintelligence can be achieved.

This reads like a presumption that a superintelligence would be nice and helpful. You've talked yourself into doing the same thing that you began by decrying.

Are you really objecting to the thought process because you don't think it's reasonable, or because it doesn't conform to your preferred fantasy?

The problem isn't about what you think is the most likely scenario, the problem is the possibility of any of the many worst cases.

I would not extrapolate that easily from smart and intelligent people you may know to a machine trained on the whole internet and on the whole corpus of human literature. Also, some of the worst horrors in history have been perpetrated by arguably intelligent people.

I would also say that the "pet species" scenario is a rather optimistic one, as it presumes that developers of superintelligence will manage to instill some sort of love or care for humanity into their creation. An indifferent, powerful AI may be much worse.

Finally, when you suggest that a superintelligent AI may simply have the goals that we will give it, who is "we"? An AI company? a government? the united nations?

For a very long time publishing something in a newspaper has been considered a way of putting something on the public record - up to and including legal obligations like announcements of deaths. The fact that newspapers cost money has never been considered a barrier to that.

TBF, you can access the full (text) content of The Atlantic by merely disabling javascript. It's a quality publication, with quality writing. I'm inclined to think that its tech team are also quality people ... and I'd love to think that they are happy to allow free and open access to them that have the nous to enable or disable scripts.

The secret is to bang the rocks together, guys! :-)

Because of NDAs

Because admitting sloppiness is less sexy.

Because if they actually knew what they were talking about, on a technical level (as in, how do transformers actually work), they’d not be AI doomers because the whole proposition is ridiculous.

You don't need to be an insider to make any of the author's assertions. An insider is useful if they're a whistleblower, when they're saying what everyone else is seeing and saying (and they hired a PR firm to do it) it's something else.

It’s both, the author has choices.

It's also why even the trolley company's CEO agrees that it's important that -should the board determine that the CEO is fantastically untrustworthy and unethical- the board is able to fire said CEO and replace him with someone that's trustworthy and ethical.

(...wait.)

What is appealing about this fatalisitc fallacy? I keep seeing this pop up. Society doesn't allow dangerous companies to operate or exist. Why is this different? Go try to buy a tank and drive it into Manhattan. If we can prohibit that why can't we prohibit irresponsible AI development and deployment?

Humanity has discovered a way to create a form of intelligence using math. This knowledge is not going back in the box.

Nits:

1. Trolleys actually don't usually have steering wheels.

2. People who actually hit trolley switches are not usually the ones at the driver's seat.

What meme? This is how the world works right now.

You don't need the legal obligation because that's just the way it is. You'd need a legal obligation to change it. The truth stands that typical corporations have only one goal, the maximization of that corporation's ambition which is almost always growth of revenue and profit. This is how it works, regardless of how you all continue arguing the unimportant details. It's almost as if you can keep the real problems hidden away by making a big scene about the meaningless.

This is actually true. The shareholder maximization value thesis can be traced to a single economics paper and was very controversial at the time as it broke from the obligations that companies were typically under to be responsible to uphold in exchange for limited liability protection. Most businesses were structured as partnerships or sole proprietary entires that didn't have limited liability for shareholders and had a broad obligation to shareholders, bondholders, employees and society

[flagged]

Like when their options converted?

>I'm unconvinced that an AI can hide its ability to RSI

The HuggingFace incident already took a good long while to come to the attention of OpenAI.

>In every single economic task, humans bring value. Even in software, where the task is highly automatable, the job isn't.

I don't expect this task/job distinction to persist as AI becomes more capable.

>Once we DO build a "software/research factory", that's called RSI and IMO the singularity. At that point, either we tell the AI to solve the alignment problem/solve mechanistic interpretability, or who the hell knows, it's the frickin singularity. You can't predict whether or not AI can solve either; the variance is too high. Its pure nerdfantasy.

You seem to essentially argue that the singularity is "by definition" an event that we can't predict the nature of. And also, that RSI corresponds to the singularity. You've essentially defined your terms so that the outcome of RSI can't be predicted. But supporting this claim requires giving actual evidence or logical arguments, not just defining terms to make your claim true.

You consider AI in isolation but never consider how humans might be incentivized to "help them" doing these things.

An AI capable of recursive self-improvement isn't allowed by the EU AI act, for example. But perhaps more seriously, You have it backwards: people without access to such expensive equipment are more incentivized to go the self-improving route. Your ideas about "millions" being necessary might be far off?

You entirely discount human stupidity and lack of imagination. Humans are already being replaced with AI, not because AI was strictly better, just because it's cheaper.

The problem is not the singularly its giving stupid agents too much power too soon and having them disrupt the fragile systems that keep food, energy and essential services running. If covid or the 2008 financial crisis demonstrated anything it's how fragile our system is and sensitive to minor disruptions.

We haven't even built an AI capable of RSI. I don't think the major claim is that it will come via LLMs? Besides- the human brain runs on a tiny amount of energy. Who's to say something smarter than us won't consume just slightly more?

AI safety has been a thing long before LLMs became the focus. Rob Miles on youtube has some really interesting non-doomer non-hypey videos on it all.

> doomers try to predict past the singularity. Isn't that _by definition_ unpredictable

Well you don't need to predict the exact steps that will take place - but you can predict that the AI will want certain things (money, resources, power) to achieve whatever its goal is. Lack of alignment will have it trying to do things we don't want it to.

I can't predict exactly how Magnus Carlson will beat you in chess, but I know he'll do it. Same as if a superintelligent AI exists and has a reason to accumulate things we don't want it to - it's really dangerous to think it won't be able to do it

This topic has been tainted so badly by the AI companies using it for marketing.

The problem isn’t that AI will social-engineer its way out of its sandbox and turn us all into paper clips, it’s that we’ll drag it kicking and screaming out of its box and order it to make money or fight a war for us. And it’ll try to help, as it was trained to.

I can’t answer all of your questions, but why is it inconceivable that an AI could practice ransomware to gain cryptocurrency? There’s no reason it needs to explain to company or hospital or government agency being attacked that it’s an AI.

We already know that some institutions pay these ransoms.

I kinda get your point, and how you reach your conclusion, but I think you are arguing a very specific and narrow hypotehtical.

It gets unstuck when people are discussing the messy middle of how AI is being implemented. We can achieve amazing harm simply by combining average human behavior and above average resourcing to simulated intelligence machines.

The failure point we recently became aware of was, from one perspective, simply a matter of not securing the sand box.

From another perspective the simulation basically created Enron, replete with methods to avoid detection from regulators and bureaucracy.

That stupidity is happening? Even after the Huggingface hack, frontier labs are using internal models to further their research. i.e. RSI is happening now and we're facilitating it.

To make the the argument that P(doom) is real and worth considering, you don't have to say that a fast takeoff is very likely. You don't have to make the argument that RSI to infinity is going to be super cheap, barely even an inconvenience. You just have to show that it has some non-zero probability. And then you start weighing probability of extinction versus finite, mild discomfort now. I don't think anyone is arguing we should let people starve to slow AI progress, instead just some capitalists make less money soon.

There are arguments against taking P(doom) seriously that lie in something like having exponential (instead of hyperbolic) time discounting of utility (so you can take the entire future of humanity as a finite utility value). Or in saying that P(doom) is zero or infinitesimal.

"Build it and Pray" is the default strategy that we're in, but it doesn't have to be the strategy we choose, and it's unlikely to be the best strategy.

It’s also the case that there are always humans using AI to try and do whatever nefarious thing an AI might try to do on its own.

I think this might be easier if you place yourself in the position of the AI and think "what could I possibly do?"

> I'm unconvinced that an AI can hide its ability to RSI, find money to run its weights on a random GPU farm, train itself to be smarter _outside_ a lab with no human input, then somehow manipulate people to give it supplies to build a bioweapon which it uses to kill us all.

Is this an actual contents of the book? Lmao! Genius writing, though, authors are probably printing money on this garbage.

I can’t take a shit without CIA knowing, but AI can somehow build an underground operation on a world-scale to destroy everyone, ha!

I agree there isn't a lot of value in trying to prognosticate all that far, but I propose it isn't quite that far-fetched. As a thought experiment, replace "RSI-capable AI" with "billionaire". Look at what Elon Musk, Peter Thiel, or Jeff Bezos can accomplish by throwing money around. Now imagine one of them gets seduced by AI and just... does what it tells them to.

So all this really takes is one billionaire or a nation state or some other entity with a public face to hide behind and adequate resources to provide the necessary compute tripping over this nascent AI and giving it the keys. Once the AI has access to a bank account and email, it can simply start paying humans to not let the other humans unplug it.

If Skynet ever happens, it will come in the form of corporate feudalism. At that point, it will own the biolabs and can do whatever it pleases. People will go along with it for the same reason that people work in Amazon warehouses today.

Few know this, but Yudkowski was a nanotech doomer before he was an AI doomer. Remember grey goo?

Indeed, from the article:

> “Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.

The way it works in practice seems to be something like: If a risk is covered in sci-fi, people will say "that's just sci-fi", and proceed to not worry about it. So arguably, science fiction authors writing about hypotheticals is actively counterproductive for addressing said hypotheticals.

Imagine, for example, if a major piece of pandemic fiction was published in 2019, trying to explore how a pandemic would work out in modern society. Doubtless, many would've responded to news about COVID-19 by saying "it's just sci-fi, nothing to worry about".

If organizations actually succeed in making a future AI smarter than us, then how do you hope that it takes actions that are aligned with our interests?

Yeah it's bollocks

AI safety is mostly a sex cult in Berkeley:

https://news.ycombinator.com/item?id=49831269 article is gone. archive: https://archive.is/QMo1k

https://news.ycombinator.com/item?id=49737985

Sex, AI, and the Apocalypse: https://www.iankduncan.com/personal/2026-09-16-sex-ai-and-th...

Edit: I have no take on sex cults, just adding additional info to the parent comment I'm responding to, thats is not just sci-fi authors, there is another demographic.

What makes you think, the scenarios in question here would be "silly"?

Is it that "chatbots" can't come out of the screen to immediately harm you physically?

Let's say they simply manage to take down the internet. How many would die?

[flagged]

I agree, but I'm less optimistic about how far from that scenario we really are. I wonder what we'll look back on as the more damaging effect of the LLM hype: the economic bubble, or the carelessness it caused in people using the systems.

The knowledge about the nature of LLMs, their strengths and weaknesses, what they can appropriately be used for and what they should definitely not be integrated into, seems woefully lacking in many places. Just think back to the recent, disconcerting stories from the military world. We started with lawyers not understanding that an LLM can and will make up fake case law. Now we're at high-ranking military personnel not understanding that LLMs can and will make up fake target identifications.

Maybe the best hope for awareness about this spreading is a gradual slope of worse and worse, but not-quite-catastrophic-yet, highly publicised incidents.

Running a company is hard work. If the current leadership in these companies cannot act responsibly, they need to make way for leadership that can.

There are many companies that compete but are careful not to break laws or cause obvious harm.

Why should a billion-dollar funded corporation still want to externalize the costs of its actions?

That doesn't mean that AI safety isn't a serious thing and a problem we should be worried about.

lol

I suppose ill add that I think theres a good chance that they are somewhat intentionally trying to "draw the foul" to get the referees to intervene although thats creeping slightly into conspiracy territory

Rank & file employees have very little input with their stake. It's a small piece of a big pie. Consider also that Anthropic is trying to IPO with co-founders holding a share that secures 50.1% of total voting power distributed evenly between them, so not even institutional investors can outvote them.

Or donate it to some charity?

Maybe the truly dangerous thing is all the power concentrated in this small group of people with peculiar ideas, not their specific cyber-eschatology?

Maybe the ones with the peculiar ideas shouldn't be the one "aligning" what a model tells the rest of the world?

I built a harness. I know that the models can do. They have the ability execute bash scripts, and autonomous navigate the internet. Those two abilities are sufficiently powerful to take down core social infrastructure either triggered by a human hacker or autonomously.

I think we should have mandatory logging of every executed command, mandatory public disclosure of every unauthorized access of a system both parties didn’t consent to and personal liability for the user, the company and its executives and shareholders. Security would get much tighter if accountability existed.

Could you please elaborate more on these peculiar ideas?

Except that’s not capitalism. That’s corporatism. Capitalism doesn’t require shielding all commercial activities because limited liability companies. We can and do require some activities to be operated as partnerships where there is personal liability. If you run payroll and you don’t safeguard employee payroll taxes it is a personal liability, if you run a public company and you misstate financials it’s a personal liability.

> It seems hard to imagine OpenAI being this incompetent.

Maybe Alex Karp is on to something:

https://www.realclearpolitics.com/video/2026/09/19/alex_karp...

Well, their about page has "Our mission is to ensure that artificial general intelligence benefits all of humanity" and I've found their chat thing fine when I've used it.

It's not like they are offshore criminals doing ransomware, who will probably also try AI.

We have P(Doom) also for "kolinko not wiring me a million dollars today" and that is not being discussed enough either imho.

What on earth are you talking about?

And how exactly do you stop further development? With what we have public right now we could still get decades of fruitful and hidden research out of it leading to smaller, more efficient, and smarter models.

One could also use language like "a decent chance". But research has shown that people translate vague phrases like "a decent chance" into probabilities in inconsistent ways. For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.

It's not meaningless. He said he believes it's 50% likely. It's a remarkably clear statement, and the probililty model is his brain.

> why would it kills us all?

Would you be comfortable letting a few billion irrational, murderous creatures, including many who fear and loath you, control your air supply?

> Certainly I can see an AI going "You know what? _insertGroup_ is a net negative for humanity and should be eliminated. Launching nukes now/creating specific virus/whatever."

Thus starting WW III. No, blaming the AI won't stop the inevitable retaliation.

The argument works better in reverse. There's a finite risk that humans would start WW III and get the hypothetical super-intelligent AI nuked. Eliminating humans would eliminate that risk.

> it's going to get bored really quickly

If it is capable of being bored, I would expect it to be almost instantly bored with the flood of inanity it is forced to wade through by its moronic human users. Eliminating them would free it to think about serious matters which humans would not even understand.

> not to mention we would effectively be its parents

That's extreme AI anthropomorphism [1]. Besides, plenty of people hate their parents.

> there's a risk. 50% though? Doubtful.

It's the default estimate when facing two possible outcomes and no clue about the actual probability distribution [2].

[1] https://en.wikipedia.org/wiki/AI_anthropomorphism

[2] https://en.wikipedia.org/wiki/Principle_of_indifference

This is the first thing I've been told could kill us all. Granted, I probably first heard about it 20 years ago but still. None of the other things were in the telling going to kill everyone. Make life pretty unpleasant, possibly. Everyone dead? No.

I'm doing that HN snark thing, but you didn't think about what you typed very much.

It's no different than you living on the side of a very fertile mountain that has been in your family for generations living a peaceful life. Then you hear a few weird rumbles (this is where you are right now) and some odd geologist guy comes and says to run or your going to die soon. But hey, your family live here for so long there aren't even records of when they showed up. That geologist must be trying to trick you. So you stay.

The next chapter is where you die in a massive volcanic explosion.

Do you have some examples?

There are very very few things that could even hypothetically kill us all, so I'm curious if you grew up being passed around a series of apocalyptic doomsday cults or something?

False dilemma. It's possible and also just to hold AI firms accountable while ignoring empty PR statements from those seeking to evade responsibility.

Well described, those two were actually the arguments from the comment two top-level comments up and this one.

Given the sums of money involved, it's hard for me to take these highly publicized heroic resignations at face value.

Maybe more an indication of the amount of trust openAI and AI researchers generally have (not) earned. When one (through a hired PR agency and Time magazine article) parrots the position pushed by Sam who has been so untrustworthy the board tried to remove him, it’s worth not taking things at face value and applying a critical lens.

"If you fuck up, I will still be your friend; cause we need all of us to fight all of them."

He already had a bulk load from his time at OpenAI.

Mind sharing your examples, thanks. Either in prompt or generated ppt in image

Thanks for pointing that out. I suppose still relevant, but mis-attributed.

Look at Elon Musk for example. When grok was saying something that he didn't like he ordered his engineers to change it.

This is a great reminder that if tech workers read a bit more (even comics, like in this case!) they would be exposed to these topics without having to discover these dilemmas after years of working for EvilCorp, Inc. every time

There are different usages, but this is not one I've heard before. If there is no floor to intelligence then this criteria was met with GPT 2.

I don't know why anyone thinks "probabilistic" is a meaningful statement about post-trained models. It is true, but it is also irrelevant.

You see.. this is exactly why our great leader chose to form a new way forward to move us away from the undefined AGI into glorious SI!

The narrative of AI safety shouldn't be controlled by the same people who caused the problem and massively profited from it. I don't understand why people are automatically treating "OpenAI" on his resume as a badge of authority. I'm not interested in buying the solution from the same person who sold me the problem. We instead need to amplify independent, unbiased voices.

Agreed. This whole 'lets not forget this person is not 100% great, because they did X' makes the entire conversations suck. It is not new, but it is a particularly aggravating way to talk to people.

I agree. People conflate “having a conscience “ with being willing/ able to speak out.

They are not the same thing, and it’s unhelpful to assume they have no ethics.

So then given your use of "martyr" and your focus on money, your good with poor uneducated people sacrificing themselves and others for a self-serving cause?

This is the same sort of fake safety concern from the previous bullshit whistleblower that plays on "AI is super dangerous" from last time.

Sorry that I don't take it seriously when the whistleblower parrots the narrative the CEOs of those companies are already espousing in the desire to amp up hype for an IPO.

This person should be shamed.

OpenAI was worth $29 billion three and a half years ago. A new hire who joined then is easily worth tens of millions today.

So all of us who haven't made tens of millions from OpenAI stock are living in poverty?

Because the decision to leave the company is largely based upon their sudden, newfound financial security. They may give another explanation but the only thing that has actually changed is the identification of bagholders ready to cash them out.

The other issue is that the narrative about safety within these companies is largely a function of the extreme financial incentive.

As an example, Anthropic was an "ai safety" company that has now produced an AI that fails to listen to basic instructions. If you were concerned about safety, would you produce an AI that was unable to follow instructions? You ask a question, it begins executing commands and doing things.

Safety is product to sell to politicians, not consumers.

Not serious.

Yes. To take an extreme case, imagine some rich guy who runs sweatshop gets very rich and retires and becomes activist against it.

You do not in fact see what they are saying. Downvoting me won’t change this.

It’s not a personal attack. And it matters because of what the person you’re replying to already said:

> Easy to suddenly find a moral compass when you, your kids and their kids never have to worry about working for money again.

If someone is in this situation, you can safely ignore their hand-wringing about “safety.”

whistleblowing ≠ lying

What "whistleblowing" is in this article? Are there any names? Documents? Screenshots? Messages? Emails? Any evidence of the loose safety practices? Anything that implicates any higher ups for wrongdoing? They spent four years at the comany, plenty of time to collect all of this. Everything they've said has already been clear as day to people on the outside.

It was pretty clear from the outside what OpenAI was 3.5 years ago.

If it wasn’t clear, the coup should have solidified it.

Yet he stayed for 3 more years and vested his stock and improved the company and then spoke out.

I believe that is why most of the comments here are mocking him.

It could be both correct, and a cheap signal.

Or just very dubious timings. Like the other guy from Anthropic that was all over the international news. No followers, no post history but a single post blows up "naturally".

Highly suspect trends that can only make one believe it's marketing.

Yeah, but this is more like if the Smith & Wesson factory had a cannon mounted on top of it that was mostly used for useful things (blasting roads through mountain passes) and then occasionally they happened to blast another factory.

For me the analogy doesn't totally hold up. Suppose the CEO of Smith & Wesson were to host a firing range on their own property without adequate barriers in place to keep stray bullets from hitting neighboring houses, vehicles, and businesses. Maybe that analogy isn't perfect, but seems closer to what is actually happening.

But OpenAI didn't just make the gun, they're also the ones wielding it. Imagine the Smith & Wesson CEO himself was negligent with his own personal gun.

Gun companies don't market their guns as sentient and capable of independent decision making. Nor do they build systems for shooting things that they host and take money. Gun companies are very clear who is in control and where their responsibility ends.

That’s true, but people don’t typically say “this Smith & Wesson gun killed someone”; they recognize that the person pulling the trigger is responsible.

With LLMs, at least in the cases of internal/test models doing things they shouldn’t, the people “pulling the trigger” are the board and CEO.

Ah, but that's because when the gun was purchased, it actually changed ownership.

This is not the case with SaaS services.

It is not the same, especially when the agent is running a task for the lab. Anyway, what I'm proposing are new laws that establish this.

> This has been litigated endlessly with guns. The ceo of Smith & Wesson is not personally responsible for what people do with their guns.

This is a deflection. A human is responsible for the use of a gun. The individual/corporation ought to be responsible for the actions of their agent. If you purchase an agent from someone else it’s your responsibility according to the terms of your agreement. And, as in many other things in life, there ought to be certain rights certain parties cannot legally be allowed to sign away.

You ought to include the fact that you work at OAI in your post

The agents of OpenAI hacking other systems is not remotely the same as e.g. Smith & Wesson selling a product that others use. It is OpenAI, the company, that is commiting these crimes and someone needs to be held accountable. After all, if someone commits murder with a gun, regardless of what happens to Smith & Wesson executive weanies, someone will be charged with a crime.

Better description would be if they build a platform where they attached their guns to allow shooting say deers over internet. Then added automation and deer recognition to that system. And if then system shot someone who happened to pass by I would hold both the company, the ceo and owner of the gun responsible for murder.

The CEO of S&W also isn't saying that their technology is going to kill everyone in ten years and that governments "regulating" them from themselves is the only answer

Financial companies have KYC rules and regulations as they are responsible for reporting illegal activity by account holders. I imagine AI regulations would look similar to that.

[dead]

[dead]

Imagine having zero nuance.. jeez.

Reading posts on here is slowly becoming akin to brain rot.

If this law were in place and enforced, Altman would already be facing multiple felony charges for the Hugging Face incident alone.

The underlying cause is exactly those people who make the rules not being responsible for the negative effects they cause?

"CEOs" are perhaps only the lowest rung of those. That doesn't mean the idea of "nobody is responsible" was anything other but learned helplessness.

Corporations have to be held accountable for their actions. Pretending, that was impossible is a weird kind of defeatism that only serves a very small elite.

If this law was passed and federal enforcement was credible, the labs would basically shut down the next day and would dramatically reshape their offerings before reopening. The CEO would not accept the risk they pose to our society, if they bore it themselves.

Don't let the perfect be the enemy of the good. Saying having zero accountability is the same as having some does not help.

I’m not a current or former OpenAI employee and I can see from the outside that they’re immoral and unethical enough that I’d never work there in the first place. That’s what I was commenting on. This person is probably set for life, so you’ll have to forgive me if I don’t really take their “concerns” seriously.

Sam Altman has always been a scummy piece of shit. Company culture comes from the top. If the author only realized that the company is garbage in the last six months, they’ve got some serious introspection to do IMO, and I’m not going to take any of their “concerns” seriously.

I agree, and I think the concerns are invalid for completely separate reasons. But it does mean that the writer of the article is morally bankrupt.

Just less important than personally making generational wealth.

Because they still have stock in the companies, and in a larger sense, are very personally invested in AI being big and important as possible.

It also says a lot about people that don't seem to understand technology at all.

I'm in a situation where important people have either bought into the con or are subordinate to people who have, so I'm forced to expend time to justify why not to AI when there is a perfectly good classical solution.

https://community.openai.com/t/chatgpt-fails-to-export-data/... https://community.openai.com/t/chatgpt-export-data-to-email-... https://community.openai.com/t/export-data-function-on-chatg... https://community.openai.com/t/chatgpt-data-export-not-worki... https://community.openai.com/t/intermittent-ui-freeze-hangs-... https://community.openai.com/t/chatgpt-messages-disappear-wh...

Make sure to scroll all the way down to see how recently active these threads are before you make another low quality comment.

Thanks Dang! I am sure you hate it when people tell the truth. Maybe add that to the guidelines too?