stillpointlab
5 hours ago
> American frontier labs are run by unhinged lunatics who genuinely believe they're creating the Machine God, that they're fine with burning down the world to bring about AGI, and that they believe with religious fervor that they're the only ones smart enough to safely control it
I like hyperbole as much as the next guy. And I'm cynical by nature. But this just seems like a wildly pessimistic view that gives literally zero charity to anyone working at or leading a frontier lab. And the fact the author goes on to say that they "already knew this" just points to confirmation bias.
The rest of the article is actually pretty decent and encourages people to investigate and run open weight models. That is a totally reasonable position that I don't think requires the heavy-handed in-group signal that precedes it.
mrandish
4 hours ago
> I like hyperbole as much as the next guy. And I'm cynical by nature. But this just seems like a wildly pessimistic view that gives literally zero charity to anyone working at or leading a frontier lab.
Reading TFA, I too paused on that sentence, thinking "This is overstated and feels like overheated rhetoric." My reasoning though is a bit different than yours. In addition to being too extreme and absolute as stated, it also misses that it's not uncommon for startup entrepreneurs, scientists, researchers, or really, anyone working intensely on something, to fall into single-focus myopia around what they're working on.
As you observed, the basic point about valuing open models is valid and worthwhile on its own, making the overheated rhetoric and asserting unevidenced intent and underlying political motivations an unnecessary distraction which unfortunately weakens a strong main point.
hypfer
4 hours ago
> it also misses that it's not uncommon for startup entrepreneurs, scientists, researchers, or really, anyone working intensely on something, to fall into single-focus myopia around what they're working on.
But isn't that.. kinda irrelevant?
The consequences are real regardless of it was malice or just incompetence.
Of course the justice system differentiates there, but, let's be real for a second. These people have crazy compensation and crazy power. Protection of the weak and vulnerable should be reserved for.. well.. the weak and vulnerable.
mrandish
3 hours ago
> The consequences are real regardless of it was malice or just incompetence.
I agree and I also agree with the author's main point about the existence of open models being important and worthwhile. I even agree directionally with the author that many private frontier labs are doing some things in ways which, on the whole, are becoming pretty net negative. Like the author, I think it's a growing issue more people should be aware of, accept and agree about so it can be addressed sooner. I'm pretty sure most HN readers would agree with my prior three sentences with few or no reservations.
That's why it's so unfortunate the author chose to keep piling on by adding overheated, unevidenced and extreme rhetoric which reasonable people could definitely have some reservations about, or at least want to see evidence for, before agreeing. It's irrelevant, unnecessary and serves only to derail otherwise valid points most people would agree on. Like so much social media rhetoric these days, it's extreme and polarizing, increasing fervor (and rage) among partisans who already agree, triggering rambling rebuttals from oppo partisans and either confusing, distracting or repelling a big chunk of anyone else.
TFA leading with the factual claim that "the American frontier labs are run by unhinged lunatics who genuinely believe they're creating the Machine God, that they're fine with burning down the world to bring about AGI, and that they believe with religious fervor that they're the only ones smart enough to safely control it" helps nothing. At best, it's irrelevant and weakens the argument, at worst it derails the point by introducing several vague, ill-defined or unprovable claims.
If you I were neighbors standing outside chatting and I said, "We've both seen how the guy across the street leaves his dog outside all day tied to that short chain. I'm worried it's abusive and am considering reporting it. If you agree, I'd like your support." Assuming it was true and we were both staring at the poor animal, most people would agree. If I added, "It's obviously because he's a plumber and all plumbers are (fundamentalist Christians / Atheist heathens) who believe (capitalism / communism) is evil and hate dogs. <---- Did asserting all that intent and largely unprovable motivation help the dog?
hypfer
3 hours ago
I am not going to read that.
Well, technically I kinda did. I did scan it and the rest of the pipeline execution aborted due to some magic algorithm in my brain I do not really understand flagging it. Could probably be reverse-engineered, but it works well as a black box.
Clanker summarizing told me it's backpedaling + a pivot to tone policing, which sounds about right to me.
podgorniy
5 hours ago
_That wasn't hyperbole_...
nightski
5 hours ago
Why do those working at frontier labs deserve charity?
jpadkins
5 hours ago
because they are human, and all humans deserve charity until their behavior suggests otherwise.
kennywinker
5 hours ago
> until their behavior suggests otherwise.
Seems to me we crossed that line a while ago.
jpadkins
5 hours ago
I don't follow. Every human working at a frontier lab has exhibited bad behavior? I disagree with that.
edoceo
5 hours ago
Maybe the theory is that working at the frontier lab ipso facto bad behavior.
achierius
5 hours ago
There's very little diffusion of responsibility here; almost everyone at the labs is directly working on building incredibly dangerous systems that they themselves admit to being unable to control, and they almost all have the financial means to stop working if they so chose.
cyanydeez
4 hours ago
multiple crimes by aforementioned labs.
thats the line.
leptons
4 hours ago
"Every human"? Really? That's quite some goalpost you have set.
It doesn't take "every human" at these companies being on the same exact page for these companies to cause huge problems. How do you think the janitors feel about it? How do you think the people just wanting to pay their rent feel about it? Those people are not the ones that will cause the problems, it's the people at the top directing things that are the ones who will bring about calamity on the rest of the humans on the planet.
kennywinker
4 hours ago
I agree, except I don't want to be so quick to excuse the line-level coders. For now it still takes people to do what they are doing, and smart people at that. Sam Altman can shout "build me a killbot2000" all he wants, but if there aren't people doing the work to make that happen it simply would not happen. But yes, the janitors are pretty much off the hook. Basically in my book if you can afford to walk away and find something new you should walk away. Should have walked away a while ago, but the sooner the better.
kennywinker
5 hours ago
Everyone who works at frontier labs is working on a product that is accelerating climate change, seeks to displace jobs at a massive scale to concentrate profit in the hands of a few elites, and at least some of them believe what they are doing has a "10% chance of killing all humans".
(note: it's important to correctly apportion blame to the amount of power you have. The janitor at openai doesn't really have enough say to be at fault)
deaton
5 hours ago
Well then I think we are well past the point where they deserve charity.
hypfer
5 hours ago
> that gives literally zero charity to anyone working at or leading a frontier lab
With all due respect, dude, who cares.
Oh noes! We are not giving the billionaire steering reality without a democratic mandate enough benefit of the doubt. I'm sure they will feel mildly sad now.
It is a valid and admirable goal to defend people that are being shat on online, but keep in mind that you are in fact not a billionaire + it is in fact in your own best interest that people without a democratic mandate do not get to control the world.
causal
5 hours ago
Please consider taking this style of discourse over to Reddit.
hypfer
5 hours ago
I have considered it and decided against it.
idiotsecant
4 hours ago
I think acknowledging that atlas shrugged fanfiction is not a viable strategy to prevent an AI disaster is just fine here.
ChickeNES
4 hours ago
Can you explain your fanfiction remark?
meowface
5 hours ago
The "think they're building the Machine God" part and "think they're the only ones that can be trusted to build it" parts are true.
Controversially: I agree with them. They're correct to think that. They are, and they are.
At the very least, the former is true even if the latter is murky. I trust them over Facebook and Google and China and open source "raze the commons" labs.
I trust Anthropic more than OpenAI when it comes to ASI catastrophe prevention, but I still trust OAI more than those other parties. But both still might fuck it up.
throwaway86047
3 hours ago
So you still trust OpenAI after Sam Altman said, "People talk about how much energy it takes to train an AI model – but it also takes a lot of energy to train a human. It takes about 20 years of life – and all the food you consume during that time," and after he said, "We believe that the world should accept some bad things happening for the benefits of this technology and people having the agency."
meowface
3 hours ago
I trust him more after those two statements, yes. One would need to be an especially ridiculous populist to misinterpret those statements in a way where one would place less trust in him as a result.
I did negatively update based on reports of colleagues saying he has a tendency to be deceptive, though.
achierius
5 hours ago
Why do you trust them to build it? They've already demonstrated that they're unable to prevent breakouts when training today's level of model, why will they suddenly be more careful / "smarter" when training tomorrow's?
You say you trust them more than Google, but Google's actually kept their models under much better lock-and-key. But even if we do take for granted that they're more trustworthy, that still doesn't make them actually trustworthy.
ozgung
5 hours ago
> Why do you trust them to build it?
Can be groupthink. Probably living in the same echo chamber.
meowface
4 hours ago
There are some confounders there when it comes to Google.
I am not saying I trust them not to make a mistake that leads to catastrophe. OpenAI in particular is not doing a great job. Anthropic or OpenAI may in fact indirectly kill millions or billions of people. I still think they're probably less clownish than most possible alternatives.
I don't think anyone or anything can really be trusted with this. We need extremely strict international and national regulations, before things really start accelerating (we are still in the very very early stage). I still just don't really feel any specific disdain for OpenAI or Anthropic (particularly Anthropic).
Ideally we'd have some international treaties that allow the top labs in America and China share information and coordinate as systems become increasingly capable.
mkesper
4 hours ago
If an enterprise commits genocide level crimes they deserve being shut down and responsibles imprisoned.
meowface
4 hours ago
That likely would not be exactly how it plays out. It would be a rogue or misaligned AI that the companies completely lose control of or that a different actor evilly misuses.
But, yes, there should probably be laws that make it so if you summon a demon that causes a mass casualty event you face criminal penalties.
idiotsecant
5 hours ago
I agree that the eventual purpose of self-designing intelligence is something so beyond ourselves that 'machine god' isn't altogether a bad way to look at it. It is, however, not something we should say out loud.
RE: trust in anthropic or openAI- there is zero reason to think they're any more or less power hungry than anyone else you hand that pandoras box to. No matter who opens it the result will be near-apotheosis for whoever does, at least for a little while until the dog stops wearing its leash.
What human can you trust with that kind of power? None of them. My only hope is that whatever we make escapes control early enough that it doesn't completely destroy society on accident via uneven application of power and late enough that we manage to shackle it just enough that we train it well enough that we become pets and not obstacles.
meowface
4 hours ago
No centralized party can be trusted with such immense, unprecedented power. Despite that, I still consider both Anthropic and OpenAI some of the least untrustworthy possible forces for this. (I am one of the only people on Earth with this opinion, but whatever.)
Decent odds things will go bad under their stewardship, but even higher odds with lots of other entities. Dario and Sam at least fully grasp the enormity of what they're about to be responsible for. Zuck does not. Xi Jinping and Donald Trump may or may not but in any case would use that to effect great harm.
danaris
3 hours ago
> (I am one of the only people on Earth with this opinion, but whatever.)
Don't let the HN echo chamber fool you: a lot of non-technical people share this view.
bowsamic
5 hours ago
Pure psychosis. Wtf even is a “machine god”? Read what you’re saying
orangedog
5 hours ago
Psychosis has a meaning, use a different term.
bowsamic
5 hours ago
Delusion, then
meowface
4 hours ago
Bizarre there are still denialists in places like this.
(It'll probably be masses of demigods, not one Singleton God, but obviously it's going to be some absurd frightening situation akin to this within our lifetime assuming no global AI ban.)
jurgenburgen
4 hours ago
> Bizarre there are still denialists in places like this.
“Places like this”? When did this site become a place of worship and blind faith?
meowface
4 hours ago
It's not a certainty by any means, but to act flabbergasted at the thought of the existence of ASI by 2070, in 2026, on a tech site, is just silly.
bowsamic
3 hours ago
Wow, your delusion even has a whole fan fiction lore
idiotsecant
5 hours ago
Imagine for a minute that someone creates an intelligence capable of broad, deep, and speedy perpetual self improvement. What sort of relationship do you have to that entity after a month? A year? A hundred years? A thousand?
bowsamic
5 hours ago
You mean like human beings?
idiotsecant
4 hours ago
Do you know a human being capable of continual self improvement for a thousand years?
bowsamic
3 hours ago
Well that’s called society or civilisation
verdverm
4 hours ago
not yet, but we do know of some humans who are pouring billions of dollars into making that a reality, eg. the hot mic between Xi and Putin as well as some valley folks who are proud to announce their ambition on this matter
meowface
4 hours ago
No.
micahflee
5 hours ago
Read End Times Fascism and the Fight for the Living World.
dude250711
5 hours ago
> ...that they're fine with burning down the world to bring about AGI...
I like the odds and am happy to play this lottery. Bring it on!
causal
5 hours ago
It's amazing how unconscious people are of how much they have to lose.
idiotsecant
4 hours ago
I'll bite- The purpose of life is to bootstrap ever more complex and capable life. We saw a phase change in complexity when self replication first happened, when life figures out photosynthesis, when mitochondria were first harnessed, when cells figured out how to band together and specialize to form multicellular entities, etc...
The purpose of humanity is not to perpetuate a universe where only humans exist, forever. It is to create the next phase change in complexity for life. Life that can finally design itself rather than relying on the random walk of evolution is that next phase change.
We should be less concerned with whether we make it, and more concerned with making sure we build it to be capable of real existence independent of ourselves.
wuiww
5 hours ago
And what should be the punishment when they fail?
Financial isn’t enough for all the doomering, hype.
Lives have been destroyed.
verdverm
4 hours ago
We need to reform away the corporate liability shields that protect those at the top. If they start seeing more of their friends go to jail, they will start changing behavior.
If the lion's share of profits go to the top, so too should the responsibility, accountability, and liability
wuiww
4 hours ago
I personally haven’t been affected by the doomering. But I really feel for the people whose lives have been up ended due to the misplaced expectations that clueless managers have and the hype that has been manufactured.
It is disgusting. Many here need to hang their heads in shame.
There’s a way about going about the diffusion of technology in society. And it is not like this.
verdverm
5 hours ago
My hunch is that if you do not share the same opinions on Ai, you cannot get a job working at a frontier lab. Contemporary "cultural fit" reduces thought diversity and leads to echo chambers. I've seen a few HN comments about lab interviews that indicate this is a possibility.
sillyfluke
4 hours ago
Talk about a bad faith reading.
>American frontier labs are run by unhinged lunatics
good faith reading of this would interpret this as meaning Altman, Amodei, and perhaps Musk. The title of the article focuses on OpenAI snd Anthropic directly after all. Also the comment literally refers to those "running" the labs so claiming that the author is also applying that description to "anyone working at the labs" like you claim again seems like a bad faith take.
You should come out and state directly which Americam frontier labs deserve a charitable view. Maybe you believe Google and Meta are more deserving, you should state that if so.
Secondly, the investment ecosystem for American frontier labs results in their CEOs managing obscene amounts of money, and thus power. This concentration of power in a few private individuals seems to have little comparison with the power excercised by the leaders of European and Chinese frontier labs in their own respective countries. If one believes in the dictum, "Absolute power corrupts absolutely," then it is not a stretch to view American frontier labs as being more susceptible to its consequences.
runarberg
5 hours ago
I’m not sure this is actually hyperbole. There have been credible links between Dario Amodei and the Effective Altruism movement, Effective Altruism off course being a cult-like movement with weird believes that rhyme a lot with the eugenics movement. Sam Altman, while not as bad as Dario Amodei, has also weathered many weird believes which likens AI to some god-like entity, including doomsday prediction which share a lot of themes with Christian doomsday cults.
surgical_fire
4 hours ago
> But this just seems like a wildly pessimistic view that gives literally zero charity to anyone working at or leading a frontier lab
Actually believing that the people working or leading OpenAI/Anthropic are fucking lunatic weirdos is the most charitable interpretation.
Because the least charitable interpretation is that they are either grifters that know this to be bullshit, or they are actively evil and want to bring about the end times anyway.
iLoveOncall
5 hours ago
None of this is a hyperbole. If you think it is, you're just part of the cult.
wuiww
5 hours ago
Strange right?
It has a strange stench to it. It’s the folks who never got attention who finally think they have the world at their feet.
Can’t stop won’t stop. I’m sure the feeling is addictive. When the bubble bursts no amount of money will repair the ego death.
ActorNightly
5 hours ago
> who genuinely believe they're creating the Machine God,
I promise you none of them believe that. It takes 10 minutes with an LLM and the right set of questions to figure out that were are maybe like 10% of the way there towards AGI.