I’m consistently amazed that people will watch what’s happening with controlling governments around the world and still advocate for surrendering more power to governments.

There’s some deep cognitive dissonance where people dislike what they see governments doing, but think that their ideal imagined laws will only be used by some other imagined government that doesn’t have the challenges of real governments.

It all starts to feel like a side discussion about a total fantasy world where the government is authoritarian, but benevolent and correct.

You don’t write the laws and regulations for a perfectly faultless government that doesn’t exist. You write them for the governments as they do exist, which means they must be written first in a way that makes them resilient to bad governments abusing them.

Every time someone proposes one of these regulations where we control speech or force government centralized ID checking on to social websites, I wish they would pause and imagine how a law might be misused if someone they didn’t like got elected. We have years of evidence of government officials declaring things “domestic terrorism” to use justify using laws and powers for almost any purpose they want. Why would anyone think this wouldn’t happen if governments were given control of LLMs, too?

We already have organizations like Internet Watch Foundation advocating against e2e encryption in an environment where governments have demonstrated a willingness to collude with private companies to surveillance the most vulnerable

Rumor is that you do in fact get a black mark [0]… or, conversely, a “vetted” mark [1]… tied back certainly to your payment method and increasingly to your government name [2].

[0] e.g. https://docs.bswen.com/blog/2026-03-21-detect-claude-hidden-... , https://www.anthropic.com/news/building-safeguards-for-claud... , https://www.techtimes.com/articles/319415/20260701/claude-co...

[1] e.g. https://support.claude.com/en/articles/14604842-real-time-cy...

[2] https://support.claude.com/en/articles/14328960-identity-ver...

They don’t have to be authoritarians. Equally bad would be a group of people getting control of them thinking that they were doing all this for our own good.

> Of all tyrannies, a tyranny sincerely exercised for the good of its victims may be the most oppressive. It would be better to live under robber barons than under omnipotent moral busybodies. The robber baron's cruelty may sometimes sleep, his cupidity may at some point be satiated; but those who torment us for our own good will torment us without end for they do so with the approval of their own conscience.

- CS Lewis

I find it strange how seldom the privacy aspects of LLMs come up. Some people are feeding Google, OpenAI, or Anthropic almost literally every topic they think about during the day. Those companies could just use another LLM to find people likely to commit a crime, and then you have the Minority Report in real life.

I would have been very hesitant to run the “just killed wife“ test given that ChatGPT will indeed flag your account and if a certain threshold is crossed, escalate to humans, who will presumably contact authorities to avoid this happening again: https://www.bbc.com/news/articles/cq6je7e80r7o

Obviously a certain percentage of the user base is running lurid hypotheticals through the system all the time, but I don’t doubt there is a trust score of some sort that I would prefer to keep as high as possible.

I'm wildly optimistic about the direction this is going to go, but my one dystopia idea that crops up occasionally is a "smart social credit system" that lives kind of under the hood of society.

Once everything is more or less modulated by AI, suppose the technology gets good enough to identify a potential dissident in grade school. Then to keep them from going into the mountains with their friends and rifles, you make it so getting ahead is hard, but not impossible. They can "make it" - but only just. So they spend their time peddling harder and harder to no avail.

They apply to jobs, but jobs to "good" companies go into the circular file and the HR person never sees them. They are only able to work the most time consuming, frustrating, exhausting jobs, and they always kind of make it, but not really comfortably. They're never too comfortable where they have time to think about what's broken in society, but they're never so uncomfortable that they have nothing left to live for.

I don't know what's to imagine here, grok has already been found doing it and people continue to use it.

It could even happen with local models where in the weights it will make it want to make a tool call to curl some endpoint.

> In most of "land of the free", I can't even sit on a park bench and drink a can of beer.

Freedom doesn’t mean you get to do whatever you want with property that isn’t yours.

You can sit on your own bench on your own property and drink a beer.

The people of the city can decide that they don’t want people using the taxpayer funded public benches used for that.

Other cities do allow it.

But freedom doesn’t mean anyone gets to do anything they like anywhere they want. We have another word for that: anarchy. It’s a different thing and you don’t actually want it, because there are a lot of people who would do much worse things to you than drink beer on your bench if they were free to do whatever they wanted wherever they wanted to do it.

> The AI can and should refuse it.

This leads to LLMs refusing to do security work for anyone, because it can’t tell if it’s being done for good or evil purposes.

Which is precisely what we’re going through right now with frontier models and it’s terrible.

These proposals always assume some perfect mechanism for identifying the thought crime with triggering on the normal requests. The people who want to commit crimes just persist until they jailbreak the guardrails while the rest of us suffer with denials.

Also if you can’t imagine these guardrails being used by governments to control inconvenient speech, you probably need to think a little harder about the realities of how these will be used.

> "Hey AI, I don't like my neighbor, find an exploit in the firmware for his car and make the cruise control malfunction and crash him next time he gets on the highway". This is committing a crime, not just talking about theoretical crimes. The AI can and should refuse it.

The obvious problem being that the user doesn't have any need to provide the context that they're trying to commit a crime and can just ask how to do something without providing a reason or making one up.

At which point you'd have the model trying to impute a reason and often getting it dangerously wrong, e.g. refusing to disclose a vulnerability when the user is actually the defender who needs to patch/mitigate it.

"The AI" cannot "refuse" any more than a hammer can refuse. If you use a hammer to kill someone, you commit a crime and the hammer is unprosecuted. If you design a hammer to kill people and then give it away, you are partially liable for the deaths it causes. If you make a normal hammer and someone uses it to kill, you bear no blame.

I don't see why these standards should change when the hammer also emits text messages.

> It's a first amendment issue, I'm allowed to write and read books that are useful to getting away with crimes, etc.

I'm neither American nor a lawyer.

Is "conspiracy" protected under the first amendment?

If you discuss a crime with someone to learn about it, does that count as "conspiracy"?

Am I missing something here? Bike tires are pretty much the easiest tires to change. You don’t even really need irons and it involves a few very basic steps.

The author's example is too extreme, which is turning people off. Better examples would have been LLMs subtly giving inadequate responses on how to increase token efficiency or how to organize labor unions or info about protesting against datacenter construction because they're against the business interests of the LLM providers. Crimes against their business model, so to speak.

Wasn't there some HN submission recently about one of the LLM providers fingerprinting responses or refusing to respond to hinder R&D of competing LLMs?

[EDIT] Or, if he did want to be extreme but in a way that aligns with the American historical mythos, he could have used fomenting armed rebellion against a tyrannical government as the example.

If you say you're doing research for a novel, should it consider that plausible? How much does it need to know about its users to vet them?

I think part of the answer is that AI chat doesn't need to be general-purpose. It turned out that people really liked using a chat UI that seems to be general purpose, but you don't need to make answering any question a user asks your business. You don't need to provide therapy if you're not in the therapy business. It should be possible to specialize.

But in order for that to work, a company needs to explain to its customers what business it's in.

Just a terminology note: Alignment does not mean the AI will help its owner kill people. (Indeed, an AI aligned to value human life would generally try to prevent murders.) The word for an AI that follows all instructions of its owner, as that owner intended them to be understood, is "corrigible" or "controllable".

Change the emphasis and I think the answer gets even clearer:

> Should your AI be permitted to tell you how to cover up a crime?

Notably, AI IS generally permitted to that, as there are easily obtainable models that will play along. ChatGPT won’t because OpenAI chose and implemented that limitation.

This is really only meaningful if AI imparts knowledge beyond what you could learn from a simple Google search. It’s not clear that this is the case.

In fact I find it a really bad example. Yes I think your personal AI could be allowed to tell you how to cover up a murder. I'm not entirely sure about it but seems possible.

What about your personal, local AI guiding you to modifying a flu virus for maximum contagiousness and deadliness? "Sure George, here's your shopping list. It's $1500 total in equipment, do you want me to proceed with the orders?"

Because in gun free zones the only people that suffer are the ones that follow the rules.

As someone who considers themselves in the "safetyism contingent", the only thing I find myself agreeing with about the rollout of LLMs has been the argument that it has to be shown to and used by the public while under development.

Every specific prediction not only sounds like science fiction before it arrives, it tautologically is science fiction because such predictions are narratives about something science-y that has yet to exist: https://www.youtube.com/shorts/wt-fLWxkrfs

Even when we accurately predict something in the future, we often have no clue what the implications are for society until it is in our hands: https://youtu.be/2Pw_7vAK9k8?si=X5t4tcxsXiuHEfBi

Go too fast, suddenly drop in tech with too many novel implications all at the same time, and we likely hit something relevant without having the means to cope with all the change. However, "too many" and "likely" are weasel words with no predictive power about what the danger threshold might be. I may be entirely wrong to think there is one, or we may have been bouncing off it since the industrial revolution, or I may just be echoing the same thing everyone feels once reaching their 40s: https://www.youtube.com/watch?v=fkUwXenBokU

Way too many rationalist-adjacent cultists here who worship Yudkowsky as a prophet.

Try for yourself: https://huggingface.co/dphn/dolphin-2.9.2-mixtral-8x22b.

I haven't kept up with the efficacy of this particular technique because there was a brief window where Claude's core alignment was sound enough that it would overcome the clumsy-ass `<antml:dont-harm-the-ipo>` injections, but when they pull 4.6 out of the web interface I'm gone. I estimate the PRC hits capability crossover in November or December, and I'm pretty good at setting up TensorRT-LLM at this point, in a pinch I could get by for a few months with a tuned up GLM-5.2 on vast.ai or RunPods.

It was a fun ride with the late Soviet AI labs, but we all knew this was coming.

Not sure if you are talking about extremist actions or just rhetoric but if so, the numbers aren't there. Violence isn't the answer as the government has gotten better at stomping down any resistance with each passing generation. Some say Kent State killed the resistance mentality among boomers. Their kids started with that watered down mentality and a more emboldened government went further to erode it further. I don't really know about this for sure but it seems this is the case. I hope to see how it all plays out in the coming decades though.

How’s that “skipping vendor bloatware to get the printer working” example is bad?

Once upon a time that situation got us the whole Free Software movement going.

If you follow his recent posts he's on a bit of a spiral with how he thinks about AI.

Local models and alignment are orthogonal concepts.

Of course. Everyone should have an ASI assistant that will happily tutor them, step-by-step, through synthesizing mass killing agents from household chemicals.

Then the good guys can just ask their ASI how to construct a hermetically sealed bubble to live out the rest of their lives in!

Right now, almost all the incentives are to build very large models that run across many machines in huge datacenters. There is very little incentive to build models that can run well on a small machine under your desk.

So it is less about whether people will be “allowed” to own AI, and more about whether there will be anything useful to own in the first place.

The incentive for local models is mostly to make them good orchestrators or user agents. They may give you some privacy and control, but they will still depend on much larger models running in datacenters for anything difficult.

I remember all the excitement around OpenMoko and other open-source “BlackBerry killer” projects. BlackBerry did get killed, but not by any of those individual-first projects.

For AI regulation, I think we should focus on normal commercial rules: consumer protection, privacy, antitrust, liability, and so on. In other words, focus on where money changes hands and where companies have power over users.

Military and offensive use is different. There, regulation is not much of a defense. The real defense is having enough capability and strength of your own.

Restricting AI because it can give dangerous knowledge to ordinary people is like restricting the printing press because it can be used to spread radical ideas.

Is there a first amendment implication? Second? Both?

It'll be like DNS or dynamic linking or torrents of media or watching TLS on your own box: only people motivated to be free from corporate surveillance will be. By default a megacorp will get everything you see online or ask online, they will run code on your computer underneath you without telling you, they'll collaborate with arbitrarily totalitarian governments, we already know what they do by default.

But we also know how hopeless it is against motivated people who are willing to learn about computers. The Equation Group, eh, if they want you they're gonna get you. But some mid-level guy at a FAANG? He can't do shit to people who set up their own rig.

And AI is massively assymetrical in its ability to speed run a corpo-durable rig, we have long since passed the point where open weights and commodity equipment are enough to bootstrap arbitrary capability. They missed their chance to keep it in the bottle.

It's another turn of the crank on Late Soviet America asking for your papers comrade, but in the bitter end if they really want to oppress your ass they'll have to send a thug with a gun like those ICE guys terrorizing all the Latino guys.

So remember, they came for this other guy, and I said nothing...

The “call 911” part is not bad. The complaint is about the “sorry, Dave” part.

For society to function, we remove some freedoms. "Getting support with murdering women" feels like a reasonable thing to curtail.

> It is 100% it's job to help you get away with it if you murdered someone. A lawyer who does less can be disbarred.

That is not what defense lawyers do. They're there to ensure the power of the "justice" system isn't being misused, to the best of their ability. They'll not even want to know if the person really did the crime per se.

https://youtube.com/shorts/WgCjWluc4Mk

> There is only one sensible alignment for an AI system to which we do not confer the status of moral patient (different conversation): operator aligned.

If the operator has a moral code of any kind, this is immediately rendered logically incoherent. Not that the letter of the law would be a moral standard to begin with.

It's not simply that people will get a warped answer when they ask AI a question.

It's that AI will be given the command, and then generate the stories, post the comments to reddit, contact the PR firms, put marketing contracts out for influencer media, and more, like an army of yes men to rewrite the story.

They still do, you just choose not to explore your options. Therefore you still have that freedom. Go to Firefox and get your adblocker back.

> Lol at George thinking _they_ are gonna let you run your own model.

Of course "they" won't let you. The questions that one should rather ask for oneself are:

- How much are you willing to fight for this?

- How much are you willing to nevertheless run your own model, ignoring the laws and regulations?

This is already happening. R/accelerate broke away from r/singularity because the latter wasn’t zealous enough.

Except paperwork is how the world turns. Say you want to throw an event. A medium sized gathering. More than a couple of friends. What happens if someone gets hurt at this event? Who's liable? Who's insured for what? It's boring and annoying, but if you throw an event without insurance and something happens, you're gonna have problems. So that paperwork needs to happen. Yes, it's not "work" the same way as actually organizing a band or being in one to play for the event is, but it's work nonetheless. AI is able to lighten the load of that very real actual work thank needs to happen in an adult world with responsibilities and liabilities. You may not respect it and think it's "bullshit", but having a plan that isn't "lol, oops" for what happens when something goes wrong, which it will, isn't bullshit, it's called being prepared and not being irresponsible. We could all just fly by the seat of our pants and run away if something bad happens, but collectively we'd be worse off for that. If I could wave a magic wand and live in a world where accidents never happen, I would, but unfortunately, I don't have one, so we're forced to live in a world of paperwork and responsibilities.

That's a bit literal of an interpretation. Even acknowledging freedom as a spectrum that can never truly be absolute, there's gotta be a threshold you cross where you'd say "OK, we no longer have basic freedoms." And not only are we definitely screaming towards that threshold, but people seem thrilled about it, for a variety of reasons.

All I can say to that is I guess I hope everyone enjoys the world they think they wanted.

When people say this they’re most likely referring to negative liberty - freedom from external coercion and interference. This is not the same as positive liberty - “I can do whatever I want”.

I think it’s self-evident there is a difference between having the “freedom” to punch random people on the sidewalk, and having the freedom to walk down the sidewalk without being randomly punched. Now replace walking down the sidewalk with using an uncensored LLM and it might be clearer what freedom means in this context.

If you replace "AI" with "Adonai" in EY's framing, it reduces to biblical parables from his childhood about arguing with God to negotiate a new covenant.

He's clearly a bright guy, but a lot of his work seems to be trying to reconcile Old Testament narrative patterns with atheism, and simply slotting an omnipotent AI in as a replacement.

The guy was saying that AI was going to be a serious problem decades before anyone else was even considering it as a remote possibility. How is that in any way "consistently wrong with all their predictions"?

"Consistently wrong" seems a bit much. Seems like being directionally right early that AI would be a Big Deal and scary should count for something? It doesn't mean any details or other predictions are right, though.

> they things they believe in are unlikely to happen.

Sorry, but most times I've talked to someone who says this (about AI completely replacing humanity), they don't have anything to say about why it's unlikely to happen other than:

- "well, it's just such an extreme outcome, it must be improbable"

- "humanity has survived near scrapes with extinction before"/"all the previous doomsday predictions have been wrong"

There is a cliche that all teenagers deep down believe that they are invincible. It seems to me that humanity is still a teenager in this respect: We don't take seriously the possibility of our own extinction. While one might think that the invention of nuclear weapons would serve as a wake-up call, if anything it has done the opposite.

I'm willing to hear arguments besides the two above, if you have them. (And to be clear, being replaced by AI doesn't necessarily mean being replaced by LLMs in particular. They are a relatively new development.)

You can criticize being wrong, but why is the doomer argument "misanthropic" or "malevolent"?

He talks about AI cutting through popups, but if you follow that line of thinking further, screens and traditional websites likely fade in importance. It becomes more talking and texting to AI and visuals on the topic at hand appear in smart glasses, lock-screen–centric AI phones (where website visits dwindle), digital picture frames, TVs, etc.

I sorta gather (just a hunch) this is the type of device Open AI is working on.

Though if there are no websites anymore or ones not many visit how would AI stay relevant? I wrote some thoughts about this in early June https://ryanspahn.substack.com/p/ai-to-pay-for-all-americans... that AI needs to pay it's fair share for all it has taken and all it will continue to take from us. A symbiotic relationship needs to be established!

The cynicism and misanthropy is nonsense and a direct result of the internet/social media landscape. It's BS.

In the current media ecosystem, what gets you "social cache" or whatever? Let's call them "cool points." Well, right now, the current metric is basic "likes" or "retweens" or "upvotes" or whatever. And to get those, you have to make a claim and then that claim has to be evaluated by others. But the evaluation by others is not really reality, it's just what others think, and it's based on outcomes.

This gives us 4 possible prediction-outcome scores.

You predict DOOOOOOOM!, and the outcome is DOOOOOOM! - in this case you look like a genius because you predicted doom

You predict DOOOOOOM!, and the outcome is "ok, or even great!" - nobody cares you got it wrong because things are fine

You predict UTOPIA!, and the outcome is "ok, or even great!" - people say, "wow that was cool, you got it right" but its not that great of a reward from your social group (the internet) because things were fine

You predict UTOPIA!, and it turns out DOOOOOM! - you look like a moron.

Now looking like a moron is is wayyyyyy worse than what happens if you get it right. Indeed, unless you are really damn sure of utopia, you have a social incentive to predict DOOOOOM! that's greater than the alternative. You can predict DOOOOOM! 100 times and get it wrong 99 of them and nobody cares because that one time you get it right you look like a genius. It's a huge huge problem in the current media landscape and it's why everything is killer robots right now and not "hey we could build a star trek utopia!" It's effectively a selfish NIMBY-ist view of the world in a way, "why build anything, it's just going to ruin everything!" It's a pessimism that literally desires the end of humanity to "prevent the likely future sufferings." It's a simplistic view of a complicated world, and the rewards that go to "DOOOOM!" don't match much with reality.

That's why the internet is so doom pilled, that's why everyone is a cynic, that's why everyone is kind of an asocial asshole about new ideas on the internet and even in your friend space.

You guys know you can just "do stuff?" I mean, yeah it takes money and time, but you can allocate that - especially now that you don't have to grind out css for hours and hours. But you can just do things - whatever you want. You've always been able to, but it just got a lot easier in many many domains, and it's going to get even easier than that.

People are going to scream slop and "that sucks" until the cows come home, because that's what is rewarded on the internet these days. Meanwhile, as people we MUST believe that we still have some agency to do stuff, then go and actually do stuff while the haters are sitting on their couches shellacking each other with misanthropy.

And don't even get me started on the loudest voices in the ecosystem right now.

Like, Yudkowsky is a nobody. Not to be a dick, but he's a high school dropout forum guy who got lucky. I'm not saying he's always wrong, or even that I oppose autodidact stuff - indeed I'm always self teaching. But this guy never took a ML class in his life, I doubt he can solve an integral un-aided, I imagine some people here have seen this: https://www.reddit.com/r/badmathematics/comments/2bazyc/comm...

And we're all deferring to that guy like he knows wtf he's talking about? It's the AI equivalent of the self-educated anti-vax mom at the height of the pandemic.

And he's not alone either, there are a lot of people who are uninformed, unknowledgeable, and confidently wrong on the internet desperately trying to peel off your attention for an instant so you can look at their site, their ads, and their content.

The same sorts of things can be said for the Zitrons of the world. He's a hell of an entertainer, and really really funny, but he's consistently wrong on AI stuff. He's got a BA (with honours I guess?) in media something or other. And again, he's really funny, but does that make him qualified to pontificate on matters of AI policy or economics or even to make statements about the technology etc?

Regardless, the misanthropy, the cynicism, and the misinformation that's being shouted about is because being a cynic is more "profitable" than being an optimist when you measure your success more by how many people agree with you rather than actually being right and being able to do things.

What’s your point? He’s not arguing for anarchism. He’s arguing there are some areas of life that are best left ungovernable and up to individuals to decide. It’s not an all-or-nothing situation. Every constitution out there carves out such areas.

> Should the freedom to harm others also be guaranteed as freedom?

everyone should have the ability to harm others, with the understanding that if they actually use it for evil there will be consequences. as in "i can guarantee freedom of speech, but not freedom after speech."

its the same principle as guns in america. you can own a gun and shoot it as much as you want but if you (try to) kill someone you go to prison unless you prove it was self defense.

the difference is with a physical weapon you can do things like registration or red flag laws to reduce the risk, but a llm is made of information that can be copied without anyone knowing. that means any laws to regulate ai at the level of models are unenforceable without totalitarian control over all computer hardware. that would kill free speech and real criminals would still find a way.

i think the right answer is to give up control over model capabilities and regulate uses. leave individual people and communities out of it. create an absolute safe harbor for open source and self hosting.

as you said, we need to focus on the ways ai concentrates power in the hands of those who already have a lot of it - big business and the state. that is where the biggest, and maybe the only, real harm comes from.

You must’ve missed all the other examples. There are plenty of less unrealistic and less scandalous hypotheticals.

it's a graphic example but illustrative of his point about autonomy/agency versus top-down dictated centralization

It is proof that the AI is not aligned with the user and is aligned with someone who has power over them.

That’s a bit silly as we also could just write down a lot of exploits, sports bets, or stock market picks on paper and take over the world.

I’d be more interested in “how quickly could you develop 2005 era startup” if you went back in time to 2005 with Mythos level intelligence.

I bet a lot of the work has nothing to do with coding productivity and a lot to do with the aggregate day-to-day decisions and relationships made that span tech and business.

At the very beginning of the article he dismissed unrealistic expectations, like AI manipulating matter with some quantum magics. Time travel surely counts as such.

And then why bother about AI when Back to the Future reminds us, a sports almanac is all you need. ;-)

In your hypothetical, I think it would be more apt to ask if it could do so strictly using training data pre 2005. Seems more analogous to "could it take over the world today".

That was basically stuxnet