It's nuts how well "Superintelligence: The Idea That Eats Smart People"[0] aged. That talk is a decade old by now and still hits just as hard as it did back then, despite the incredible advances made in AI in the meantime.

[0] https://idlewords.com/talks/superintelligence.htm

The other thing about this article is that it contains falsifiable claims that are, well, false. Consider the following direct quote: "The 2028 election cycle is heated, as usual. AI is the biggest topic." No poll of the American public has AI as one of the top concerns except in polls where the question is specifically about AI. The general polls show topics like healthcare costs, inflation/affordability, government dysfunction, and immigration as the top concerns. AI doesn't even show up in the top 20 in every list I looked at (Gallup, Pew, YouGov).

It sounds more like "everybody I know thinks AI is the top concern" when everybody they know is people like them, i.e. an echo chamber. It gives me no confidence in any other claims.

Or you deeply suffer from normalcy bias. Not believing that AGI is possible is irrational and unscientific imo, the human brain exists, it is not made of magic, it can reproduced. It is as simple as that. We can debate about timelines, architectures that may lead us there, etc... You can even talk all day long about definitions of AGI, and people waste their time doing that. But saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point.

I remember thinking exactly the same thing around 5-7 years ago, in the GPT-2/GPT-3 era. "Oh sure they can produce semi-coherent output, but truly intelligent behavior is still far away. This isn't science fiction, they're just falling prey to Pascal's Mugging same as my religious friends did." Now I'm not so sure. I give the AI safety subculture as a whole a lot of credit for putting it on my radar back when it was otherwise still science fiction. I don't know if they're right about what comes next, but I think their case deserves to be evaluated on its merits, rather than assumed to be the result of psychological flaws.

> Yall, with your incredible wealth and resources you could do real good in this world and make society better, healthier, better educated, and the whole world more equal, just, and reduce the desperation and suffering. Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective". You can change a person's whole life in a moment.

Confused at who this is directed towards. I'm fairly certain that the article was written by people who (at some point) identified as effective altruists, most of whom would enthusiastically agree with this. This community didn't start as AI researchers and later choose effective altruism; they were effective altruists who chose AI safety research as the most effective way to improve the world. Given that you apparently share their goals (a better world,) isn't it worth at least hearing them out on their methods?

I would rather see you engage with the substance of the article rather than skipping right to insulting the authors. I don't think this sort of comment is up to the standard I have come to expect from HN.

> Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective"

Who is this supposed to be arguing with? It sort of reads like it's trying to disparage "effective altruism", but I'm not sure.

Setting aside any of the AI stuff, I've started to find it pretty grating when people seem to imply that transferring millions of dollars from wealthy people in California and the UK to impoverished Kenyans and Rwandans, or buying malaria bednets which can save a child's life for the cost of a fancy new gaming rig, is "self-serving" or something because weirdos are doing it, while true caring for other people involves [unspecified thing that doesn't appear to ask any material sacrifice comparable to donating a large percentage of income].

> I grew up in evangelical christianity, and to them the end of the world is just around the corner

I grew up in evangelical Christianity, I'm still an evangelical Christian, and you're wrong to paint with a broad brush like that. Are there evangelicals that are millenarian? Yep, of course.

But that is not my experience across multiple evangelical churches across decades.

To interpret charitably, I guess we could solve this for religion = technological development so advanced as to be indistinguishable from magic. It's been done in some Star Trek episodes I think.

The end of the world is always just out of reach. It’s the perfect carrot on a stick for people who are conditioned to be afraid.

Imagine observing powered flight for the first time and saying: "This is religious fervor folks. I remember the mythological story of Icarus from when I was younger. Key word, mythological."

The existence of mythology describing Scenario X is not a valid argument against the plausibility of Scenario X.

If we can acknowledge the possibility of nuclear doomsday without running an RCT of sample size 100 Earths, 50 of which undergo nuclear Armageddon, to verify that nuclear holocaust indeed a real phenomenon... then we can do the same for AI. Understand the arguments being made instead of engaging in these guilt-by-association arguments.

Modern AI capabilities are already mind-boggling by the standards of 20 years ago. We should at least prepare for the possibility that trends continue on the current trajectory.

From TFA

> Plan A is primarily a recommendation, not a prediction.

That sounds nothing like “religious fervor”

The fact that technology can increase existential risk for civilization is not fantasy. It’s a risk that should be reasonably discussed.

>Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective". You can change a person's whole life in a moment.

You can change more people’s lives, more substantially, if you donate effectively. Effective altruism started out as (and the majority of effective altruist financing is committed to) an effort to rationalize what has historically been a very emotionally driven activity by deploying insights from developmental economics. If you want to take longtermists to task, go right ahead, but please refrain from torching anti-malarial or child vaccination programs while doing so.

AI 2027 made substantive predictions about the near term future of the technology and the implications of it. I think the worst you can say about it is that the Agent-1 moment might come next year rather than this year. But being off by a year is far better than this 2040 slop, which is mostly disconnected from reality.

The one thing I will say that they are correct about is that AI does have the potential to be highly destabilizing geopolitically, even if they get everything downstream of that wrong.

When I learned that this website is by the same people who gave us AI 2027, I immediately thought about the Wikipedia page on doomsday predictions.

https://en.wikipedia.org/wiki/List_of_dates_predicted_for_ap...

Notice the many times when a prediction failed and the so called prophet would come back in a few years and give you a new date. And they would tell you, well, this time it's real.

I do find it ironic that many of the AI predictions are coming from the self titled "rationalists." It seems like building your identity around being rational and immune to psychological pitfalls is a good way to ensure that you don't even notice that you have walked straight into the one psychological trap every cult has employed since time immemorial.

People who've gone full true believer in "AGI" remind me a little bit of like, a random person from the most boring small town in the midwest who joined the Rajneeshee/Osho cult, or Hare Krishnas or something.

But instead of weird religious deities and practices, they've wrapped up their true believer zealotry in some kind of mishmash of "AGI is coming real soon now" like some kind of manifest destiny.

You could probably put people in FMRI machines and ask them to give a 30 minute lecture on the topic of AI and find that the same parts of the brain are activated.

https://www.cnn.com/2016/11/29/health/religious-brain-mormon...

[flagged]

[dead]

I completely agree. But not entirely sure that triggering the EA crowd is the effective way to deliver this message.

They are the crowd who need to understand your framing the most, but they’re completely shutdown by your framing, as any religious follower would be.

Your message gives me hope that not everyone’s drunk on the kool aid.

I love to model and simulate. As the dead economy theory[0] (discussion [1]) was submitted here, I decided to simulate it. It was really hard to figure out a path "good for the humanity", in the sense of a balanced system, not a winner take all situation, etc.

I think this is the reason why you have the tendency to propose some freeze-all policies, full control or similar. If you want to find the equilibrium, you need to accept that it will be a controlled equilibrium, most likely on a saddle point, with underlying process changing all the time, requiring fast changes in regulations. Our democratic systems, laws, etc. are not built to do that, they are built on the idea of intrinsic stability of our world where incremental improvements do not need cutting through what was decided before.

[0]: https://www.owenmcgrann.com/p/the-dead-economy-theory

[1]: https://news.ycombinator.com/item?id=48324712

> to me this article feels like an exercise in creative writing rather

because it is. Previously: https://news.ycombinator.com/item?id=43571851 / https://ai-2027.com/

To me, this feels like a last ditch effort to revive the AGI narrative to reject the coming and current commoditisation of these models, contrary to all current evidence. https://artificialanalysis.ai/

Yes. Anyone who doesn't acknowledge the efficiency difference between pretraining vs RL and assume that since we've run out of data for the former, we have to do the latter, is not making a serious attempt at modelling the future:

https://www.tobyord.com/writing/inefficiency-of-reinforcemen...

This is similar to that other exponential, which happened with CPUs - we ran out of true geometric scaling in the mid 2000s, and everything else supporting Moore's Law has been cleverness that arrived in the nick of time, supported by a bit of marketing, and very optimizable benchmarks, far from guaranteed gains coming from making a single physical metric better.

>Because AI represents the distillation of the very best human knowledge and expertise. AI is compression of human capabilities, the very best ones.

I'm confused if this is satire, sarcasm, or genuine belief. If this was the case, then AI companies should absolutely remove the "it may make mistakes", because doing mistakes would imply that "the very best human knowledge and expertise" is what actually fails, and not the AI.

With that being said, I'll still urge people to visit a professional therapist for health problems and I generally still trust human knowledge workers for critical scenarios. I will reconsider your claim when chatGPT can effectively play Yu-Gi-Oh! (or at the very least respond with the correct rules appropriately), which is a significantly lower stakes scenario than betting your entire company on its aptitude.

> Because AI represents the distillation of the very best human knowledge and expertise.

You say it like it's a fact, but in reality everyone sees the phenomenon of AI slop.

P.S. Information search and retrieval if the best and most direct way to use LLMs.

If you read the scenario, you'll see that the regulations are mostly about what people can do with giant compute clusters, and not about the ideas themselves. The ideas themselves are required to be totally transparent to the public.

As for historic precedents: Human cloning, human genome editing, and mirror life seem like one precedent; nuclear weapons and nuclear energy another; come to think of it I think drone delivery was strangled by regulations too...? Plan A isn't a proposal to never build superintelligence, it's a proposal to build it more cautiously and transparently.

Yes, there are examples of where we have collective decided not to pursue a particular technology tree.

For one, Japan banned guns for a few centuries. (Its warrior class was politically powerful and judged that guns would disrupt class relations too much.)

And there have been successful world-wide bans.

For example, following the invention of recombinant DNA technology, scientists convened the Asilomar Conference in 1975. They established a voluntary self-moratorium on certain types of genetic engineering until strict laboratory containment protocols were created.

In the 1980s, bioethicists, theologians, and researchers established a hard ethical line between somatic editing (treating an existing patient's non-reproductive cells) and germline editing (altering future generations).

No one has performed the latter form of genetic engineering except for Chinese scientist He Jiankui in 2018. (Chinese society used to be more ambivalent about the technology than the West is.) In response, Beijing heavily tightened its laws, classifying heritable gene editing as a high-risk medical technology subject to the penal code, and He Jiankui was sentenced to three years in prison.

> Are there examples of where we have collective decided not to pursue knowledge? Successfully?

Intrinsically, the knowledge humans choose not to pursue will not be much publicized. There's limited value in calling attention to it and it doesn't make for good entertainment. Plenty of examples provided by other comments nonetheless.

> Perhaps I'm horribly naive, perhaps I just see the SciFi future I've spent my life reading and dreaming about on the horizon and I'm blinded by the reality, perhaps my ideals around "knowledge deserves to be free/accessible" are misguided. I don't know.

I don't personally think there's intrinsic benefit in disseminating arbitrary knowledge. There's quite some difference between the printing press and nukes.

No, of course not. That would be an insane trust fall. Even relatively small advances in technology give a country world dominating power. Fun fact: India was militarily superior to Britain in the 1600s—a gunpowder empire with a million soldiers—but was taken over by it in the 1700s. Britain’s edge was small: lighter, more maneuverable cannons, standardized ammunition, better military and political organization. Not a first world country versus a third world country—more like the dynamic US versus a sclerotic EU. And that modest edge led to 200 years of colonization.

If we slow down on ASI voluntarily we’d be allowing a gap to open up that would make the difference between colonial europe and colonized Asia/Africa look trivial. It would be insane.

While technology has empowered governments, it’s also empowered the individual, and more importantly shifted the material dynamics to better align the incentives of governments with the people. Democracy followed material change, it didn’t precede it. Democracy came about because it was optimal for a power seeking government, not out of the kindness of their heart.

A resource extraction based economy sees people as slaves. The true source of power is the resource, people are just a means to an end, so you mistreat the people as much as you can get away with in pursuit of the resource while avoiding revolt.

With stable infrastructure, the government makes far more from an educated, rich population that it can tax and use the innovation from. It’s against its own quest for power to interfere too much in the prosperity of its citizens. The incentives are aligned.

Solving the AI problem isn’t about stopping the tech or making a bunch of brittle laws. It’s always been about alignment: aligning the large AGI-like entities that are the modern state, the modern economy, representative democracy, or AGI itself, with human prosperity

The decision to not go with the development of extremely large thermonuclear weapons might count - the US Sundial Project was supposed to be about 10 gigatons of TNT. Not the most practical weapons but once you get to a certain size delivery arguably stops being a problem - its going to kill everyone anyway so doesn't matter where you let it off!

https://en.wikipedia.org/wiki/Sundial_(weapon)

Edit: Mind you, I wonder if the design for Sundial is stored somewhere...

Perhaps the burning of the library at Alexandria would qualify. How intentional that was is somewhat in question, but the world certainly turned its back on the only collection of written knowledge and let it turn to ash.

The 1972 Biological Weapons Convention? Probably partly successful.

Biological weapons? Yes, there is research on defense, but no big arsenals of weapons etc.

My impression from the origin of the bioweapons convention is that collectively people decided that these things are too dangerous in various ways for any advantage that might be derived from them.

Taking the approach with AI that we took with the atomic bomb would be catastrophic. If the only people who are allowed to use this technology are governments, intelligence agencies, and a select few anointed companies, then the risk of authoritarian misuse will skyrocket.

I worry that any attempt to limit their use and development will be abused and misdirected. We are already seeing people like Anthropic doing this, they are trying to use anti-AI sentiment to engage in regulatory capture. Go watch Dario’s speeches about how open weight models are dangerous and how they are “not really open”. Everyone can see that much of this “safety” conversation is ultimately just a tactic to shut potential competitors out of the market and establish a monopoly/duopoly.

Human embryo genetic modification has been effectively taboo if not banned until just recently since WWII and the aftermath of the Holocaust. I think some people in the US are proposing doing it now but I don't think anyone has tried it yet without repercussions.

I can think of many examples that I won't name but you can imagine in biology/medical fields where certain lines of investigation are not performed due to ethical and legal repercussions.

If there were examples, their example status would drop the odds that I know about them.

It’s telling terrorists how to make bombs better apparently. Continuing to lower the barrier for that kind of stuff is clearly a negative in the “knowledge deserves to be free” world.

> Are there examples of where we have collective decided not to pursue knowledge? Successfully?

Isn’t that like all of the Middle Ages where we replaced knowledge with an alternate religious reality.

we should never make ASI, what I am saying is strongly sane. ASI = not good. AGI, ~human brainpower, can be made safe enough.

> Are there examples of where we have collective decided not to pursue knowledge? Successfully?

human GMO, some bioweopns, I'm sure theres a long list of awful stuff no one wants to exist.

> Are there examples of where we have collective decided not to pursue knowledge? Successfully?

Studying human bio-diversity since WW2 is the most obvious example, though it hasn't been entirely successful.

Genomics is what finally broke the barrier, especially in the last decade or so.

I like your optimism and I think you will be vindicated. AI is democratic and AI talent is globally distributed. It will just take a while to get online. AI labs do not have a monopoly on human talent, and open source AI only empowers independent science and meritocracy.

On a funny note, I think their prompt was:

"Hey Fable. Please attribute every piece of scientific and economic progress to AI until 2040. And predict every major geopolitical event. Make no mistakes."

Uhh north Korea?

[dead]

[flagged]

A successful example of reigning in progress: electric bicycles intentionally speed limited for safety

The "and physical" is the part I'm particularly skeptical of. Sure, drones are scary, but nobody's really solved getting a robot to deliver a package to your front porch in a civilian setting, and it seems unlikely to be solved quickly.

> Nor do I think a 74% unemployment rate is even remotely possible

250 years of constant automation has never produced large scale unemployment, despite obsoleting everyone's jobs several times over.

If I took you back to 2020 and said in a little over 5 years there will basically be no human coders writing code anymore you'd almost certainly not believe me.

And similar things can be said about many technologies in recent history – cars replacing the horse, first flight to man on the moon, even the creation of early internet to its mass adoption.

You're talking generally a decade or 2 for society to completely change from the rapid advancement of a new technology.

I'm not saying I agree with the 2035 prediction, but it doesn't seem impossible to me, if AI can help us improve the pace that we're already developing disruptive robotics.

In 2010 the idea of self-driving cars and autonomous delivery drones seemed very sci-fi and a long way out. But today, just 15 years on, these things are increasingly starting to be rolled out.

If they dropped that 95% number to 50-60%, I think I'd probably lean towards agreeing. Not because it makes sense in my gut, but because the logical part of my brain knows exponential trends (if one exists) do things that we wouldn't instinctively predict. But even if you assume exponentials 95% does seem very high.

Worse, there's actually a very detailed supplement about their economic modeling: https://ai-2040.com/supplements/economics-of-plan-a

Which predicts that explosive growth of robot production will lead to problems such as

> a deflationary debt spiral, where the AI and robot companies can’t pay back loans in dollars because the robots and AIs are worth nominally less than the loans written the year before.

In other words, the companies go bankrupt because they produced an oversupply of cheap goods, the bubble pops, and there's less new investment for a while. Plenty of precedent for such a development.

But instead of adjusting their predicted output growth downwards accordingly, they instead propose that

> One way to solve this could be for the loans to be denominated in AI and robots, so the companies pay back the loans with some percentage of the AI and robots instead of dollars.

Try doing this today with a battery factory for example. You expect that battery prices will fall to the point where the revenue from selling batteries won't ever cover the cost of building the factory. So you propose to a bank that they'll be the ones to build the factory, and you'll borrow it from them (not paying rent?), make your batteries, then give back the factory when you're done. All the profit is yours, all the risk is theirs! Which is of course why a real bank won't agree to this, all you're going to get is a dollar loan with the factory as collateral.

+1 this 2027/2040 AI articles are devoid of real-world practical economic/finance thinking. nor even human psychology.

> Without massive amounts of investment, AI development stops dead.

Would the US government not pour enormous resources in AI labs if needed, knowing that China might be doing the same? What happens if an adversary develops an AI capable of finding and implementing exploits in every software run by your country's strategic infrastructure?

> For AI2027 to be real, the money has to come from somewhere to carry on building the economy. If >10% of the workers suddenly become unemployed, and the rest taking paycuts, then money supply dries up. (unless central banks do something, but then that can be highly inflationary)

Why would it be inflationary?

The economy doesn't need workers as consumers necessarily. It would of course be a huge shock to the economy but the economy could adjust to it eventually. Maybe it compromises the 2040 timeline. Still, billionaires are increasingly holding the assets. The money supply drying up can be countered, as it is controlled by central banks and can be arbitrarily increased.

People have been saying this since GPT-1. This idea that we can only squeeze a little bit more of intelligence out of LLMs isn't a new one. And thus far, it has always been wrong.

What leads you to believe that?

I completely agree, but I also think that an industry disrupting architecture tweak akin to the “Attention is all you need” is VERY possible to emerge at any moment.

It feels like the cognitive gaps on current LLMs are indeed structural, but also that if we solve that structural issue with a new or extended transformer type of architecture, we’ll be looking at a whole new ballgame.

I mean, basically we’re just looking at needing some type of new post training learning architecture. It’s very clear that extending context windows isn’t that. What’s needed is an honest to god, continuous learning and modification process.

I'm eagerly awaiting "AI 2100" from the same people.

This seems way more accurate. Up until early 2026.

Though it has bizzare fixation on geopolitics and China which it severely understimated. It's pretty obvious that China is going to outinnovate and outcompute US companies quite soon. Even if just because they care about higher education, providing enough electricity and letting smart people do smart things instead of randomly muzzling them with bans and export controls and coddling them with financial protectionism.

I think you need to explain why the problems of developing countries are related to the issues of the article.

No, 2027 was never their median forecast; it was their modal forecast. See https://blog.aifutures.org/p/clarifying-how-our-ai-timelines...

It's more like it moved from 2028-2032 to 2030-2035 (depending on the author).

Fast takeoff was always only possible if you ignore all of physics and everything about how practical reality works, nothing exponential lasts for long in a finite system. If they believe that, then a certain level of self delusion is required anyway and you can ignore anything you like. Sort of like communists saying socialism will definitely work this time.

The book Superintelligence was so highly praised years ago but if you actually go and read the thing today practically all cases it presents read like raypunk retro-futurism that makes up imaginary fantastical nonsense in place of the missing knowledge it would've needed to make any sensible predictions. Practically none of its assumptions apply to LLMs as they currently exist, and some we've learned since about human inteligence are wrong too.

If that is true and one cares about a moratorium on progress in the US then it seems like the number one way is to meet people where they are: so water use misinformation, degrowth, power supply constraints. That does place all the people who push for these things in a different light. They may well be attempting to do what the AI safety labs are ostensibly trying to do.

As an AI safetyist, one’s closest ally (in a distributed coordinated way) is the populist misinformer. Fascinating.

No, nobody can be held liable for the extinction of humanity, for there is nobody left to pay up. If your objection is that you think AIs won't cause human extinction, you should say that instead, as that would be where you disagree with the authors.

https://www.youtube.com/watch?v=d6lDZpvHAoo&t=527s

The question (7:35) is "Where would you like to see people investing more time?" And Sam seems to be saying AI safety, I guess? This is 2015, and he refers to founding OpenAI. Based on his actions since then, yeah, seems like it's not a joke to him. This is Altman we're talking about.

I think he means the end of the world as we know it. Which is probably true, but the timeline could be 20 years, 2000 years, or more.

We are only some data centers away

These campaigns have historically amplified conflict. They do not care what the conflict is about.

But unlike some of the others, I’m hearing anti-AI sentiment from a wide range of people who don’t even use social media.

Which then gives more ammunition to folks like Kevin O'Leary et al to pretend any objections to data centers or AI are Chinese plots, or "hysteria" as you put it.

While I can admit some of the anti-data center arguments are overblown, many are more than valid in my opinion. Data centers are fundamentally extractive technologies. They are enormous, windowless boxes that take resources from one location to make someone else in a far off location enormously rich and powerful, with extremely few benefits to the local community.

Plus, as another commenter mentioned, it's not exactly like the Chinese and Russians have been fanning the flames that AI is going to take all of our jobs - it's the leaders of the frontier AI companies in the US saying that. Remind me again why I think putting up a giant data center in my state, that was proposed to use more electricity than my state already currently uses, is a good thing for the average joe where I live??

Honestly, I feel like many commenters here are in their own bubble and don't understand how much AI and tech generally is widely viewed as a net negative for society by huge swaths of the the population, and I don't really think it's an unwarranted perception.

There are 22 golf courses within a 30 minute drive from me, and people here a losing their minds over datacenter water usage...

Russia and China aren’t the ones constantly telling us AI will put us all out of a job, here’s why that’s a good thing, and why the government should dedicate billions of dollars to incumbent AI providers.

Nothing makes me dismiss and distrust a person or organization faster than hearing them say "russia/china are funding {controversial_thing}"

Its boogeyman thinking.

source needed. I'm not saying ur wrong, but source needed

What choice does that rag have? Of course they're going to backpedal and try to launder blame.

Surprisingly enough the us constitution specifies a right to free speech. Meanwhile who is funding the past two decades of pie in the sky bullshit from silicon valley in the media? Is promoting and unpopular opinion illegal? Since when do we judge the merits of an argument based on who articulates it?

Source or link?

By legislative design? If a nuclear bomb could be made with hardware store finds the world would already be over. Big collaborative works raise the stakes and the observability for surveillance. Apply for a job at a defense supplier or even and energy company.

If AI production is limited to big labs and big data centers then it is de facto contained and monitored. If you know where all the ASML machines are then you know the reproduction rates of chips. If no one can buy or build the machines required to concentrate uranium or plutonium to critical levels then the threat is contained and monitored.

You can dig up all the Uraninite you want. It was never much of a secret that uranium had dark applications. The machines and processes where thankfully big and expensive enough that only the most focused bad actors could aquire them and then hold the world hostage to the degree they do. If al-qaeda or isis could have used $40 bombs from home depot instead of expensive planes they would have (and they do).

You have to legislate and control the big, expensive, and slow things. Dynamite and phentanyl are so dangerous because they move much more easily. Freedom does not have to be a suicide pact. If the inconvenience of requiring prescriptions or access to dynamite reduces harm then it is net positive?

In Plan C the government essentially misses the opportunity to implement the multinational deal while the threat of covert projects is still low (fewer latest-gen chips unaccounted by tracking measures, worse models/algorithms to use for RSI). That's why it says the probability of a deal is lower and lower each month rather than outright zero.

The AI narrative requires an enemy and a “race” that the US has to win. It’s a pretext for justifying spending, rules, urgency, etc and making it sound more important.

Perhaps it’s because they are #2 strongest entity? Or maybe they are number 1, no one can say for sure.

Your comment also implies that those who haven’t participated in colonization might not fear being colonized. Seems… off.

Nothing follows from this empty platitude though, right? It can't inform you choices or decisions? It's just a disempowering thought?

They are buying up all the RAM today. Do you think "this is fine because in 5 years post-crash I can buy some cheap RAM"? If everyone with money is betting differently, do you have some information they don't, or is the whole economy just slipping away from you?

You experience luxuries today, that no king 1000 years ago could afford. Instant access to communication, food, medicine for the right price of course.

The consumer economy was great while it lasted but it's over now. We have machines that do useful mechanical work (engines) and useful intellectual work (llm-computers). Capital will move productive work from people to machines(if we let them), and the only jobs left will be delivery driver and warehouse, and then those will be gone too.

Human population was exponential and now its flat, but that's a function of what exacly? It could go back down to 1 billion or less. When jobs demanded a person supply was ready to match it. When jobs dont demand a person? Go to a degrowth rally take the temperature (and average age and child-per-person ratio) to get a taste of the future shape of supply and demand in a pessimistic world of sentences that don't have subjects just vague plattitudes. Are they net shutting down grade schools or building them in your neck of the woods?

Whenever I encounter these people I'm reminded of the meme about the baby who has doubled his weight in the three months since birth. At that growth rate, he'll weigh trillions of pounds by age 10.

> But every exponential is at a certain point a sigmoid.

strongly, no. its just hard to distinguish them. for example, radioactive decay. cmon

I’m not a booster by any stretch but I had a more positive read. To me it seems to describe a post scarcity society where ultimately the data centres are in e.g. the middle of the ocean and the robots’ labour replaces ours while still leaving us with the rewards.

I am extremely skeptical about whether this is even possible and even less convinced it’s a likely scenario but it seems like a good path.

The article reads like a compilation of online chatter, social media post, etc. on AI. Not surprising to find unsubstantiated imaginary numbers there.

Well... yes, it would. In theory, efficiency gains are deflationary. Huge efficiency gains are hugely deflationary.

If we're producing 10x as much, why not print 10x as much money? The goods and services each dollar could buy would remain similar.

1) Star Wars is fictional 2) There are autonomous boats.

Exactly, this article is fear mongering at its best for click baits. I’m pretty sure they will publish another one next summer for AI 2028 with same narrative

One of the authors talks about this here: https://www.astralcodexten.com/p/introducing-plan-a

> It’s increasingly clear that nobody has a plan for if this AI thing turns out to be real.

> ...

> Plan A isn’t another prediction. It’s a wish list, a positive vision, a road map for navigating the future.

> ...

> If we’re merely on track for a few cool gee-whiz AI innovations in the 2040s, then I’m wrong about everything and none of this really matters one way or the other.

I think their position is: "it would be great if current tech such as LLMs doesn't get us to AGI and only leads to some cool new innovations, but if it does, that's scary, because nobody has a plan for what to do, so here's our plan".

The jingoism is off putting. I think Daniel says it's a political necessity: https://x.com/DKokotajlo/status/2075261194978640096

A policy paper is intended to change policy. That means appealing to the party currently in power.

I mean, what is a robot? If you add up all the vacuum cleaners, 3d printers, and dishwashers, that’s probably close or more than the human population.

Hey, author here! Scott did contribute, but less than before.

On his blog he says: "I did a lot of writing for AI 2027 and was listed as a co-author. Some of my writing made it into Plan A too, but it was a bit less. The difference is of degree rather than kind, but because of this - and to give me more latitude to discuss it the way I like with less PR blowback - we decided not to put me as a co-author this time. I continue to be proud of having a part in this, small as it may be. (related: everything in this post is my opinion only, and not officially endorsed by the AI Futures Project)"

But do they at any useful scale? China is probably the only other country deploying AI at any appreciable fraction of their economy, and it's certainly much less than the US.

If the production, and distribution of the tech is confined to one or two companies, then those two companies and the resulting power that comes to them, makes them the de facto world government. (Democratic choice may be a farce but it is a useful friction on net.)

If they own all the RAM, models, and the means to do any work, then you are at their mercy. They will buy all the RAM, leaving you none, then all the transport, then all the electricity. You will be as boxed out of the current economy as the Amish and it will get its plug pulled.

Gradual Disempowerment is the default plan right now. War and tyrrany are not remotely the worst case scenario. I'll take Butlerian Jihad over being turned into cattle any day.

Imagine solving for equilibrium with two classes of beings. One requires agricultural land and 20 years to become individually productive and barely maintains a healthy population in a entertainment saturated landscape. The second eats only electricity and is ready to work on day 1. Round 1 goes to the strongest gorilla for sure, round 100?

If LLMs had come to earth in spaceships would you have welcomed them into your work and your home?

> Governments exerting this much control will only end in war

elaborate. regulation -> war, how?

That bet isn't one anyone will take the other side of, surely - how is your counterparty supposed to collect if they win? I guess you'd propose effectively supplying a loan with a massive interest rate?

> Because AI cannot retain memories or gain experience or insight based on the transformer/attention mechanism powering all modern AI models, it follows that AI lacks judgment and can never be trusted to handle truly critical decision-making responsibilities.

There are multiple teams working on adding long-term memory to AI. This is not a fundamental problem.

Fine-tuning to store memory in the weights is something that almost works right now, btw.

What predictions about the technology are the authors making that you do not believe?

There is plenty falsifiable in this in ai-2027.com, and they have not gotten everything right. But some things they have: for example, the Pentagon has already invoked export controls to restrict the deployment of a frontier model. This level of government oversight wasn't predicted until 2027 in the original scenario.

LLM adoption is 30% in the charts I saw googling for "ai adoption". An example of capibility: I have had Claude one shot an RL agent that learns connect four in 30 minutes. That's PhD level stuff.

LLMS are 4 years old and the companies that sell them 10x every year. What evidence can you cite? Could you convince a disinterested 3rd party you have anything other than cope? What facts about the world make you think this is anything other than the new (and probably temporary) normal?

> Ensuring Chinese compliance would be incredible hard to enforce or to check

consider SALT and other treaties from USSR vs USA cold war. they checked

I recommend actually reading their recommendation, because they get into the weeds about precisely how the US and China could address this in a trustless/auditable way. The TL;DR is that basically all of the relevant compute can be tracked.

Edit: Also, definitely not a Chinese op. The authors are prominent Americans, and are the folks responsible for the AI 2027 forecast that has pretty accurately predicted the current state of affairs today: https://ai-2027.com/

Are you trolling? OpenAI's models resolved the Cycle Double Conjecture yesterday, which has been an open problem for 50 years.

One of the lead authors, Daniel Kokotajlo, worked at OpenAI for years before quitting. In 2021 he wrote a remarkably accurate forecast of how LLMs would develop over the following 5 years[0].

I think it should be obvious that he understands that LLMs are trained via next-token prediction.

[0]: https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-...