I think there is something missed in how you're thinking about this problem.

Let's imagine that everything you say is 100% correct. LLMs can't access materials for developing weapons without weapons tests conducted at their behest, and humans choose not to - of their own volition - use world-ending weapons.

Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.

A "superintelligence" persuades humans to perform weapons tests and then to give it the ability to use the weapon.

I think everything you've said is reasonable except the bit where you think step one of an "AI" exterminating humanity wouldn't look like a human doing it. In my opinion, the most likely vector for disaster is a human being who is "guided" by a computer model to do something unfortunate, not the model doing so directly.

I think you lack imagination, but I'm not going to post ideas here for novel weapons that would render the humans' existing defenses and deterrence strategies ineffective.

It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons. They could destroy civilization, but not the planet or even the species. You can make straightforward calculations showing that we already know how to build such weapons, but we haven't.

As for tools, alcohol, tobacco, fentanyl, pornography, monotheism, warfare, gambling, and money itself are "tools wielded by humans", but they still seem to get into a lot of trouble where it seems like the humans are the ones being "wielded" by the tool. And neither monotheism nor gambling is smart enough to write a compiler for me, much less choose persuasive tactics according to my personality type. A lot of humans' basic survival instinct seems to fail or even work backwards in the presence of these "tools".

> AI is a tool wielded by humans. If we’re wiped out by AI, it will be at the direction of a human.

Being deliberately wiped out seems like it's mostly a misdirection. I'm sure some of the AI people high on their own supply believe it. And because the consequences are so negative, we should react as though the supposed experts were sane -- but I think it's a long shot that they are. (Aside: This should probably lead to some questions about the effects of AI exposure on the mind)

The bigger risk is that we end up in a rapidly enshittifying dystopia, treated as a cost center that needs to be optimized, and unable to competitively participate in the economy.

I have a number of successful AI investors in my social circle who are already placing their bets on a poorer and more disenfranchised humanity, with money going away from consumer tech, and towards a mix of B2B tech and policing.

This isn't really comforting to me for 2 reasons:

1. A key insight of foundation models is that capabilities are correlated. For example, training on code improved general reasoning capabilities: https://arxiv.org/abs/2309.16298

2. A profound capabilities explosion in easily measured domains will still feel very different to today. Like, the moment we come up with a systematic, objective way to verify a task, we become obsolete at it?

But picture the next 10 years, where all kinds of people, organizations, etc train in a ll different ways with diff philosophies and goals - I feel like there will be some good, unusual, interesting ones in there, even if they all have their own complex set of limitations.

Apart from physical world interactions which is obviously lacking, which domains lag behind and do not improve then?

I think where LLMs truly shine is personal greenfield projects, maybe start a side project

> work which only exists because of AI

Could you elaborate? What work are you doing which only exists because of AI?

You aren’t changing the world at all.

The bureaucracy is expanding to meet the needs of the expanding bureaucracy. -Oscar Wilde

I mean, for many years I worked on software applications for provisioning and controlling cloud orchestration platforms that only existed... to serve software applications.

It's not that deep—there were many more verticals outside of control plans, and there are now many more verticals outside of machine learning.

Or am I missing the point?

Counterarguments to that premise:

1. Intelligence per dollar spent/token output seems to be increasing rapidly. Opus 4 was $15/75 per million input/output tokens when released. Opus 5.5, which is significantly better, is $4/15. Sure it cost more money upfront to train Opus 5.5, or the bigger model (Fable 5.5) which tuned it, but once it is reached, there are massive diffusive effects.

2. The capital equation is still favoring more spend. Anthropic's revenue is growing rapidly and they are profitable on inference. Huge upfront spend for realized gains later on is standard for R&D. As long as serving the models is profitable and generates more value than the energy/compute consumed

3. The search space is too large to assume that improvement in AI assisted research is just models dumbly trying every combination until something works. For every additional bit searched, the space doubles, so even a trivially small combinatorial exploration incurs prohibitively large costs. Intelligent searching which continuously prunes that space along dimensions of effectiveness is not a trivial skill and is among the emergent accoutrements of these smarter models.

The article's author didn't have direct access to Anthropic's infrastructure, but she references a report from colleagues who did (https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleratio...), where they say:

> Anthropic has made roughly two years of historical progress over the last year [...] We are confident that the acceleration was not due to an acceleration in the scaling of training compute.

Note that Anthropic has essentially heavily censored this report, and the authors themselves say "the public has to rely on METR’s judgment much more than would be ideal".

I grok that you're not trying to discredit anything, or anything like that but what are you trying to do, if I may ask?

What information are you trying to either ask for or share with us?

The original article is pretty clear, agent swarms and token spend are AI automating AI, and the RSP attribution condition specifically excludes your capital thrown at training concern?

PSA: This is a very dated take from September; Ramez says himself that the new data has his attention and he’s digging into it now: https://x.com/ramez/status/2118472046812213577?a=KjLVobKV0gb...

In the original article he noted that getting to a 2x increase in capabilities would be crazy and require ~40x researcher productivity; that’s exactly the increase the new data now shows, and thus a huge number of the assumptions made in this earlier post about how it’s unlikely we would get to this point have literally already been proven wrong less than two months later.

I expect to see an update from Ramez moving substantially more in the direction of RSI in fact very much increasing the odds of reaching ASI soon.

Interesting write-up, but I feel like a lot of its argument relies on the interpretation of ECI scores as merely showing linear improvement in AI capabilities. Sure, Epoch AI themselves claim that ECI's scale is linear, meaning that a large increase in capabilities should result in a large increase in ECI. Yet, the widely perceived jump in coding abilities in late 2025 that started making agentic coding actually productive doesn't really show on the software engineering ECI.

The ECI is basically an aggregate of benchmark scores, but benchmark complexity might not increase linearly. In fact, if LLM capabilities grew exponentially, then new benchmarks would also need to increase their difficulty / complexity exponentially to stay unsaturated and relevant.

This article is really well written. Thanks for sharing.

> automated AI research is going to get us to "smarter models with fewer parameters"

That is indeed what we have seen over the last 5 years or so. Algorithmic improvements have made a massive difference to every dimension of efficiency and performance.

AI researchers can do things like discover new attention algorithms to optimize inference efficiency (likely part of why Anthropic and OpenAI are slashing inference costs in their frontier models and no longer capacity crunched).

I think we're bottlenecked on custom harnesses and well engineered context. Models can get smarter and smarter and smarter about math and software, and there's no reason to expect that those narrowly-godlike models will be any smarter about other things if taken out of their comfort zones. No such thing as general intelligence.

Getting past this bottleneck is gonna mean training our models not in datacenters, but in places that more closely resemble the ones where we want them to act. A factory, a wet lab, outer space... imagine a million robots spending all day finding every way to fall off the monkey bars so that the next version doesn't.

I assume this is indeed talking of those "smarter models with fewer parameters". Whether that's really happening is another question.

This seems kind of aggrandizing when BigAI shifts goalposts constantly whether it's "we created AGI" by a definition that an autopilot in a commercial plane fulfills or "Maths research" where everyone else has to do the actual research.

I find this as well for models that make it to production.

On a more fun note, I also find just trying some hair brained idea is now extremely cheap - so why not? I used to have to weigh up spending a week of effort on something likely not to work and now I can now get spit out test in 20 minutes.

In 2025, METR published a study that found AI slowing down developers by 19%... https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...

Are you saying the data in the charts in the article is fake?

Does BigAI want to hear that AI can maybe make sufficiently capable AI fit inside a consumer GPU?

Exact same here. Now I worry he was right about everything, more or less.

Even if he turns out to be a decade or two off, that’s basically exactly correct in the sweep of history sense.

Wild time to be alive.

I've found I can dramatically curtail the nonsense by creating good prompts about readability and directness.

We have addictive and harmful drugs. Heroin, meth, cigarettes

From Larry's Niven's Known Space series, including Ringworld

https://en.wikipedia.org/wiki/Wirehead_(science_fiction)

This is already true? Ultra processed foods and tiktok are both deleterious and both wildly successful. Maybe what you are concerned with is a theoretical sandwich that was unfailingly addictive so that no one could escape its effects.

fentanyl exists

Turns out that METR, the PR operation for big AI labs is better at predicting AI capabilities than AI skeptics.

I'm so tired of these articles. They are basically fortune telling with graphs. I have no doubt ai will help improve ai research. Will it accelerate it? How much? Will it hit plateaus? Who knows.

Trying out problems and making tangible progress towards AGI are entirely different categories. Likewise what’s changed in terms of paper submissions is also the amount of slop. Most reviewers say that they haven’t seen a lot more papers that are good but rather they’ve seen a lot more papers and almost all of them are terrible slop that’s desk rejected. This is happening in OSS too where a lot of projects are closing the gate at the onslaught of terrible contributions.

I agree in general that AI accelerates AI research, but good lord neurips iclr papers (and reviews might I add...) are the worst example of that. It's all complete slop that is totally meaningless.

A lot of great papers by great people had a lot of AI involved and accelerating it of course, but the majority is meaningless (sometimes autoresearch) slop.

Specifically talking about recent neurips/iclr.

In October/September my LinkedIn wall was probably… 55% people complaining that they were getting garbage AI submissions to the conferences they review for, and 35% people complaining that they got garbage AI reviews back from their conference submission… Of course, I have no idea if that means anything given how aggressively that site bubbles your feed.

And? Does HPC have no other applications than more HPC?

AI is dramatically accelerating my return to Emacs from Zed.

We can only hope.

vi research is dramatically accelerating research into how to quit vi

That doesn't line up with their claimed divergence point, which was pre-Mythos.

Could be a mix of it but I thought it was RL.

Perhaps scaling to fable for the first time also gave Opus a push (distillation?)

Nukes don’t kill people, people with nukes kill people. How do you feel about giving everyone on earth access to nukes?

A simple thought experiment - imagine AI discovers a way to make nuclear bombs capable of destroying the entire world radically more cheaply and accessibly with common items anyone can purchase. AI can help anyone build these nukes trivially. Even if the AI and the nukes don’t kill people on their own, do you want literally every human on earth to have this power in their pocket? Do you think it would be a net positive outcome for us vs a world where that’s not the case?

https://nickbostrom.com/papers/vulnerable.pdf

I.e. by 6 months, you double once By 6+3 months you double twice By 6+3+1.5 months you double thrice

Since the sequence 6+3+1.5+0.75+... never actually reaches 12 (geometric series), within a year they capabilities would have already been 'doubled' infinitely many times.

Obviously this argument relies on a ton of spherical cow assumptions (and the author does acknowledge this), but i think there overall point is correct that this form of growth (where time horizons compress) is _even faster_ than exponential growth.

> People don't get how serious this is. If AI doubles revenue but drives silicon valley firms' tax rates from 5% to 10%, it's a loss.

Would you rather make $1,000,000 and pay $50,000 in taxes, or make $2,000,000 and pay $200,000 in taxes?

Unless you’re terrible at math, i feel like i’m missing something ;)

Where does it say this?

I think you're not appreciating the work that Ajeya Cotra has done over the years. She really cares about timelines and existential risk.

Not everything is grift.

I feel you are committing a Goomba fallacy?

The author of this article works for METR and not Anthropic, their whole job is supposed to be providing independent oversight on the labs. They are definitely not in the "this is incredible" crowd, they are the ones asking for development and deployment to slow down.

Antrophic shouldn't self-regulate, it's government responsibility to do that. Unfortunately US has demented manchild as a president so that doesn't work like it should.