You don't need an AI to tell you where to buy castor beans or their seeds.

Lately "one-shotting" means "without human steering / feedback"

I think its a consequence of the improvements in harnesses and LLMs of the last year. Before they had a hard time self-reviewing the results and proceeding to a second iteration, now they are quite good at it although not perfect.

Have we already seen that impact? Or is it speculative? Because the speculative part is discussed all the time

The issue is, you kind of have to not get wrecked by the negative impacts to collect those positive impacts.

Unfortunately, "AI that's competent at bioweapon R&D" might be an easier mark to hit than "AI that can put a dent in the aging process". And biorisks can have a very favorable "resource input -> suffering output" rate of returns.

Cutting sugars, not smoking, regular checkups, watching your weight, managing stress, and getting enough exercise and sleep still trumps any other therapeutic-based life extension approach. But I know, I know, it's actual effort, and technology is about making things easier so it's a non-starter.

But I do agree that if we're on the verge* of von Neumann replicators and getting replaced by Pokhumans** to cheer on the ASI, then curing cancer, immortality and cryo-suspension are likely trivial achievements in comparison.

*Nope, not even close, but don't tell that to the ASI cultists, that's like questioning the veracity of the gospels now.

**Yes, that's the strong claim in AI 2027. And the democratic party apparently believes it.

But a virus's "goal" would not be to wipe out a species, that would be a dead end. Nature has explored flight on an unimaginable scale, yet our machines fly better (at least in most aspects) than birds, some even go to space.

yes - I don't want to publicize what I know. I worked briefly at a leading university bio lab. It's haunted me for awhile that it would only take 1 disgruntled postdoc to wipe out humanity. There are {AI things} now that will tell you how.

The obvious argument is that natural organisms are restricted in ways that artificial ones are not (e.g. natural viruses are not selected to kill their host too quickly, natural selection can only reach solutions which can be reached incrementally).

>EDIT: Also, in this world, couldn’t an AI just “one shot” a vaccine and an effective treatment?

This does not follow automatically. E.g. it is enormously easier to create an ICBM than to create something capable of intercepting an ICBM. Some technologies naturally benefit an attacker and some naturally benefit a defender.

Same idiots think folks are at risk from firearms printed from a desktop PLA 3d printer

An AI doesn't need to kill even nearly 100% of humanity in order to render us a non-threat.

Also there are plenty of doomer scenarios along the lines of e.g. some ethnosupremacist cult making a virus that kills everyone outside of their group.

The successes are in areas where you can get AI to execute an iterative loop autonomously. Try, fail, refine, retry - succeed eventually.

Coding is, inherently, quite amenable to that. Is bioscience?

They're certainly working on making it so:

https://www.anthropic.com/news/model-hardware-standard-resea...

Hah, well, you must have never encountered LessWrong before. It has been like that for well over a decade, in fact it's probably approaching its second decade...

One thing you've got to give them credit for is that they've been talking about AI -- about the present moment, really -- from the beginning. Nick Bostrom, who wrote "Superintelligence" about ten years ago, has always been in their orbit.

Yes, it’s also the way human society processes new stuff, be it novel technologies and the philosophical ramifications or other topics. Call it distributed Bayesian processing.

Sure each 2-cent philosophical posts isn’t worth much on its own but add up many of them over time and we get something more than its parts.

I guess you haven’t yet read Amodei’s essays? There is a significant overlap between the early SF AI scene and LessWrong contributors. That comes directly from the top, look at Anthropic leadership

Well with each new thing we get the grifters, lesswrong is an AI grifter hangout.

> if you can do the worst “bad” thing possible, you can also do the best “good” thing possible

You say this like it's a truism, but I don't think it is. The bad things may be categorically easier than the good things. It's easier to destabilize a delicate system than to stabilize it. It's easier to break something than to fix it.

Suppose it matters what order you do it in.

The best good thing possible for humans who are all suffering from an extremely painful, incurable disease might just be ending that misery.

The worst bad thing possible in a perfect utopia might be a minor inconvenience comparatively.