A lot of things being potential existential threats doesn't make them equally threatening.

Things like climate change, psychometric test levels and birth rates change slowly and give us time to react, things like nuclear weapons are accessible to only a select number of actors, AI enabled bioweapons is neither of those.

Your source of comfort is that we could restore the species with as few as 3,000 people?

'decide'.. look at how pro-ai people surrender to it, and how pro-war ai is...

If we could, it would be wise to prevent the sun from enveloping the earth at some point before that happened, yes.

(obviously "we" won't be around or whatever, but the point remains: yes, you should solve as many existential threats as you can before they're realized)

Having recently read both of Annie Jacobsen’s books — _Nuclear War_ and _Biological War_ — I’m much more concerned about a bioweapon than a nuclear one.

Her conclusion to the latter book comes out and says it, paraphrasing here:

> Nuclear war is terrifying in its destruction, but the chain of command limits the risk of a rogue actor. Whereas bio weapons can be designed and manufactured in relatively low tech environments with equal or greater magnitudes of impact.

Not even close to the same thing. That's taking an existing viral transfection system and swapping in a single gene.

Thermo Fisher sells a kit to do this [1]. It's something you might do a few thousand times on the way to actually generating a novel virus.

> I’d argue estimate the risks of a synthetically induced pandemic of massive impact this century are very high

I'd argue that it probably already happened, and had nothing to do with AI.

[1] AAV transfection.

“There is another theory which states that this has already happened.”

But the next one (or the first one, if that one wasn’t manmade) will probably be much worse.

>the risks of another synthetically induced pandemic of massive impact

No, you literally cannot. Synthesizing a stretch of DNA is trivial, and what you can send away for [1].

Building a functional virus is not even remotely a turn-key operation. Assuming you could get the genome, expressing that genome in a way that results in infectious virus is non-trivial.

[1] Even then, the companies will rarely give you long enough reads to make a full genome scaffold. It looks like there are two companies that will do 50-100kb assemblies as a service today, but I guarantee you they're going to look at what you're trying to make, and who is asking them to do it.

Would they not be like “hmmm this one looks kinda similar to a known virus”

They really just synthesize anything for you?

Yes in much the same way you can literally build a rocket and go to the moon.

You may be skipping a few of the involved steps.

One of the easiest and most effective AI regulations would be to simply require all these labs to run all incoming requests through a suite of the most advanced models asking if the request is likely to be a pathogenic agent.

Yes, the tech should arrive eventually but its needs a breakthrough to use readily available chemicals that are not regulated.

You are right, social and legal policy is important but it's not really gonna stop a terrorist organization in Africa.

An EUV factory is a moderately sized building in its own right[0]; on the other hand, DNA is manufactured separately inside most of the cells in your body[1], the plant on your windowsill, the mould on your bread…

[0] "43 freight containers and weighing in at 165 tons." - https://www.theregister.com/offbeat/2024/12/03/asml-releases...

[1] not in red blood cells

My understanding is DNA synthesis is now like a 2nd or 3rd year undergrad tier challenge and doesn't even require much specialized equipment.

Versus EUV which one could dump literally unlimited amounts of money into solving and still probably fail for decades.

It’s a tech tree question. Nuclear weapons have some very specific choke points that can be monopolized or policed, eg highly enriched uranium is largely a weapon input. There are also specific technologies you need to crack, which includes ignitions tests.

EUV is the pinnacle of a highly advanced supply chain, you could clone it but it requires many billions of dollars of R&D and massively specialized components; an economy like China likely can support the manufacturing tree required.

I would not be surprised if DNA printers seem end up closer to EUV or even more commoditized, if they are an extension of how all pharma/biotech manufacturing and research gets done.

But this is ultimately contingent on what the technology tree looks like. You couldn’t prevent diffusion of 3D printed ghost guns, even if you wanted to.

Sure we can restrict it, just like the only country that has used nukes offensively restricts nukes.

its not really a question of will, its a question of difficulty. if it were easy and cheap to build EUV processes, no amount of complaining and posturing would prevent someone from reproducing that.

Not sure if you realize it but biology is a lot harder to regulate. Most of the secret sauce in biotech is in the manufacturing process, not the reagents itself.

I will give you a classic example, the Lipid Nanoparticle used for the delivery of mRNA vaccine is the real lynchpin of covid vaccines. The media never talks about it. Any biology student knows how lipid bilayers work (they are the basic building block of cells, it's like asking a programmer what a terminal is). But the process of manufacturing those lipid nano particles at scale was the main bottleneck for delivering mRNA to the target cell. That was the trade secret. It's not sexy or hot like mRNA so the media doesn't talk about it. It's why many developing countries tried to make their own vaccines and failed completely. The mRNA was the easy part but what use is a bunch of mRNA if you can't get them delivered to cells?

Any biology student (even high school ones) can describe how a cell membrane works. But designing a man-made one that can be created at scale and be guaranteed to fuse with mammalian cells is mostly a matter of trial and error. It's a bit like finding viable pseudo linear memory use alternatives to the basic attention architecture in machine learning. Every lab has their own take on it.

With AI nowadays the search space of viable biotech ex nihilo experimentation is much smaller given that you can just prompt it to give you plausible protocols.

I hope I'm wrong.

Keep in mind the world is run by hierarchs. Of course the solution will look hierarchical in nature. The problem is fundamentally speaking, the hierarchism is a self perpetuating problem, because we still have to solve who amongst us can be trusted with the top of the hierarchy, which hierarchism states by definition "most of us can't be". It's like encryption, it's excellent at turning every other hard problem into a key management problem.

No, I think they mean that it used to take a giant lab full of specialists with lots of funding to do lots of damage before.

Now, the worry is a lone nut in a basement with no money could pull off an act of bio terrorism.

I had the same reaction, but I think they meant fewer as in "requires less consensus-building to wield meaningful power." Used to have to convince lots of people to go to war in order to kill a couple million people. Fast approaching a one- or two-man weekend project.

Fewer as in the size of an individual group, even though the total number of people who could find themselves in such a group is greater.

I'm so surprised that HN is this much anti-government. I'm a doomer myself, I fear the rich, I fear the terrorists, and I think the only one that

  - can save my life, 
  - is interested in saving my life
  - i have some control over

Yes, they are currently snatching kids of (illegal) immigrant parents and it is horrible, but who else would protect me against lone nuts in a basement, or robor armies of trillionairs?

It's super cringe and d e c e l to think technological advancement could have any dangers whatsoever.

Just like in a car, the true demonstration of intelligence is closing your eyes and slamming the pedal to the floor.

No, I'm saying AI has not introduced any fundamentally new capabilities to anyone in this space. Making a virus that grows in a cell is one thing. Making a virus that is an existential risk to humans is altogether another and would require significant new capabilities that do not exist and are not likely to from these training data. (Pre-existing sequences).