> When people believe that AI could figure this out on its own, without first running extensive and expensive basic science experiments, they have left the realm of science and have entered what can only be described as religion.

I strongly agree with this. Too many people have the Arthur C Clarke quote infecting their brains.

So much of science is contingent on the physical world and cannot be determined from first principles i.e. by thinking hard about it. (Sorry Plato, you can’t think your way to the moon.) You need increasingly precise measurements, which means increasingly complex tools which means…supply chains. Science requires supply chains. Magic doesn’t.

Native americans had like a 95% population mortality rate from infectious diseases in the first generation of European settlement, no? Measles, smallpox, flu, etc.

The mortality rate from those diseases was higher than today because the population had no immunity at all, leading to both higher transmission rates and higher mortality rates. This would be true of an engineered virus as well.

And it wasn't just the one virus, it was an entire continent's infectious disease arsenal landing in overlapping waves. Again, something you'd do if you were trying to maximize damage.

So I don't find the bulk of the article about it being impossible for a virus to have this effect all that compelling.

The author seems to be arguing that it is biologically impossible or something to have a virus with the same transmission properties as COVID but higher virulence. I don't understand what principle he is invoking to make this claim.

Everyone is gathering around patting themselves on the back, but smallpox depopulated an entire continent. Clearly it is possible for a highly transmittable deadly virus to exist.

Just because the author can't think of a way to cause mass death with an engineered virus, doesn't mean an AI won't be able to think of one.

(There are several biological threats the author hasn't considered. I'll just leave it at that.)

Oh my God THANK YOU.

I do not have the words how tired I am of hearing tech people and economists talk about biology.

"The problems with which the universe can confront any society are, for practical purposes, infinite in number and endless in variety." - Joseph Tainter

AI will really aid the universe in generating more problems. Maybe it's not a supervirus, but hell, maybe it is.

Ye domesday weapon will not be a disenchanted teenager whipping up COVID-19 variants (whatever that means) and spraying them on people.

The real deal needs a lab messing around with viruses and whatever happened a few years ago ... and it did not destroy humanity.

Obviously, the scenario portrayed in The Economist article is a little far-fetched. I agree that a teenager won’t plan a deadly virus with ChatGPT anytime soon.

But I think the overarching argument here is mostly correct, a lot of really complicated science has been democratized to the point where it is much more accessible. You don’t need a team of brilliant molecular biologists, biochemists, and bioinformaticians to do what you can do with a few agents today. And the pace is increasing. this is the nature of exponential improvement when you approach what Kurzweil popularly call "singularity"

Sure, today, peptide inhibitors and enzyme design are still out of reach for many tasks, I know because that's what I also, as the post authoer, do for a living for the past 15 years. However, only five years ago, protein structure prediction was an open problem, and everyone in our field now regards it as solved.

I’m not worried about a lone wolf taking advantage of AI to design viruses, but I’m concerned that the bar has been lowered to the point where many rogue governments or terrorist organizations can now access tools and skills that were previously available to very few people with a prohibitively high price tag.

The author here looks at the problem with the tunnel vision of a scientist well versed in the field. I know because I’m one as well, but experience has taught me to be more concerned about the unknown unknowns. There could be a team somewhere working on something we haven’t thought about. It doesn’t have to be a population destroyer, either. It could be a tactical infectious agent or something that isn’t meant to kill everyone but ends up being far more dangerous than it initially seems, with implications that are much longer lived than the team anticipated.

I’m not saying we should ban AI or do anything radical, but I do think that with great power comes great responsibility, and we aren’t dedicating enough effort to the responsibility part. There are many benchmarks measuring AI capabilities, but not enough measuring safety. It is becoming increasingly important to develop those.

The author doesn't have a security mindset.

Has anyone read https://en.wikipedia.org/wiki/Biological_War:_A_Scenario

Sounds like there already exist some pretty crazy biological weapons in existence. I dont know if it will be 5 years or 20 years, but at some point some fanatic will try to kill the rest of us.

https://archive.is/kH6U5

AIs have no concept of time. If they wanted to wipe out humans they would release a virus that increases infertility by a small amount, and wait a few hunderd years for population collapse.

I am not a biology person, but, candidly, this is a bad article, because the author is generalizing from the properties of viruses that tend to arise naturally, and using this to argue about the properties of deliberately designed viruses.

Firstly, it's not even clear that naturally occuring viruses can't be extremely lethal. There are examples (eg black death).

Even the existence of rare examples should be concerning, if attempting to argue against the possibility.

But even if there were no natural examples, it could just be because the ecosystem is in a balance, which them unlikely. That doesn't preclude someone from designing one.

To use an analogy. Natural predator prey ecosystems tend to be in balance. You could look around the world, before modern humans, and conclude that superpredators aren't a real threat, because predators that eat their prey too rapidly typically then die out. Thus most ecosystems you observe are in some balanced state.

But this doesn't mean modern humans can't come along, using their superior intelligence, and act as a superpredator. Ask the wolf population of western Europe.

If something is stopping us from doing this more often, it's alignment rather than capability.

You can't generalized from the typical behaviour that you see in a balanced natural ecosystem, and use that to say it proves there are no much more dangerous possibilities.

If the author made first principles argument about the inherent difficulty of doing something bad, even with abundant intelligence, I'd be much more reassured; or about defense offense asymmetry favouring defence.

What's here seems very naive.

> “Oh man. I used to think Smith was a person with informed opinions, but reading his take on a topic close to my own expertise makes me question this premise.”

Snap. Mike drop. Why continue? lol. Blogs are so snappy. I love a good hot take, and make many myself. But one thing that bugs me are personal insults. We’re living in the most abusive time I can recall. You want to bitch-slap someone do it with a smile, or just tackle their arguments. Otherwise, get bent.

That sounds like a good video game to spin up in Astra. Missile command was pretty cool. We didn’t destroy mankind with that one either.

AI is just one of many things that concerns me. But it has lately risen higher on the worry chart.

I agree.

It'll probably just hack and disable our water works, power stations and supply chains and let us all starve.

"Well you see, a thief will never be able to open this lock, because it needs a key. And to build the key you first need to invent steel and build all of the complex machinery that can process metals. Which is completely unfeasible for a common thief to accomplish".

Well, not with THAT attitude...

“I can’t do it so you can’t” is not a valid argument in the age of AI. That’s the point.

We don’t need this nonsense. The author doesn’t actually know that for certain.

Yeah, if only.

My sci fi story I came up with was,not that a virus is good to kill us,but a rogue AI is going to use a virus or something to hijack our minds and have us do the physical work it wants done like an ant colony.

I wonder if there is a name to this phenomena. Where being knowledgeable in one thing (such as computer science) means you incorrectly believe you are an expert in another domain (such as virology)

Now here’s a real risk involving measles that I sometimes worry about. What if the measles virus experienced a set of immune escape mutations where suddenly existing vaccines no longer work? If this happened the consequences would be horrifying. We’d have an immediate, world-wide measles epidemic long before we could develop and roll out a new vaccine. What’s the likelihood this will happen? Informed estimates are that it is low. But realistically speaking, even if the likelihood is low, it’s probably higher than the likelihood that a disgruntled teenager will design a supervirus with AI. So, something to think about.

I'm confused. Would a bit of gain-of-function exploration + measles virus genome = supervirus?

I'm no biologist, but it seems like the genome is readily available. https://www.ncbi.nlm.nih.gov/nuccore/NC_001498.1

And it seems like the spots that have to be mutated are already known about: https://newsnetwork.mayoclinic.org/discussion/researchers-cl...

And this experiment has already been done: https://pmc.ncbi.nlm.nih.gov/articles/PMC8080110/

Five minutes of research suggests the sites -- Loop Epitope etc -- only need a few pairs of mutations each. this feels doable... It's not creating a supervirus from whole cloth, but why bother. Measles without a vaccine would be a catastrophe.

Well, this is disappointing.

Thank goodness someone wrote this, I was also thinking about writing an article like this.

This is such a massively overblown worry that AI companies who are seeking regulatory capture love to roll out whenever they are trying to gin up some FUD.

Creating viruses at home is extremely difficult, even if you have a perfect instruction manual and someone sitting behind you telling you what to do. It's full of trial and error and validating each step requires rigor.

It's also a process which requires fairly expensive and specific supplies, which can be easily gatekept. Instead of trying to gate-keep knowledge of biology, we can do things like KYC around specific types of lab supplies (like we already do with fertilizer which can make bombs, or specific types of supplies needed to make nukes). Heck the Biden admin already added a KYC requirement for plasmids and plasmid companies already have implemented screening for sequences which could be dangerous.

I dont 100% agree.

I agree an AI isnt going to create one. but the number of raving loons out there that think 5G chips are in vaccines are going to go absolutely stark raving mad the next time there is a pandemic.

then they'll be super susceptible to AI generated political adverts from the next fuckwit political hack trying to get into power.

looking at how the world is panning out atm moment... I expect AI to have a massive influence on the sort of idiots that will get voted into power and make decisions that will get a vast number of us killed. by all sorts of means... war, famine, unregulated corporate poisoning...

AI is going to get us killed, because we'll use it to kill ourselves.

tldr: 2029 -> 2049

The magical level of intelligence required is not really a flaw of the argument's deductions, it is its very premise. If you believe that there is a possibility of an AI working on itself, you get the approximate differential equation d/dt f(t) >= C*f(t). In plain English, at least some proportion of development's progress goes back into future development. The (least case growth, assuming equality) solution to this is an exponential function, and exponential blowup leads to stupendous and magical seeming results. I then ask - what is evolution if not a dumb optimization process? How are we supposed to believe that it devised miracles such as the human mind, insect flight, echolocation and so on while denying the possibility of a future EXTREMELY power simulation to do the same? I went in really wanting to be convinced by this article for the sake of my own mental health, but I am disappointed. What we really need for such a statement is concrete physical evidence that a virus is simply incapable of having such facilities. Or perhaps an equivalence established showing biological defense is no harder than offense. Until it exists, I will continue to worry.

I'm far from anything resembling an expert in biological weapons, but this seems to me puzzlingly overconfident in the opposite direction.

> PhD students in 2026 using state-of-the-art AI software are spending months or years trying to design simple peptide binders that inhibit some enzyme or pull down some protein

This confuses me, as state-of-the-art AI software is moving quite quickly. Any software that is currently state-of-the-art AI.. hasn't existed for years, maybe not even for months.

It also seems quite odd to claim that there are no sufficiently dangerous diseases when we have high-security research labs dedicated to exploring.. exactly these kinds of diseases.

And, even if it's not a disgruntled teenager, it just takes one leak from a lab that has 10xd productivity using AI tools.

This is a real risk that existed before AI, one that I think we can now confidently say that society is under-prepared for since 2020. It is certainly magnified by AI, even if it's not 100x.

The author even seems to recognize this:

> So, if you’re worried about AI-designed superviruses, think COVID, not Ebola. And if you’re now a little less worried, maybe you understand where I’m coming from.

This is exactly where I think most people are starting from, and that's _why we're worried_.