LessWrong 2.0 Reader

I am still fascinated by how people are so sure they are right on this issue, in opposite directions

This is the first LW census I've taken!

It's not aligned at every possible point in time.

I think corrigibility is "AGI doesn't try to kill everyone and doesn't try to prevent/manipulate its modification". Therefore, in some global sense such AGI is aligned at every point in time. Even if it causes a local disaster.

Over 90% , as I said

Then I agree, thank you for re-explaining your opinion. But I think other probabilities count as high too.

To me, the ingredients of danger (but not "> 90%") are those:

• 1st. AGI can be built without Alignment/Interpretability being solved. If that's true, building AGI slowly or being able to fix visible problems may not matter that much.
• 2nd and 3rd. AGI can have planning ability. AGI can come up with the goal pursuing which would kill everyone.
• 2nd (alternative). AIs and AGIs can kill most humans without real intention of doing so, by destabilizing the world/amplifying already existing risks.

If I remember correctly, Eliezer also believes in "intelligence explosion" (AGI won't be just smarter than humanity, but many-many times smarter than humanity: like humanity is smarter than ants/rats/chimps). Haven't you forgot to add that assumption?

I think I must be missing something. As the number of traders increases, each trader can be less risk averse as their personal wealth is now a much smaller fraction of the whole, and this changes their strategy. In what way are these individuals now not EU-maximizing?

Consider someone asking the open source de-censored equivalent of GPT-6 how to create a humanity-ending pandemic. I expect it would read virology papers, figure out what sort of engineered pathogen might be appropriate, walk you through all the steps in duping multiple biology-as-a-service organizations into creating it for you, and give you advice on how to release it for maximum harm.

This commits a common error in these scenarios: implicitly assuming that the only person in the entire world that has access to the LLM is a terrorist, and everyone else is basically on 2023 technology. Stated explicitly, it's absurd, right? (We'll call the open source de-censored equivalent of GPT-6 Llama-5, for brevity.)

If the terrorist has Llama-5, so do the biology-as-a-service orgs, so do law-enforcement agencies, etc. If the biology-as-a-service orgs are following your suggestion to screen for pathogens (which is sensible), their Llama-5 is going to say, ah, this is exactly what a terrorist would ask for if they were trying to trick us into making a pathogen. Notably, the defenders need a version that can describe the threat scenario, i.e. an uncensored version of the model!

In general, beyond just bioattack scenarios, any argument purporting to demonstrate dangers of open source LLMs must assume that the defenders also have access. Everyone having access is part of the point of open source, after all.

Edit: I might as well state my own intuition here that:

• In the long run, equally increasing the intelligence of attacker and defender favors the defender.
• In the short run, new attacks can be made faster than defense can be hardened against them.

If that's the case, it argues for an approach similar to delayed disclosure policies in computer security: if a new model enables attacks against some existing services, give them early access and time to fix it, then proceed with wide release.

Which does make me wonder whether we were correct that shellac is what was on there before! Since that did seem sturdy.

A simple experiment I did this morning: github notebook. It does indeed seem like we often get more power-seeking (measured by the correlation between the value and degree) than is optimal before we get to the equilibrium policy. This is one plot, for 5 samples of policy iteration. You can see details by examining the code:

I feel like people like Scott Aaronson who are demanding a specific scenario for how AI will actually kill us all... I hypothesize that most scenarios with vastly superhuman AI systems coexisting with humans end in the disempowerment of humans and either human extinction or some form of imprisonment or captivity akin to factory farming

Aaronson in that quote is "demanding a specific scenario" for how GPT-4.5 or GPT-5 in particular will kill us all. Do you believe they will be vastly superhuman?

1. You start out talking about "large scale" attacks, then segue into the question of killing everyone, as though it were the same thing. Most of the post seems to be about universal fatality.

The scale of the attacks I'm trying to talk about are ones aimed at human extinction or otherwise severely limiting human potential (ex: preventing off-world spread). Either directly, through infecting and killing nearly everyone, or indirectly through causing global civilizational collapse. You're right that I'm slightly sloppy in calling this "extinction", but the alternatives are verbosity or jargon.

2. You haven't supported the idea that a recognizably biological pathogen that can kill everyone can actually exist. To do that, it has to ...

I agree the post does not argue for this, and it's not trying to. Making the full case is really hard to do without making us less safe through information hazards, but:

it has to be have a 100 percent fatality rate

Instead of one 100% fatal pathogen you could combine several, each with a ~independent lower rate.

keep the host alive long enough to spread to multiple other hosts

See https://dam.gcsp.ch/files/doc/securing-civilisation-against-catastrophic-pandemics-gp-31 for the idea of "wildfire" and "stealth" pandemics. The idea is that to be a danger to civilization would likely either need to be so infectious that we are not able to contain it (consider a worse measles) or have a long enough incubation period that by the time we learn about it it's already too late (consider a worse HIV).

have modes of spread that work over long distances and aren't easily interrupted

In the wildfire scenario, one possibility is an extremely infectious airborne pathogen. In the stealth scenario, this is not required because the spread happens before people know there is something to interrupt.

probably be able to linger in the environment to catch isolated stragglers

This depends a lot on how much you think a tiny number of isolated stragglers would be able to survive and restart civilization.

either be immune to therapy or vaccination, or move fast enough to obviate them

In the wildfire scenario, this is your second one: moving very fast. In the stealth scenario, we don't know that we need therapy/vaccination until it's too late.

be genetically stable enough that the "kill everybody" variant, as opposed to mutants, is the one that actually spreads

I think this is probably not possible to answer without getting into information hazards. I think the best I can do here is to say that Kevin Esvelt (MIT professor, biologist, CRISPR gene drive inventor, etc) doesn't see this as a blocker.

(for the threat actor you posit) leave off-target species alone

This doesn't seem like much of a barrier to me?

3. If it can exist, you haven't supported the idea that it can be created by intentional design.

This is another one where right now for information hazards reasons the best I can offer is that Esvelt thinks it can.

4. If it can be created by intentional design, you haven't supported the idea that it can be created confidently without large-scale experimentation

Ditto

5. you haven't supported the idea that it can be manufactured or delivered without large resources, in such a way that it will be able to do its job without dying out or falling to countermeasures

This is the scary thing about a pandemic: once it is well seated it spreads on its own through normal human interaction. Most things where you might want to cause similar harm you would need to set up a massive distribution network, but not this.

6. It isn't easy to come up with plausible threat actors who want to kill everybody.

In an LW context I think the easiest actors to imagine are suffering-focused ones. Consider someone who thinks that suffering matters far more than anything else, enough that they'd strongly prefer ending humanity to spreading life beyond earth.

why isn't 95 percent fatality bad enough to worry about? Or even 5 percent?

I also think those are quite bad, and worth working to prevent! And, note that everything I've proposed at the end of the post is the kind of thing that you would also do if you were trying to reduce the risk of something that kills 5%.

But the point I am arguing in the post is that something that might kill everyone, or close enough to end global civilization, is much more likely than you would get from extrapolating historical attacks by small groups.

Bioweapons in general are actually kind of lousy for non-movie-villains at most scales, including large scales, because they're so unpredictable, so poorly controllable, and so poorly targetable.

I don't think those apply for the kind of omnicidal actors I'm covering here?

It would be kind of sidetracking things to get into the reasons why, but just to put it on the record, I have serious doubts about your countermeasures, too.

Happy to get into these too if you like!

Overall, I do think folks who are skeptical of experts who won't share their full reasons or who trust different experts who don't think this is practical should end up with a much more skeptical view than I have. I think we can make some progress as we get a clearer idea of which concepts are too dangerous to share, but probably not enough.

Done.