Warning Shots Probably Wouldn't Change The Picture Much

The only viable counterargument I've heard to this is that the government can be competent at X while being incompetent at Y, even if X is objectively harder than Y. The government is weird like that. It's big and diverse and crazy. Thus, the conclusion goes, we should still have some hope (10%?) that we can get the government to behave sanely on the topic of AGI risk, especially with warning shots, despite the evidence of it behaving incompetently on the topic of bio risk despite warning shots.

Or, to put it more succinctly: The COVID situation is just one example; it's not overwhelmingly strong evidence.

(This counterargument is a lot more convincing to the extent that people can point to examples of governments behaving sanely on topics that seem harder than COVID. Maybe Y2K? Maybe banning bioweapons? Idk, I'd be interested to see research on this: what are the top three examples we can find, as measured by a combination of similarity-to-AGI-risk and competence-of-government-response.)

I broadly agree with this general take, though I'd like to add some additional reasons for hope:

1. EAs are spending way more effort and money on AI policy. I don't have exact numbers on this, but I do have a lot of evidence in this direction: at every single EAG, there are far more people interested in AI x-risk policy than biorisk policy, and even those focusing on biorisk are not really focusing on preventing gain-of-function (as opposed to say, engineered pandemics or general robustness). I think this is the biggest reason to expect that AI might be different.

I also think there's some degree of specialization here, and having the EA policy people all swap to biorisk would be quite costly in the future. So I do sympathize with the majority of AI x-risk focused EAs doing AI x-risk stuff, as opposed to biorisk stuff. (Though I also do think that getting a "trial run" in would be a great learning experience.)

2. Some of the big interventions that people want are things governments might do anyways. To put it another way, governments have a lot of inertia. Often when I talk to AI policy people, the main reason for hope is that they want the government to do something that already has a standard template, or is something that governments already know how to do. For example, the authoritarian regimes example you gave, especially if the approach is to dump an absolute crapton of money on compute to race harder or to use sanctions to slow down other countries. Another example people talk about is having governments break up or nationalize large tech companies, so as to slow down AI research. Or maybe the action needed is to enforce some "alignment norms" that are easy to codify into law, and that the policy teams of industry groups are relatively bought into. 

The US government already dumps a lot of money onto compute and AI research, is leveling sanctions vs China, and has many Senators that are on board for breaking up large tech companies. The EU already exports its internet regulations to the rest of the world, and it's very likely that it'd export its AI regulations as well. So it might be easier to push these interventions through, than it is to convince the government not to give $600k to a researcher to do gain-of-function, which is what they have been doing for a long time. 

(This seems like how I'd phrase your first point. Admittedly, there's a good chance I'm also failing the ideological Turing test on this one.)
 
3. AI is taken more seriously than COVID. I think it's reasonable to believe that the US government takes AI issues more seriously than COVID---for example, it's seen as more of a national security issue (esp wrt China), and it's less politicized. And AI (x-risk) is an existential threat to nations, which generally tends to be taken way more seriously than COVID is. So one reason for hope is that policymakers don't really care about preventing a pandemic, but they might actually care about AI, enough that they will listen to the relevant experts and actually try. To put it another way, while there is a general factor of sanity that governments can have, there's also tremendous variance in how competent any particular government is on various tasks. (EDIT: Daniel makes a similar point above.)

4. EAs will get better at influencing the government over time. This is similar to your second point. EAs haven't spent a lot of time trying to influence politics. This isn't just about putting people into positions of power---it's also about learning how to interface with the government in ways that are productive, or how to spend money to achieve political results, or how to convince senior policymakers. It's likely we'll get better at influence over time as we learn what and what not to do, and will leverage our efforts more effectively. 

For example, the California Yimbys were a lot worse at interfacing with the state government or the media effectively when they first started ~10 years ago. But recently they've had many big wins in terms of legalizing housing!

(That being said, it seems plausible to me that EAs should try to get gain-of-function research banned as a trial run, both because we'd probably learn a lot doing it, and because it's good to have clear wins [LW · GW].)

Though I'm unsure whether warning shots will ever even occur, my primary hope with warning shots has always just been that they would change the behavior of big AI labs that are already sensitive to AI risk (e.g. OpenAI and DeepMind), not that they would help us achieve concrete governmental policy wins.

This post seems to make an implicit assumption that the purpose of a warning shot is to get governments to do something. I usually think of a warning shot as making it clear that the risk is real, leading to additional work on alignment and making it easier for alignment advocates to have AGI companies implement specific alignment techniques. I agree that a warning shot is not likely to substitute for a technical approach to alignment.

(EDIT: Whoops, I see Evan made basically this comment already)

Disclaimer: writing quickly. 

Consider the following path: 

A. There is an AI warning shot. 

B. Civilization allocates more resources for alignment and is more conservative pushing capabilities.  

C. This reallocation is sufficient to solve and deploy aligned AGI before the world is destroyed. 

I think that a warning shot is unlikely (P(A) < 10%), but won't get into that here. 

I am guessing that P(B | A) is the biggest crux. The OP primarily considers the ability of governments to implement policy that moves our civilization further from AGI ruin, but I think that the ML community is both more important and probably significantly easier to shift than government.  I basically agree with this post as it pertains to government updates based on warning shots. 

I anticipate that a warning shot would get most capabilities researchers to a) independently think about alignment failures and think about the alignment failures that their models will cause, and b) take the EA/LessWrong/MIRI/Alignment sphere's worries a lot more seriously. My impression is that OpenAI seems to be much more worried about misuse risk than accident risk: if alignment is easy, then the composition of the lightcone is primarily determined by the values of the AGI designers.  Right now, there are ~100 capabilities researchers vs ~30 alignment researchers at OpenAI [LW · GW]. I think a warning shot would dramatically update them towards worry towards worry about accident risk, and therefore I anticipate that OpenAI would drastically shift most of their resources to alignment research. I would guess P(B|A) ~= 80%. 

 P(C | A, B) primarily depends on alignment difficulty, of which I am pretty uncertain, and also how large the reallocation in B is, which I am anticipating to be pretty large. The bar for destroying the world gets lower and lower every year, but this would give us a lot more time, but I think we get several years of AGI capabiliity before we deploy it. I'm estimating P(C | A, B) ~= 70%, but this is very low resilience. 

Written and forecasted quickly, numbers are very rough. Thomas requested I make a forecast before anchoring on his comment [LW(p) · GW(p)] (and I also haven't read others).

I’ll make a forecast for the question:  What’s the chance a set of >=1 warning shots counterfactually tips the scales between doom and a flourishing future, conditional on a default of doom without warning shots?

We can roughly break this down into:

1. Chance >=1 warning shots happens
2. Chance alignment community / EA have a plan to react to warning shot well
3. Chance alignment community / EA have enough influence to get the plan executed
4. Chance the plan implemented tips the scales between doom and flourishing future

I’ll now give rough probabilities:

1. Chance >=1 warning shots happens: 75%
   1. My current view on takeoff is closer to Daniel Kokotajlo-esque fast-ish takeoff than Paul-esque slow takeoff. But I’d guess even in the DK world we should expect some significant warning shots, we just have less time to react to them.
   2. I’ve also updated recently toward thinking the “warning shot” doesn’t necessarily need to be that accurate of a representation of what we care about to be leveraged. As long as we have a plan ready to react to something related to making people scared of AI, it might not matter much that the warning shot accurately represented the alignment community’s biggest fears.
2. Chance alignment community / EA have a plan to react to warning shot well: 50%
   1. Scenario planning is hard, and I doubt we currently have very good plans. But I think there are a bunch of talented people working on this, and I’m planning on helping :)
3. Chance alignment community / EA have enough influence to get the plan executed: 35%
   1. I’m relatively optimistic about having some level of influence, seems to me like we’re getting more influence over time and right now we’re more bottlenecked on plans than influence. That being said, depending on how drastic the plan is we may need much more or less influence. And the best plans could potentially be quite drastic.
4. Chance the plan implemented tips the scales between doom and flourishing future, conditional on doom being default without warning shots: 5%
   1. This is obviously just a quick gut-level guess; I generally think AI risk is pretty intractable and hard to tip the scales on even though it’s super important, but I guess warning shots may open the window for pretty drastic actions conditional on (1)-(3).
       

Multiplying these all together gives me 0.66%, which might sound low but seems pretty high in my book as far as making a difference on AI risk is concerned.

I totally sympathize with and share the despair that many people feel about our governments' inadequacy to make the right decisions on AI, or even far easier issues like covid-19.

What I don't understand is why this isn't paired with a greater enthusiasm for supporting governance innovation/experimentation, in the hopes of finding better institutional structures that COULD have a fighting chance to make good decisions about AI.

Obviously "fix governance" is a long-term project and AI might be a near-term problem. But I still think the idea of improving institutional decision-making could be a big help in scenarios where AI takes longer than expected or government reform happens quicker than expected. In EA, "improving institutional decisionmaking" has come to mean incremental attempts to influence existing institutions by, eg, passing weaksauce "future generations" climate bills. What I think EA should be doing much more is supporting experiments with radical Dath-Ilan-style institutions (charter cities, liquid democracy, futarchy, etc) in a decentralized hits-based way, and hoping that the successful experiments spread and help improve governance (ie, getting many countries to adopt prediction markets and then futarchy) in time to be helpful for AI.

I've written much more about this in my prize-winning entry to the Future of Life Institute's "AI worldbuilding competition" (which prominently features a "warning shot" that helps catalyze action, in a near-future where governance has already been improved by partial adoption of Dath-Ilan-style institutions), and I'd be happy to talk about this more with interested folks: https://www.lesswrong.com/posts/qo2hqf2ha7rfgCdjY/a-bridge-to-dath-ilan-improved-governance-on-the-critical [LW · GW]

I said things like: if you can't get the world to coordinate on banning gain-of-function research, in the wake of a trillions-of-dollars tens-of-millions-of-lives pandemic "warning shot", then you're not going to get coordination in the much harder case of AI research.

To be clear I largely agree with you, but I don't think you've really steel-manned (or at least accurately modeled) the government's decision making process.

We do have an example of a past scenario where:

• a new technology of enormous, potentially world-ending impact was first publicized/predicted in science fiction
• a scientist actually realized the technology was near-future feasible, convinces others
• western governments actually listened to said scientists
• instead of coordinating on a global ban of the technology - they instead fast tracked the tech's development

The tech of course is nuclear weapons, the sci-fi was "The World Set Free" by HG Wells, the first advocate scientist was Szilard, but nobody listened until he recruited Einstein.

So if we apply that historical lesson to AI risk ... the failure (so far) seems two fold:

• failure on the "convince a majority of the super high status experts"
• and perhaps that's good! Because the predictable reaction is tech acceleration, not coordination on deacceleration

AGI is coup-complete.

The problem with banning risky gain-of-function research seems to be that, surprisingly, most people don't understand that it's risky and although there's a general feeling that COVID-19 may have originated in the lab at Wuhan, we don't have definitive evidence.

So the people who are trusted on this are incentivized to say that it's safe, and there also seems to be something of a coverup going on.

I suspect that if there was some breakthrough that allowed us to find strong technical evidence of a lab origin, there would be significant action on this.

But, alas, we're trapped in a situation where lack of evidence stalls any effort to seriously look for evidence.

(Which is then subject to a bunch of the usual skepticism that applies to arguments of the form "surely my political party will become popular, claim power, and implement policies I like".)

In my experience these types of arguments are actually anti-signals. 

Incidentally, has anyone considered the possibility of raising funds for an independent organization dedicated to discovering the true origin of covid-19? I mean a serious org with dozens of qualified people for up to a decade.

It seems like an odd thing to do but the expected value of proving a lab origin for covid-19 is extremely high, perhaps half a percentage point x-risk reduction or something.

1. Perhaps the sorts of government interventions needed to make AI go well are not all that large, and not that precise.

I confess I don't really understand this view.

Specifically for the sub-claim that "literal global cooperation" is unnecessary, I think a common element of people's views is that: the semiconductor supply chain has chokepoints in a few countries, so action from just these few governments can shape what is done with AI everywhere (in a certain range of time).

Banning gain-of-function research would be a mistake. What would be recklessly foolish is incentivising governments to decide what avenues of research are recklessly foolish. The fact that governments haven't prohibited it in a panic bout (not even China that otherwise did a lot of panicky things) is a good testament of their abilities, not an inability to react to warning shots.

The reaction seems consistent if people (in government) believe no warning shot was fired. AFAIK the official reading is that we experienced a zoonosis, so banning gain of function research would go against that narrative. It seems true to me that this should be seen as a warning shot, but smallpox and ebola could have prompted this discussion as well and also failed to be seen as a warning shot. 

I think this is probably true; I would assign something like a 20% chance of some kind of government action in response to AI aimed at reducing x-risk, and maybe a 5-10% chance that it is effective enough to meaningfully reduce risk.  That being said, 5-10% is a lot, particularly if you are extremely doomy.  As such, I think it is still a major part of the strategic landspace even if it is unlikely.

Has EA invested much into banning gain-of-function research? I've heard about Alvea and 1DaySooner, but no EA projects aimed at gain-of-function. Perhaps the relevant efforts aren't publicly known, but I wouldn't be shocked if more person-hours have been invested in EA community building in the past two years (for example) than banning gain-of-function research.

Obviously, governments don't believe in autonomous AI risk, only in the risk that AI can be used to invent more powerful weapons. 

In the government's case, that doubt may come from their experience that vastly expensive complex systems are always maximally dysfunctional, and require massive teams of human experts to accomplish a well-defined but difficult task.