Yes, AI research will be substantially curtailed if a lab causes a major disaster

I think the real answer is that we don't know what would happen, and there are a variety of possibilities. It's entirely plausible that a "warning shot" would lengthen timelines on net, and entirely plausible that it would shorten timelines on net.

I'm more confident in saying that I don't think a "warning shot" will suddenly move civilization from 'massively failing at the AGI alignment problem' to 'handling the thing pretty reasonably'. If a warning shot shifts us from a failure trajectory to a success trajectory, I expect that to be because we were already very close to a success trajectory at a time.

Good catch on the natural-vs-man-made accidental bait-and-switch in the common argument. This post changed my mind to think that, at least for scaling-heavy AI (and, uh, any disaster that leaves the government standing), regulation could totally help the overall situation.

Well, there's a significant probability COVID isn't a "natural" pandemic, although the story behind that is too complicated without an unambiguous single point of failure which hinders uptake among would-be activists.

If there's an AI failure will things be any different? There may be numerous framings of what went wrong or what might be addressed to fix it, details sufficient to give real predictive power will probably be complicate and it's a good bet that however interested "the powers that be" are in GOF, they're probably much MUCH more interested in AI development. So there can be even more resources to spin the story in favor of forestalling any pressure that might build to regulate.

Nuclear regulation also might not be a good example of a disaster forcing meaningful regulation because the real pressure was against military use of nuclear power and that seems to have enjoyed general immunity against real regulation. So it's more like if an AI incident results in the general public being banned from buying GPUs or something while myriad AI labs still churn toward AGI. 

This argument is important because it is related to a critical assumption in AGI x-risk, specifically with regard to the effectiveness of regulation.

If an AGI can be created by any person, in their living room, with a 10 year old laptop, then regulation is going to struggle to make a difference. Case in point: strong encryption was made illegal (and still is) in various places, and yet, teenagers use Signal and the internet runs on HTTPS.

If, on the other hand, true agent-like AGI turns out to be computationally expensive and requires very specialized hardware to run efficiently, such that only very large corporations can foot the bill to do so (e.g. designing and building increasingly custom hardware like Google TPUs, Nvidia JETSON, Cerebras Wafer, Microsoft / Graphcore IPU, Mythic AMP), then regulation is going to be surprisingly effective, in the same way that it has become stupidly difficult to build new nuclear reactors in the United States, despite advances in safety / efficiency / etc.

I think the natural/manmade comparison between COVID and Three Mile has alot of merit but there are other differences which might explain the difference. Some of them would imply that there would be a strong response to an AI , others less so. 

Local vs global

To prevent nuclear meltdowns you only need to ban them in the US - it doesn't matter what you do elsewhere. This is more complicated for pandemic preparedness.

Active spending vs loss of growth

Its easier to pass a law putting in nuclear regulations which limit growth as this isn't as obvious a loss as spending money from the public purse on measures for pandemics.

Activity of lobbying groups

I get the impression that the anti-nuclear lobby was alot bigger than any pro-pandemic-preparedness lobby. Possibly this is partly caused by the natural vs manmade thing so might be kind of a subpoint.

Tractability of problem

Preventing nuclear disasters seems more tractable than pandemic preparedness

1979 vs 2020

Were our institutions stronger back then?

FWIW I agree that a large AI disaster would cause some strong regulation and international agreements, my concern is more that a small one would not and small ones from weaker AIs seem more likely to happen.

Nice post - seems reasonable.

Minor suggestion to revise the title to something like "Yes,  AI research will be substantially curtailed if a major disaster from AI happens". Before I read your post, I was pretty sure it was going to be about generic disasters, arguing that e.g. a major climate disaster or nuclear disaster would slow down AI research. 

If the media had been able to squarely and emotively pin millions of deaths on some Big Tech AI lab,

Covid may well have been caused by some lab. Partly there is a big difference between AI causes X and it quickly becoming common knowledge that AI caused X.

Suppose some disaster. To be specific, lets say a nuclear warhead detonates in a city.  Some experts blame an AI hacking the control systems, other experts disagree. Some people say this proof AI is dangerous. Other people say its proof that badly secured nukes are dangerous. Some people say humans hackers took control of the nukes. Others say terrorists stole the nukes. Others say it was an accident. The official position is to blame an asteroid impact that hit a truck carrying nuclear waste. A year later, there is a fairly convincing case it was probably an AI, but some experts still disagree. The public is unsure. The disaster itself is no longer news. No one has a clue which AI. 

Interesting post, and I generally agree.

One note - you appear to be quoting David Chapman, not Yudkowsky. The Twitter post you linked to was written by Chapman. It's also not exactly what the tweet says. Can you maybe update to reflect that it's a Chapman quote, or directly link to where Yudkowsky said this? Apologies if I'm missing something obvious in the link.

An AI "warning shot" plays an important role in my finalist entry to the FLI's $100K AI worldbuilding contest [LW · GW]; but civilization only has a good response to the crisis because my story posits that other mechanisms (like wide adoption of "futarchy"-inspired governance) had already raised the ambient wisdom & competence level of civilization.

I think a warning shot in the real world would probably push out timelines a bit by squashing the most advanced projects, but then eventually more projects would come along (perhaps in other countries, or in secret) and do AGI anyways, so I'd be worried that we'd get longer "timelines" but a lower actual chance of getting aligned AI.  For a warning shot to really be net-positive for humanity, it would need to achieve a very strong response, such as the international suppression of all AI research (not just cumbersome regulation on a few tech companies) with a ferocity that meets or exceeds how we currently handle the threat of nuclear proliferation [LW · GW].

Deaths being from natural phenomena seem to just be one factor determining how strong our emotional response to disasters is, and there are plenty of others. People seem to give a greater emotional response if the deaths are flashy, unexpected, instant rather than slow (both in regards to each individual and the length of the disaster), could happen to anyone at any time, and are inversely correlated with age (people care much less if old people die and much more if children do). This would explain why 9/11, school shootings, or shark attacks have a much greater emotional response than covid or the classic comparison of 9/11 to the flu. It would also help if the disaster was international. So a lot probably depends on the circumstances of the AI disaster. 

A new and unfamiliar disaster could also come with fewer preconceptions that the size of the threat is bounded above by previous instances and that we can deal with it with known tools like medicine and vaccines in the case of pandemics. On the other hand, it could have the effect of making people set an upper bound on the possible size of future AI disasters.

It also seems to me like it would be a lot more actionable, easier, and less costly to regulate AI research than to put effective measures in place to prevent future pandemics, so the reluctance should be less.

would at least succeed at introducing extraordinarily imposing costs

How could that work? What does it look like? How can you in practice e.g. ban all GPU clusters? You'd be wiping out a ton of non-AI stuff. If you don't, then AI stuff can just be made to look like non-AI stuff. Just banning "stuff like the stuff that caused That One Big Accident" seems like it doesn't do that much to slow AGI research. 

I think the only way governments could have any hope of preventing AI progress is by actually physically taking control of the factories all over the world that produce GPUs and making them require a license to use or else just discontinuing their manufacture altogether. And that just means people will try to invent other computing substrates to use for it besides GPUs. And I don't think it's very plausible even this will happen. Probably, as other commenters have said, the reaction would be bungled and only make matters worse.

because Covid did not come from U.S. gain-of-function labs

Do you have any particular reason to believe this?

I've never seen that argument you're responding to before. Admittedly, I'm probably only thinking this in hindsight, but it seems like there are a ton of counterarguments, in addition to what you've presented. There isn't a large opposition to OSHA or the USDA.

That being said, I don't agree with the cause being about natural vs anthropogenic problems. I think the difference might be how much of an impact the decisions have on most people (rather than just companies). There's no way I can think of to prove either is correct, and there's certainly more than one factor involved, so a combination of the two is possible. My intuition is that the impact on the general population is a more important distinction.