AGI ruin scenarios are likely (and disjunctive)

(This seems to me to be what many people imagine will happen to the pieces of the AGI puzzle other than the piece they’re most familiar with, via some sort of generalized Gell-Mann amnesia: the tech folk know that the technical arena is in shambles, but imagine that policy has the ball, and vice versa on the policy side. But whatever.)

Just wanted to say that this is also my impression, and has been for months. Technical & Policy both seem to be planning on the assumption that the other side will come through in the end, but the things they imagine this involving seem to be things that the other side thinks is pretty unlikely.

Some people like to tell themselves that surely we'll get an AI warning shot and that will wake people up; but this sounds to me like wishful thinking from the world where the world has a competent response to the pandemic warning shot we just got.

When I think "AI warning shots", the warning shot I'm usually imagining involves the death of 10-50% of working-age and politically relevant people, if not from the shot itself then the social and political upheaval that happened afterwards. The "warning" in "warning shot" is "warning" that the relevant decision makers (congresspeople, etc.) die if the problem remains unsolved, not a few million miscellaneous grandmothers, whose early deaths can safely be ignored in favor of writing more blogs about the culture war. Thus this category of events generally doesn't include flash crashes, some kind of "semi"-advanced computer worm, or an industrial accident that costs us a hundred billion dollars or an economic depression but is then resolved by some heroic engineers, unless one of those happens to instill a fear of AI systems as being personally dangerous to lawmakers.

One specific (unlikely) example, that I myself can elaborate upon in detail, and I think could come before actually existentially threatening systems, is an AI just destroying the internet by doing what regular hackers do on a smaller timetable. It would be within the capabilities of existing not-super-impressive humans, with 100 subjective person-years to burn, to find or copy a couple dozen or so different zero-day bugs ala EternalBlue, write an excellent worm that trivially evades most commercial IDS, and release it and several delayed-release backup versions with different exploits and signatures for another bag of common network services when the initial wave starts to become ineffective. An AGI that wasn't smart enough to build nanobots could still pretty successfully shut down or commandeer an arbitrarily large number of web servers and banks and (internet-adjacent) electrical grids and factories and oil pipelines and self-driving cars all around the same time, and keep them disabled long enough to the point that humans replace them with things that don't use computers because fighting the worms will take too long and people have begun starving. Communities of worms built this way would be capable of consistently destroying almost all important unairgapped computers in an obfuscated and hard-to-debug way, well after the incident response teams working to guard these things manages to figure out that bootkit #3 gets onto the network from IOT toaster #2 on the next office floor. This technique destroys or commandeers lots of "airgapped" subnets too because most things people claim are airgapped outside of a highly specific national security context are only "airgapped" and accidentally connect to public DNS or regularly have some system administrator walking in an "inspected" USB drive with the latest version of Debian or whatever. Maybe some makeshift bombs go off or some second-world nations' poorly protected drones get commandeered too, but I expect not nukes, because those are actually consistently Airgapped.

If it's extremely unlikely to happen (which like any overly detailed story is probably true), I don't think it's because of a reason like "if an AI can do that it can definitely build nanobots". It's not even that difficult to accomplish. A team of I and maybe three other people I personally know could do it right now with maybe ~50% success rate, if we had an absurd amount of prep time inside of a DragonBallZ turnstyle to do it with. The primary reason people haven't launched the worm-nuke already isn't because people aren't smart enough, it's because they don't have the time, there's no motivation for anyone to do anything like this, they fear arrest if they tried to gather a team to coordinate it, and most of all the territory is changing too quickly. And these things - relative speed, cooperative replicas, single-mindedness towards weird goals - are the first critical advantages over regular people I expect early intelligent systems to have. Transformers already write code at an absurd pace compared to living breathing humans, and despite the age-old protestations about how we represent a small window on the intelligence spectrum, so far SOTA models are pretty much steadily climbing across the village idiot to Einstein spectrum in lockstep with log(training cost). Assuming we solve the context window problem a cluster of replicas of DeepMind's private "Codex" implementation will be able to do this in 5-10 years, and it's not clear to me without further assumptions that the cluster would also therefore be capable of doing something instrumentally, existentially threatening on its first critical try.

Needless to say, we are not prepared for all computers to suddenly stop working or act destructively for extended periods of time, in the same way that we are not prepared to go back to hunting and foraging even if that could somehow in theory sustain modern populations. And if it's somehow obvious (say, partly based on the timing of a very public release or an announcement) that DeepMind launched the thing that ended up killing 20% of the populace and/or sends us back to a 1970 standard of living, every former employee of Google's is getting declared guilty by association, whether they're the "ML capabilities engineers" or not. At minimum Demis Hassabis and a bunch of key figures in Google leadership get executed or permanently imprisoned, because the party(ies) directing the rioters or army during ensuing martial law rationally and predictably exploit that opportunity to mobilize a base and look competent and tough. That might happen even if there's a 50% chance they did it because Google is not China and is not a politically inconvenient scapegoat for any major faction in western politics. And this is a sample of the class of events I suggest would have sufficiently shaken things up that coordination on the alignment problem or the delayed-AGI-problem might be possible, not some weaksauce covid-19 pseudo-emergency.

And, lest you wonder what sort of single correlated already-known-to-me variable could make my whole argument and confidence come crashing down around me, it's whether humanity's going to rapidly become much more competent about AGI than it appears to be about everything else.

I conclude from this that we should push on making humanity more competent at everything that affects AGI outcomes, including policy, development, deployment, and coordination. In other times I'd think that's pretty much impossible, but on my model of how AI goes our ability to increase our competence at reasoning, evidence, argumentation, and planning is sufficiently correlated with getting closer to AGI that it's only very hard.

I imagine you think that this is basically impossible, i.e. not worth intervening on. Does that seem right?

If so, I'd guess your reasons are something like this:

1. Any system that can make a big difference in these domains is extremely dangerous because it would need to be better than us at planning, and danger is a function of competent plans. Can't find a reference but it was discussed in one of the 2021 MIRI conversations [? · GW].
2. The coordination problem is too hard. Even if some actors have better epistemics it won't be enough. Eliezer states [LW · GW] this position in AGI ruin:

weaksauce Overton-abiding stuff about 'improving public epistemology by setting GPT-4 loose on Twitter to provide scientifically literate arguments about everything' will be cool but will not actually prevent Facebook AI Research from destroying the world six months later, or some eager open-source collaborative from destroying the world a year later if you manage to stop FAIR specifically.

Does that sound right? Are there other important reasons?

I'm really glad that this post is addressing the disjunctivity of AI doom, as my impression is that it is more of a crux than any of the reasons in https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities. [LW · GW]

Still, I feel like this post doesn't give a good argument for disjunctivity. To show that the arguments for a scenario with no outside view are likely, it takes more than just describing a model which is internally disjunctive. There needs to be some reason why we should strongly expect there to not be some external variables that could cause the model not to apply. 

Some examples of these, in addition to the competence of humanity, are that deep learning could hit a wall for decades, Moore's Law could come to a halt, some anti-tech regulation could cripple AI research, or alignment could turn out to be easy (which itself contains several disjunctive possibilities). I haven't thought about these, and don't claim that any of them are likely, but the possibility of these or other unknown factors invalidating the model prevents me from updating to a very high P(doom). Some of this comes from it just being a toy model, but adding more detail to the model isn't enough to notably reduce the possibility of the model being wrong from unconsidered factors.

A statement I'm very confident in is that no perpetual motion machines will be developed in the next century. I could make some disjunctive list of potential failure modes a perpetual motion machine could encounter, and thus conclude that their development is unlikely, but this wouldn't describe the actual reason a perpetual motion machine is unlikely. The actual reason is that I'm aware of certain laws of physics which prevent any perpetual motion machines from working, including ones with mechanisms wildly beyond my imagination. The outside view is another tool I can use to be very confident: I’m very confident that the next flight I take won’t crash, not because of my model of planes, but because any crash scenario which non-negligible probability would have caused some of the millions of commercial flights every year to crash, and that hasn’t happened.  Avoiding AGI doom is not physically impossible and there is no outside view against it, and without some similarly compelling reason I can’t see how very high P(doom) can be justified.

Seeing "most of it doesn't seem to me to be even trying by their own lights to engage with what look to me like the lethal problems." makes it seem to me that you are confused. The correct lesson isn't that they're unwilling to deal with the serious parts, it's that their evidence is not trivially compatible with your conclusions. In other words, that the balance of evidence they've seem indicates things either aren't so serious, or that there is plenty of time. Reading the link there, it seems like you simply don't like their solutions because you think it is much harder than they do, which should mostly lower your faith in the little bit of evidence we have on AI meaning what you think it does, rather than you putting all the weight of evidence on them not doing the important stuff. You seem very overconfident.

I personally disagree with you both on the difficulty of alignment and on how rapidly AI will become important, with most of the evidence being for the latter ending up a slow thing. Without a singularity beforehand, just increasing the amount of resources spent is quickly becoming infeasible even for the largest companies. The rate at which computing power is going down in cost is itself slowing down quite a lot despite exponential resources being spent on keeping up its scaling. Current narrow AIs need to get exponentially larger for each noticeable gain in ability, so we're pretty much relying on algorithmic enhancements to get high-end AI soon, and at the same time needing to make it more general. Stochastic Gradient Descent is an extremely well understood algorithm, and it is unlikely there is much to be gained from simply coding it better, so we pretty much need to replace the idea of transformers to make quick progress, or possibly even neural-nets themselves. Beyond that, we need to gain a great deal more generality to make an AGI. Once we do (assuming we do), it will still need all of these impediments to just go away for there to be a quick takeoff. These aren't proof of it not happening, but they should vastly lower your confidence.

You need to include a very large term for 'Super-AGI will not happen for obvious reasons' and another very large term for 'Super-AGI doesn't happen for reasons I am unaware of' in the near term (also that it never will), to your currently small term for 'we'll figure out how to avoid misalignment.'.

This is the world where the US federal government's response to COVID was to ban [LW · GW] private COVID testing, 

Private COVID testing, in early 2020, ran the risk of diverting millions of people away from public testing sites, which, in early 2020, may have been the only way for the military to get data about the pandemic (which was poorly understood at the time, and at the time it was right for biodefense officials to be paranoid about it).

confiscate PPE bought by states,

The military had no reason to trust state governments to use all the PPE they ordered in a manner that was optimal for national security, especially if the military was always the first to get solid info about COVID.

and warn [LW · GW] citizens not to use PPE.

They wanted citizens to choose not to buy PPE, which again makes sense with the limited stockpiles of early 2020.

I don't mean to give anyone any hope where hope isn't due. Nate's arguments about incompetence and non-coordination generally hold pretty strongly, including with COVID, even if these aren't the best examples (resource/information hoarding).

However, I think this highlights a common problem among alignment thinkers where Government is assumed to be totally incompetent, when in reality there is an extremely complicated mix of competence disguised as incompetence and incompetence disguised as competence (and, of course, plenty of undisguised incompetence). This especially differs depending on the area; for example, in China during the Cultural Revolution, the nuclear weapons program was relatively untouched by the mass purges that devastated almost everything else in the government and severely disrupted the military. 

Goodhart's Law is not to be taken lightly, and there's tons of people in the National Security Establishment who take Goodhart's Law extremely seriously. Maybe not plenty, certainly not enough, but definitely a lot.

It feels like it is often assumed that the best way to prevent AGI ruin is through AGI alignment, but this isn't obvious to me. Do you think that we need to use AGI to prevent AGI ruin?

Here's a proposal (there are almost certainly better ones): Because of the large amount of compute required to create AGI, governments creating strict regulation to prevent AGI from being created. Of course, the compute to create AGI probably goes down every year, but this buys lots of time, during which one might be able to enact more careful AI regulation, or pull off state-sponsored AGI powered pivotal act project. 

It seems very unlikely that one AI organization will be years ahead of everyone else on the road to AGI, so one of the main policy challenges is to make sure that all of the organizations that could deploy an AGI and cause ruin somehow decide to not do this. The challenge of making all of these organizations not deploy AGI seems easier to pull off than trying to prevent AGI via government regulations, but potentially not by that much, and the benefit of not having to solve alignment seems very large. 

The key downside of this path that I see is that it strictly cuts off the path of using the AGI to perform the pivotal act, because government regulation would prevent that AGI from being built. And this government prevented AGI means that we are still on the precipice -- later AGI ruin might happen that is more difficult to prevent, or another x-risk could happen. But it's not clear to me which path gives a higher likelihood of success. 

You have oversimplified vision on rationality of humanity. You see decisions that are harmful for humanity and conclude that they are irrational. But this logic only works under the assumption that humanity is one individual. Decisions that are harmful for humanity are in most cases beneficial to the decision-making person, and therefore they are not irrational - they are selfish. This gives us much more hope, because persuading a rational selfish person with logic is totally possible. 

Alex Lintz's take: https://forum.effectivealtruism.org/posts/eggdG27y75ot8dNn7/three-pillars-for-avoiding-agi-catastrophe-technical [EA · GW] 

This is not what a community that cares about solving these problems looks like. If the main bottleneck to our survival is the human race suddenly becoming much more competent, why aren't rationalists more evangelistic than Christians? Why is this an insular community?

All the component pieces including plenty of compute are already laying on the table in plain sight. Mostly it’s down to someone understanding how to stitch them together. My optimism relies on AI getting soft failure modes that steer it away from bad things rather than banning the outright.