Homogeneity vs. heterogeneity in AI takeoff scenarios

Nice post, I like the focus on trying to describe what actually happens when AI systems are deployed. However, it's pretty different from my picture of how a takeoff might go. I'll outline some disagreements below.

----

Disagreements about whether takeoff will be "homogenous"

First, you seem to be assuming that there is a single variable that can either be homogenous vs heterogenous. I don't see why this should be the case -- my baseline prediction is that systems are homogenous in algorithms but heterogenous in finetuning data. 

It seems to me that your argument goes like this:

1. Arguments 1-3 suggest that we will have homogeneity in algorithms. (I do not think they support homogeneity in finetuning data.)
2. Implicitly, you then assume that homogeneity in algorithms leads to homogeneity in alignment.
3. Argument 4 suggests that once we have homogeneity in alignment, it will stay that way.

I disagree with step 2 of this argument; I expect alignment depends significantly on how you finetune, and this will likely be very different for AI systems applied to different tasks. See e.g. how GPT-3 is being finetuned for different tasks.

I do still think we will get homogeneity in alignment, but not because of homogeneity in algorithms, but because humanity will put in effort to make sure systems are aligned.

More broadly, I think talking about takeoff as "homogenous" or "heterogenous" is misguided, and you should ~always be saying "homogenous / heterogenous in X".

----

Disagreements about the implications of homogeneity of alignment

Here, I'm going to assume that we do have homogeneity of alignment (despite disagreeing with that position above).

which rules out some of the ways in which the strategy-stealing assumption [AF · GW] might fail.

Huh? Every way that the strategy-stealing assumption might fail is about how misaligned systems with a little bit of power could "win" over a larger coalition of aligned systems with a lot of power. How does homogeneity of alignment change that?

I could see a position where homogeneity of alignment means that there's no point thinking about the strategy-stealing assumption. Either nearly everything is misaligned and we're dead, or nearly everything is aligned and we're fine. This doesn't sound like what you're saying though.

Cooperation and coordination between different AIs is likely to be very easy as they are likely to be very structurally similar to each other if not share basically all of the same weights. As a result, x-risk scenarios involving AI coordination failures or s-risk scenarios involving AI bargaining failures (at least those that don't involve acausal trade) are relatively unlikely.

This seems like it proves too much. Humans are very structurally similar to each other, but still have coordination and bargaining failures. Even in literally identical systems, indexical preferences could still cause conflict to arise.

Maybe you're claiming that AI systems will be way more homogenous than humans, and that they won't have indexical preferences? I'd disagree with both of those claims.

It's unlikely you'll get a warning shot [AF · GW] for deceptive alignment [? · GW], since if the first advanced AI system is deceptive and that deception is missed during training, once it's deployed it's likely for all the different deceptively aligned systems to be able to relatively easily coordinate with each other to defect simultaneously and ensure that their defection is unrecoverable (e.g. Paul's “cascading failures [AF · GW]”).

At a high level, you're claiming that we don't get a warning shot because there's a discontinuity in capability of the aggregate of AI systems (the aggregate goes from "can barely do anything deceptive" to "can coordinate to properly execute a treacherous turn").

I think all the standard arguments against discontinuities can apply just as well to the aggregate of AI systems as they can to individual AI systems, so I don't find your argument here compelling.

In other words, I agree with "discontinuity => fewer warning shots" and disagree with "homogeneity of alignment => discontinuity".

Homogeneity makes the alignment of the first advanced AI system absolutely critical (in a similar way to fast/discontinuous takeoff without the takeoff actually needing to be fast/discontinuous), since whether the first AI is aligned or not is highly likely to determine/be highly correlated with whether all future AIs built after that point are aligned as well. Thus, homogenous takeoff scenarios demand a focus on ensuring that the first advanced AI system is actually sufficiently aligned at the point when it's first built rather than relying on feedback mechanisms after the first advanced AI's development to correct issues.

If you literally condition on "Takeoff will be homogenous in alignment, even across time", then yes, this is an implication. But surely the point "we can rely on feedback mechanisms to correct issues" should make you less convinced that AI systems will be homogenous in alignment across time?

----

Nitpicks

It's also worth noting that a homogenous takeoff doesn't necessarily imply anything about how fast, discontinuous, or unipolar the takeoff might be

What's a heterogenous unipolar takeoff? I would assume you need to have a multipolar scenario for homogenous vs. heterogenous to be an important distinction.

I wonder if there's a really strong outside-view argument that it will be homogenous:

While there are many ways to design flying machines (Balloons, zeppelins, rockets, jets, monoplanes, biplanes, helicopters, ...) at any given era and for any particular domain (say, passenger transport, or air superiority) the designs used tend to be pretty similar. (In WW1 almost all the planes were biplanes, and they almost all used slow but light cloth-on-frame construction, in WW2 all the planes were monoplanes with aluminum or other metal skins and more powerful prop engines, the Me109 and Spitfire and Zero were different but in the grand scheme of things very very similar). Moreover this seems to be the norm throughout history, by stark contrast with science fiction where the spaceships, vehicles, etc. of one faction are often wildly different from those of another. Historically, if we want to find cases of wildly different designs competing with each other, we usually need to look to "First contact" scenarios in which e.g. European armies colonize faraway lands. Perhaps it's just really rare for two dramatically different designs to be almost equally matched in competition, and insofar as they aren't almost equally matched, people quickly realize this and retire the inferior design.

I guess an important question is: If AIs are homogeneous to the same extent that e.g. military fighter planes are, is that sufficient homogeneity to yield your conclusions 1-4? I think so. I think they'll probably have the same architecture and training environment, with only minor details different (e.g. the Chinese GPT-N might have access to more chinese data, might have 1.5x the parameter count, might be trained for 0.5x as long) Of course these details will feel like a big deal in competition, just like the Me109 and Spitfire and Zero had various advantages and disadvantages over each other, butfor purposes of coordination, alignment correlation, etc. they are minor.

Neat post, I think this is an important distinction. It seems right that more homogeneity means less risk of bargaining failure, though I’m not sure yet how much.

Cooperation and coordination between different AIs is likely to be very easy as they are likely to be very structurally similar to each other if not share basically all of the same weights

In what ways does having similar architectures or weights help with cooperation between agents with different goals? A few things that come to mind:

• Having similar architectures might make it easier for agents to verify things about one another, which may reduce problems of private information and inability to credibly commit to negotiated agreements. But of course increased credibility is a double-edged sword as far as catastrophic bargaining failure is concerned, as it may make agents more likely to commit to carrying out coercive threats.
• Agents with more similar architectures / weights will tend to have more similar priors / ways of modeling their counterparts and as well as notions of fairness in bargaining, which reduces risk of bargaining failure [AF · GW] . But as systems are modified or used to produce successor systems, they may be independently tuned to do things like represent their principal in bargaining situations. This tuning may introduce important divergenes in whatever default priors or notions of fairness were present in the initial mostly-identical systems. I don’t have much intuition for how large these divergences would be relative to those in a regime that started out more heterogeneous.
• If a technique for reducing bargaining failure only works if all of the bargainers use it (e.g., surrogate goals), then homogeneity could make it much more likely that all bargainers used the technique. On the other hand, it may be that such techniques would not be introduced until after the initial mostly-identical systems were modified / successor systems produced, in which case there might still need to be coordination on common adoption of the technique.

Also, the correlated success / failure point seems to apply to bargaining as well as alignment. For instance, multiple mesa-optimizers may be more likely under homogeneity, and if these have different mesa-objectives (perhaps due to being tuned by principals with different goals) then catastrophic bargaining failure may be more likely.

Interesting!

It's unlikely for there to exist both aligned and misaligned AI systems at the same time—either all of the different AIs will be aligned to approximately the same degree or they will all be misaligned to approximately the same degree.

Is there an argument that it's impossible to fine-tune an aligned system into a misaligned one? Or just that everyone fine-tuning these systems will be smart and careful and read the manual etc. so that they do it right? Or something else?

I'd very much like to see more discussion of the extent to which different people expect homogenous vs. heterogenous takeoff scenarios

Thinking about it right now, I'd say "homogeneous learning algorithms, heterogeneous trained models" (in a multipolar type scenario at least). I guess my intuitions are (1) No matter how expensive "training from scratch" is, it's bound to happen a second time if people see that it worked the first time. (2) I'm more inclined to think that fine-tuning can make it into "more-or-less a different model", rather than necessarily "more-or-less the same model". I dunno.

For those organizations that do choose to compete... I think it is highly likely that they will attempt to build competing systems in basically the exact same way as the first organization did

...

It's unlikely for there to exist both aligned and misaligned AI systems at the same time

If the first group sunk some cost into aligning their system, but that wasn't integral to its everyday task performance, wouldn't a second competing group be somewhat likely to skimp on the alignment part?

It seems like this calls into the question the claim that we wouldn't get a mix of aligned and misaligned systems.

Do you expect it to be difficult to disentangle the alignment from the training, such that the path of least resistance for the second group will necessarily include doing a similar amount of alignment?

Thanks for this, I for one hadn't thought about this variable much and am convinced now that it is one of the more important variables.

--I think acausal trade stuff means that even if all the AIs on Earth are homogenous, the strategic situation may end up being as if they were heterogenous, at least in some ways. I'm not sure, will need to think more about this.

--You talk about this being possible even for gradual, continuous takeoff, yet you also talk about "The first advanced AI system" as if there is a sharp cutoff between advanced and non-advanced AI. I think this isn't a problem for your overall point, but I'm not sure. For alignment (your point 4) I think this isn't a problem, because you can just rephrase it as "As we gradually transition from non-advanced systems to advanced systems, it is important that our systems be aligned before we near the end of the transition, and more important the closer we get to the end. Because as our systems become more advanced, their alignment properties become more locked-in." For deception, I'm less sure. If systems get more advanced gradually and continuously then maybe we can hope there is a "sordid stumble sweet spot" where systems that are deceptive are likely to reveal this to us in non-catastrophic ways, and thus we are fine because we'll pass through the sweet spot on the way to more advanced AI systems. Or not, but the point is that continuity complicates the point you were making.

If we run two non-communicating copies of the same AI, could it be helpful in detecting failures?