Instrumental convergence is what makes general intelligence possible

I basically agree with the argument here. I think that approaches to alignment that try to avoid instrumental convergence are generally unlikely to succeed for exactly the reason that this removes the usefulness of AGI.[1] I also agree with jacob_cannell that the terminology choice of "power seeking" is unfortunate and misleading in this regard.

I think this is (at least for me) also one of the core generators of why alignment is so hard: AGI is dangerous for exactly the same reason why it is useful; the danger comes from not one specific kind of failure or one specific module in the model or whatever, but rather the fact that the things we want and don't want fall out of the exact same kind of cognition.

[1]: I do think there exists some work here that might be able to weasel out of this by making use of the surprising effectiveness of less-general intelligence plus the fact that capabilities research mostly pushes this kind of work currently, but this kind of thing has to hinge on a lot of specific assumptions, and I wouldn't bet on it.

The problem is not that we don't know how to prevent power-seeking or instrumental convergence, because we want power-seeking and instrumental convergence.

Yes, this is still underappreciated in most alignment discourse, perhaps because power-seeking has unfortunate negative connotations. A better less loaded term might be Optionality-seeking. For example human friendships increase long term optionality (more social invites, social support, dating and business opportunities, etc), so a human trading some wealth for activities that increase and strengthen friendships can be instrumentally rational for optionality-maximizing empowerment, even though that doesn't fit the (incorrect) stereotype of 'power-seeking'.

The problem is that we don't know how to align this power-seeking, how to direct the power towards what we want, rather than having side-effects that we don't want.

Well if humans are also agents for which instrumental convergence applies, as you suggest here:

Imitation learning is useful due to Aumann's Agreement Theorem and because instrumental convergence also applies to human intelligence

Then that suggests that we can use instrumental convergence to help solve alignment, because optimizing for human empowerment [LW · GW] becomes equivalent to optimizing for our unknown long term values.

There are some caveats of course: we may still need to incorporate some model of short term values like hedonic reward, and it's also important to identify the correct agency to empower which is probably not as simple as individual human brains. Humans are not purely selfish rational but instead are partially altruistic; handling that probably requires something like empowering humanity or generic agency more broadly, or empowering distributed software simulacra minds instead of brains.

I think to maximize legibility to different kinds of people, it helps to include "instrumental convergence usually involves all things converging on the path of accumulating resources (including information) and self-preservation, and, by extension, anything that contributes to self-preservation and accumulating resources". 

Instrumental Convergence might seem like common language, but it certainly is not, and it really should be. Evocative examples increase word count, but they're so helpful that cutting them out in order to lower word count is basically Goodharting. At minimum, it will help an unfamiliar reader quickly and conveniently explain it to others in many casual conversations.

Also, this post is very helpful, I have archived it in my open-source intelligence/research folder and I plan to cite it in the future. The way this post was written is actually the ideal way to explain AI safety to someone for the first time [LW · GW] since it allows a wide variety of intelligent people to really dive into the problem in a way they understand. I recommend submitting it to the AI Safety Public Materials Contest [LW · GW] which I recently confirmed will still read, evaluate, and implement good papers for explaining AI safety to people for the first time. Regardless of whether they have any money to award, they will still credit the author with a ranking that will build credibility.

I basically agree with this post but want to push back a little bit here:

The problem is not that we don't know how to prevent power-seeking or instrumental convergence, because we want power-seeking and instrumental convergence. The problem is that we don't know how to align this power-seeking, how to direct the power towards what we want, rather than having side-effects that we don't want.

Yes, some level of power-seeking-like behavior is necessary for the AI to do impressive stuff. But I don't think that means giving up on the idea of limiting power-seeking. One model could look like this: for a given task, some level of power-seeking is necessary (e.g. to build working nanotech, you need to do a bunch of experiments and simulations, which requires physical resources, compute, etc.). But by default, the solution an optimization process would find might do even more power-seeking than that (killing all humans to ensure they don't intervene, turning the entire earth into computers). This higher level of power-seeking does increase the success probability (e.g. humans interfering is a genuine issue in terms of the goal of building nanotech). But this increase in success probability clearly isn't necessary from our perspective: if humans try to shut down the AI, we're fine with the AI letting itself be shut off (we want that, in fact!). So the argument "we want power-seeking" isn't strong enough to imply "we want arbitrary amounts of power-seeking, and trying to limit it is mis-guided".

I think of this as two complementary approaches to AI safety:

1. Aligning more powerful AI system (that seek power in more ways). Roughly value alignment.
2. Achieving the same tasks with AI systems that are less power-seeking (and are hopefully less risky/easier to align). Roughly corrigibility, trying to find "weak-ish pivotal acts", ...

I see this post as a great write-up for "We need some power-seeking/instrumentally convergent behavior, so AI safety isn't about avoiding that entirely" (a rock would solve that problem, it doesn't seek any power). I just want to add that my best guess is we'll want to do some mix of 1. and 2. above, not just 1. (or at least, we should currently pursuer both strategies, because it's unclear how tractable each one is).

No Free Lunch theorems only apply to a system that is at maximum entropy.  Generally intelligent systems (e.g. AIXI [? · GW]) are possible because the simplicity prior is useful in our own universe (in which entropy is not at a maximum).   Instrumental convergence isn't at the root of intelligence, simplicity is.

As an example, consider two tasks with no common subgoals: say factoring large integers and winning at Go.  Imagine we are trying to find an algorithm that will excel at both of these while running on a Turing machine.  There are no real-world resources to acquire, hence instrumental convergence isn't even relevant.  However an algorithm that assumes a simplicity prior (like AIXI) will still outperform one that doesn't (say sampling all possible Go playing/number factoring algorithms and then picking the one that performs the best).