I appreciate this generalization of the results - I think it's a good step towards showing the underlying structure involved here.

One point I want to comment on is transitivity of ${\ge }_{most}^n$, as a relation on induced functions $f:\Theta \to R$. Namely, it isn't, and can even contain cycles of non-equivalent elements. (This came up when I was trying to apply a version of these results, and hoping that ${\ge }_{most}^n$ would be the preference relation I was looking for out of the box.) Quite possibly you noticed this since you give 'limited transitivity' in Lemma B.1 rather than full transitivity, but to give a concrete example:

Let $$V=\left(\begin{array}{ccc}1& 2& 3\\ 3& 1& 2\\ 2& 3& 1\end{array}\right)$$ and $f\left(i|j\right)=V_{ij}$. The permutations are $\sigma \in S_3$ with the usual action on $\{1,2,3\}$. Then we have ^[1] $f_1{\ge }_{most}^2{f}_2{\ge }_{most}^2{f}_3{\ge }_{most}^2{f}_1$ (and $f_2{\ngeqq }_{most}^2{f}_1$). This also works on retargetability directly, with $f$ being $A\stackrel{2}{\to }B$, $B\stackrel{2}{\to }C$, $C\stackrel{2}{\to }A$ retargetable. Notice also that $f$ is invariant under joint permutations (constant diagonals), and I think can be represented as EU-determined, so neither of these save it.

A narrow point is that for a non-transitive relation, I think the notation should be something other than $\ge$ (maybe $\succcurlyeq$).

But more importantly, I think we would really rather a transitive (at least acyclic) relation, if we want to interpret this is 'most $\theta$ prefer' or any kind of preference / aggregation of preferences. If our theorem gives us only an intransitive relation as our conclusion, then we should tweak it.

One way you can do this: aim for a stronger relation like ${\ge }_{\text{o-m}}^n$:

Definition (Orbit-mean dominance?): Let $O_{f,A\ne B}\left(\theta \right)=\{{\theta }^{\prime }\in \text{Orbit}{|}_{\Theta }(\theta ):f(A|{\theta }^{\prime })\ne f(B|{\theta }^{\prime }\left)\right\}$. Write $f\left(B|\theta \right){\ge }_{\text{o-m}}^{n}f\left(A|\theta \right)$ if $\forall \theta :{\sum }_{O_{f,A\ne B}\left(\theta \right)}f\left(B|{\theta }^{\prime }\right)\ge n{\sum }_{O_{f,A\ne B}\left(\theta \right)}f\left(A|{\theta }^{\prime }\right)$.

Since the orbits are under $S_d$ i.e. finite, it's easy to just sum over them. More generally, you could parameterize this with an arbitrary aggregator $g:{\text{Orbits}}_{\Theta }\left(f\right)\to R$ in place of summation; I'm not sure whether this general form or the $\sum$ case should be the focus.

This is transitive for $n=1$ and acyclic for^[2] $n>1$ (consider $\theta$ by $\theta$); and possibly any orbit-based transitive relation is representable in basically this form^[3] (with some $g$), since I'd guess any partial order on sets with cardinality $\le c$ can be represented as a pointwise inequality of functions, but I haven't thought about this too carefully.

With this notion of ${\ge }_{\text{o-m}}^n$, we also need a stronger version of retargetability for the main theorem to hold. For the $\sum$ version, this could be

Definition (scalar-retargetability): Write $f$ is $A\underset{scalar}{\overset{B}{\to }}$ if there exists $\sigma \in S_d$ such that for all $\theta$ with $f\left(A|{\theta }^A\right)-f\left(B|{\theta }^A\right)=c>0$ we have $f\left(B|\sigma {\theta }^A\right)-f\left(A|\sigma {\theta }^A\right)\ge c$ (and likewise multiply scalar-retargetable).

Then scalar-retargetability from $A$ to $B$ will imply $f\left(B|\theta \right){\ge }_{\text{o-m}}^{n}f\left(A|\theta \right)$.

And: I think many (all?) of the main power-seeking results are already secretly in this form. For example, $\theta$-wise comparison of ${\sum }_{{\theta }^{\prime }\in \text{Orbit}{|}_{\Theta }\left(\theta \right)}\text{IsOptimal}(X|C,{\theta }^{\prime })$ gives a preference relation ${\ge }_{\text{o-m}}^n$ identical to the relation ${\ge }_{most}^n$. Assuming this also works for the other rationalities, then the cases we care about were transitive all along exactly because the relations can be expressed in this way.

What do you think?

I'm finally engaging with this after having spent too long afraid of the math. Initial thoughts:

• This result is really impressive and I'm surprised it hasn't been curated. My guess is that it's not presented in the most accessible way, so maybe it deserves a distillation.
• The conclusion isn't as strong or clean as I'd want. It's not clear how to think about orbit-level power-seeking. I'd be excited about a stronger conclusion but wouldn't know how to get it.
• I found the above sentence from the explainer [AF · GW] interesting: "There is no possible way to combine EU-based decision-making functions so that orbit-level instrumental convergence doesn't apply to their composite." Elliott Thornley also has a theorem [LW · GW] deriving nonshutdownability from assumptions like "Indifference to Attempted Button Manipulation: The agent is indifferent between trajectories that differ only with respect to the actions chosen in shutdown-influencing states." Together, maybe these point at a general principle that corrigible agents must care about means, not just ends.
• Some confusions I'm still trying to resolve:
  • Can we say that power-seeking agents will disempower humans? I saw a post in the sequence about POWER in multi-agent games [? · GW].
  • How do AUP agents get around these theorems?
  • If LLMs end up being useful, how do they get around these theorems? Can we get some result where if RLHF has a capabilities component and a power-averseness component, the capabilities component can cause the agent to be power-seeking on net?
  • Can we get a crude measure of how power-seeking agents will be in the real world, especially with the weakened assumptions of this paper?

At some point, I'd be interested in seeing in a distillation of the results in this sequence. (Neither a request nor even a suggestion, but simply an observation about what I think would be nice.)

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

Thanks for sharing. Though can you explain this phrasing in the abstract?:

Eventually, retargetable training procedures may train real-world agents which seek power over humans.

As I understand, agents inherently have some non-zero possibility of seeking power over humans, other agents, etc., by definition.

Very interesting. In general I agree concerns about EU maximisation are subtly misguided, but how would you square this result with Shard Theory? Where does Shard Theory fit in with corrigibility?