When Most VNM-Coherent Preference Orderings Have Convergent Instrumental Incentives

post by TurnTrout · 2021-08-09T17:22:24.056Z · LW · GW · 4 comments

Contents

    Intuition
  Formalism
  Implications
    The quest for better convergence theorems
None
4 comments

This post explains a formal link between "what kinds of instrumental convergence exists?" and "what does VNM-coherence tell us about goal-directedness?". It turns out that VNM coherent preference orderings have the same statistical incentives as utility functions; most such orderings will incentivize power-seeking in the settings covered by the power-seeking theorems [? · GW]. 

In certain contexts, coherence theorems can have non-trivial implications, in that they provide Bayesian evidence about what the coherent agent will probably do. In the situations where the power-seeking theorems apply, coherent preferences do suggest some degree of goal-directedness. Somewhat more precisely, VNM-coherence is Bayesian evidence that the agent prefers to stay alive, keep its options open, etc. 

However, VNM-coherence over action-observation histories tells you nothing [LW · GW] about what behavior to expect from the coherent agent, because there is no instrumental convergence for generic utility functions over action-observation histories [LW · GW]!

Intuition

The result follows because the VNM utility theorem lets you consider VNM-coherent preference orderings to be isomorphic to their induced utility functions (with equivalence up to positive affine transformation), and so these preference orderings will have the same generic incentives as the utility functions themselves.

Formalism

Let $o_1,...,o_n$ be outcomes, in a sense which depends on the context; outcomes could be world-states, universe-histories, or one of several fruits. Outcome lotteries are probability distributions over outcomes, and can be represented as elements of the $n$-dimensional probability simplex (ie as element-wise non-negative unit vectors). 

A preference ordering $\prec$ is a binary relation on lotteries; it need not be eg complete (defined for all pairs of lotteries). VNM-coherent preference orderings are those which obey the VNM axioms. By the VNM utility theorem, coherent preference orderings induce consistent utility functions over outcomes, and consistent utility functions conversely imply a coherent preference ordering.

Definition 1: Permuted preference ordering. Let $\varphi \in S_n$ be an outcome permutation, and let $\prec$ be a preference ordering. ${\prec }_{\varphi }$ is the preference ordering such that for any lotteries $L,M$: $L{\prec }_{\varphi }M$ if and only if $\varphi \left(L\right)\prec \varphi \left(M\right)$.

EDIT: Thanks to Edouard Harris for pointing out that Definition 1 and Lemma 3 were originally incorrect.

Definition 2: Orbit of a preference ordering. Let $\prec$ be any preference ordering. Its orbit $S_n\cdot \prec$ is the set $\{{\prec }_{\varphi }\mid \varphi \in S_n\}$. 

The orbits of coherent preference orderings are basically all the preference orderings induced by "relabeling" which outcomes are which. This is made clear by the following result:

Lemma 3: Permuting coherent preferences permutes the induced utility function. Let $\prec$ be a VNM-coherent preference ordering which induces VNM-utility function $u$, and let $\varphi \in S_n$. Then ${\prec }_{\varphi }$ induces VNM-utility function $u^{\prime }\left(o_i\right)=u\left(\varphi \right(o_i\left)\right)$, where $o_i$ is any outcome.

Proof. Let $L,M$ be any lotteries. 

1. By the definition of a permuted preference ordering, $L{\prec }_{\varphi }M$ if and only if $\varphi \left(L\right)\prec \varphi \left(M\right)$.
2. By the VNM utility theorem and the fact that $\prec$ is coherent, $\varphi \left(L\right)\prec \varphi \left(M\right)$ iff $E_{\ell \sim \varphi \left(L\right)}\left[u\right(\ell \left)\right]<E_{m\sim \varphi \left(M\right)}\left[u\right(m\left)\right]$.
3. Since there are finitely many outcomes, we convert to vector representation: $u^{\top }\left(P_{\varphi }l\right)<u^{\top }\left(P_{\varphi }m\right)$.
4. By associativity, $\left(u^{\top }{P}_{\varphi }\right)l<\left(u^{\top }{P}_{\varphi }\right)m$.
5. But this is just equivalent to $E_{\ell \sim L}\left[u\right(\varphi \left(\ell \right)\left)\right]<E_{m\sim M}\left[u\right(\varphi \left(m\right)\left)\right]$.

QED.

As a corollary, this lemma implies that if $\prec$ is VNM-coherent, so is ${\prec }_{\varphi }$, since it induces a consistent utility function over outcomes.

Consider the orbit of any $\prec$. By the VNM utility theorem, each preference ordering can be considered isomorphic to its induced utility function (with equivalence up to positive affine transformation). 

Then let $u$ be any utility function compatible with $\prec$. By the above lemma, consider the natural bijection between the (preference ordering) orbit of $\prec$ and the (utility function) orbit of $u$, where $\{{\prec }_{\varphi }\mid \varphi \in S_n\}\leftrightarrow \{u\circ \varphi \mid \varphi \in S_n\}$.${}^{\text{Footnote}\text{representative}}$ 

When my theorems [? · GW] on power-seeking [LW · GW] are applicable, some proportion of the right-hand side is guaranteed to make (formal) power-seeking optimal. But by the bijection and by the fact that the preference orderings incentivize the same things (by the VNM theorem in the reverse direction), the (preference ordering) orbit must have the exact same proportion of elements for which (lotteries representing formal) power-seeking are optimal. 

Conversely, if we know that some set A of lotteries tends to be preferred over another set B of lotteries (in the preference order orbit sense), then the same argument shows that A tends to have greater expected utility than B (in the utility function orbit sense). This holds for all (utility function) orbits, because every utility function corresponds to a VNM-coherent preference ordering.

So: orbit-level instrumental convergence for utility functions is equivalent to orbit-level instrumental convergence for VNM-coherent preference orderings. 

Implications

• Instrumental convergence does not exist when maximizing expected utility over action observation histories (AOH) [LW · GW].
  • Therefore, VNM-coherence over action observation history lotteries tells you nothing [LW · GW] about what behavior to expect from the agent.
  • Coherence over AOH tells you nothing because there is no instrumental convergence in that setting!
• In certain contexts, coherence theorems can have non-trivial implications, in that they provide Bayesian evidence about what the coherent agent will probably do.
  • In the situations where the power-seeking theorems apply, coherent preferences do suggest some degree of goal-directedness.
  • Somewhat more precisely, VNM-coherence is Bayesian evidence that the agent prefers to stay alive, keep its options open, etc.
• In some domains, preference specification may be more natural than utility function specification. However, in theory, coherent preferences and utility functions have the exact same statistical incentives.
  • In practice, they will differ. For example, suppose we have a choice between specifying a utility function which is linear over state features, or of doing behavioral cloning on elicited human preferences over world states. These two methods will probably tend to produce different incentives.

The quest for better convergence theorems

Goal-directedness seems to more naturally arise from coherence over resources. (I think the word 'resources' is slightly imprecise here, because resources are only resources in the normal context of human life; money is useless when alone in Alpha Centauri, but time to live is not. So we want coherence over things-which-are-locally-resources, perhaps.)

In his review of Seeking Power is Often Convergently Instrumental in MDPs [? · GW], John Wentworth wrote [? · GW]:

in a real-time strategy game, units and buildings and so forth can be created, destroyed, and generally moved around given sufficient time. Over long time scales, the main thing which matters to the world-state is resources - creating or destroying anything else costs resources. So, even though there's a high-dimensional game-world, it's mainly a few (low-dimensional) resource counts which impact the long term state space. Any agents hoping to control anything in the long term will therefore compete to control those few resources.

More generally: of all the many "nearby" variables an agent can control, only a handful (or summary) are relevant to anything "far away". Any "nearby" agents trying to control things "far away" will therefore compete to control the same handful of variables.

Main thing to notice: this intuition talks directly about a feature of the world - i.e. "far away" variables depending only on a handful of "nearby" variables. That, according to me, is the main feature which makes or breaks instrumental convergence in any given universe. We can talk about that feature entirely independent of agents or agency. Indeed, we could potentially use this intuition to derive agency, via some kind of coherence theorem; this notion of instrumental convergence is more fundamental than utility functions.

In his review of Coherent decisions imply consistent utilities [LW · GW], John wrote [LW · GW]:

"resources" should be a derived notion rather than a fundamental one. My current best guess at a sketch: the agent should make decisions within multiple loosely-coupled contexts, with all the coupling via some low-dimensional summary information - and that summary information would be the "resources". (This is exactly the kind of setup which leads to instrumental convergence.) By making pareto-resource-efficient decisions in one context, the agent would leave itself maximum freedom in the other contexts. In some sense, the ultimate "resource" is the agent's action space. Then, resource trade-offs implicitly tell us how the agent is trading off its degree of control within each context, which we can interpret as something-like-utility.

This seems on-track to me. We now know what instrumental convergence looks like in unstructured environments [LW · GW], and how structural assumptions on utility functions affect the shape and strength of that instrumental convergence [LW · GW], and this post explains the precise link between "what kinds of instrumental convergence exists?" and "what does VNM-coherence tell us about goal-directedness?". I'd be excited to see what instrumental convergence looks like in more structured models [LW(p) · GW(p)].

---

Footnote representative: In terms of instrumental convergence, positive affine transformation never affects the optimality probability [LW · GW] of different lottery sets. So for each (preference ordering) orbit element ${\prec }_{\varphi }$, it doesn't matter what representative we select from each equivalence class over induced utility functions — so we may as well pick $u\circ \varphi$!

4 comments

Comments sorted by top scores.