Attainable Utility Landscape: How The World Is Changed

post by TurnTrout · 2020-02-10T00:58:01.453Z · LW · GW · 7 comments

Contents

  AU landscape as a unifying frame
  Technical appendix: AU landscape and world state contain equal information
    Possibility isomorphism
    Representation equivalence
  Technical appendix: Opportunity cost
    Notes
None
7 comments

AU landscape as a unifying frame

Attainable utilities are calculated by winding your way through possibility-space, considering and discarding possibility after possibility to find the best plan you can. This frame is unifying.

Sometimes you advantage one AU at the cost of another, moving through the state space towards the best possibilities for one goal and away from the best possibilities for another goal. This is opportunity cost.

Sometimes you gain more control over the future: most of the best possibilities make use of a windfall of cash. Sometimes you act to preserve control over the future: most Tic-Tac-Toe goals involve not ending the game right away. This is power.

Other people usually objectively impact you by decreasing or increasing a bunch of your AUs (generally, by changing your power). This happens for an extremely wide range of goals because of the structure of the environment.

Sometimes, the best possibilities are made unavailable or worsened only for goals very much like yours. This is value impact.

Sometimes a bunch of the best possibilities go through the same part of the future: fast travel to random places on Earth usually involves the airport. This is instrumental convergence.

Exercise: Track what’s happening to your various AUs during the following story: you win the lottery. Being an effective spender, you use most of your cash to buy a majority stake in a major logging company. Two months later, the company goes under.

Technical appendix: AU landscape and world state contain equal information

In the context of finite deterministic Markov decision processes, there's a wonderful handful of theorems which basically say that the AU landscape and the environmental dynamics encode each other. That is, they contain the same information, just with different emphasis. This supports thinking of the AU landscape as a "dual" of the world state.

Let $⟨S,A,T,\gamma ⟩$ be a rewardless deterministic MDP with finite state and action spaces $S,A$, deterministic transition function $T$, and discount factor $\gamma \in (0,1)$. As our interest concerns optimal value functions, we consider only stationary, deterministic policies: $\Pi :=A^S$.

The first key insight is to consider not policies, but the trajectories induced by policies from a given state; to not look at the state itself, but the paths through time available from the state. We concern ourselves with the possibilities available at each juncture of the MDP.

To this end, for $\pi \in \Pi$, consider the mapping of $\pi \mapsto (I-\gamma T^{\pi }{)}^{-1}$ (where $T^{\pi }(s,s^{\prime }):=T(s,\pi (s),s^{\prime })$); in other words, each policy $\pi$ maps to a function mapping each state $s_0$ to a discounted state visitation frequency vector $f_{s_0}^{\pi }$, which we call a possibility. The meaning of each frequency vector is: starting in state $s_0$ and following policy $\pi$, what sequence of states $s_0,s_1,\dots$ do we visit in the future? States visited later in the sequence are discounted according to $\gamma$: the sequence $s_0{s}_1{s}_2{s}_2\dots$ would induce $1$ visitation frequency on $s_0$, $\gamma$ visitation frequency on $s_1$, and $\frac{{\gamma }^2}{1-\gamma }$ visitation frequency on $s_2$.

The possibility function $F\left(s\right)$ outputs the possibilities available at a given state $s$:

Put differently, the possibilities available are all of the potential film-strips of how-the-future-goes you can induce from the current state.

Possibility isomorphism

We say two rewardless MDPs $M$ and $M^{\prime }$ are isomorphic up to possibilities if they induce the same possibilities. Possibility isomorphism captures the essential aspects of an MDP's structure, while being invariant to state representation, state labelling, action labelling, and the addition of superfluous actions (actions whose results are duplicated by other actions available at that state). Formally, $M{\simeq }_F{M}^{\prime }$ when there exists a bijection $\varphi :S\to S^{\prime }$ (letting $P_{\varphi }$ be the corresponding $|S|$-by-$|S^{\prime }|$ permutation matrix) satisfying $F_M\left(s\right)=\{P_{\varphi }{f}^{\prime }|f^{\prime }\in F_{M^{\prime }}(\varphi \left(s\right)\left)\right\}$ for all $s\in S$.

This isomorphism is a natural contender^[1] for the canonical (finite) MDP isomorphism:

Theorem: $M$ and $M^{\prime }$ are isomorphic up to possibilities iff their directed graphs are isomorphic (and they have the same discount rate).

Representation equivalence

Suppose I give you the following possibility sets, each containing the possibilities for a different state:

$$\{\left(\begin{array}{c}4\\ 0\\ 0\end{array}\right),\left(\begin{array}{c}1\\ .75\\ 2.25\end{array}\right),\left(\begin{array}{c}\frac{1}{.4375}\\ 4-\frac{1}{.4375}\\ 0\end{array}\right)\}\left\{\left(\begin{array}{c}0\\ 0\\ 4\end{array}\right)\right\}\{\left(\begin{array}{c}0\\ 1\\ 3\end{array}\right),\left(\begin{array}{c}4-\frac{1}{.4375}\\ \frac{1}{.4375}\\ 0\end{array}\right),\left(\begin{array}{c}3\\ 1\\ 0\end{array}\right)\}$$

Exercise: What can you figure out about the MDP structure? Hint: each entry in the column corresponds to the visitation frequency of a different state; the first entry is always $s_1$, second $s_2$, and third $s_3$.

Theorem: Suppose the rewardless MDP $M$ has possibility function $F$. Given only $F$,^[2] $M$ can be reconstructed up to possibility isomorphism.

In MDPs, the "AU landscape" is the set of optimal value functions for all reward functions over states in that MDP. If you know the optimal value functions for just $|S|$ reward functions, you can also reconstruct the rewardless MDP structure.^[3]

From the environment (rewardless MDP), you can deduce the AU landscape (all optimal value functions) and all possibilities. From possibilities, you can deduce the environment and the AU landscape. From the AU landscape, you can deduce the environment (and thereby all possibilities).

All of these encode the same mathematical object.

Technical appendix: Opportunity cost

Opportunity cost is when an action you take makes you more able to achieve one goal but less able to achieve another. Even this simple world has opportunity cost:

Going to the green state means you can't get to the purple state as quickly.

On a deep level, why is the world structured such that this happens? Could you imagine a world without opportunity cost of any kind? The answer, again in the rewardless MDP setting, is simple: "yes, but the world would be trivial: you wouldn't have any choices". Using a straightforward formalization of opportunity cost, we have:

Theorem: Opportunity cost exists in an environment iff there is a state with more than one possibility.

Philosophically, opportunity cost exists when you have meaningful choices. When you make a choice, you're necessarily moving away from some potential future but towards another; since you can't be in more than one place at the same time, opportunity cost follows. Equivalently, we assumed the agent isn't infinitely farsighted ($\gamma <1$); if it were, it would be possible to be in "more than one place at the same time", in a sense (thanks to Rohin Shah for this interpretation).

While understanding opportunity cost may seem like a side-quest, each insight is another brick in the edifice of our understanding of the incentives of goal-directed agency.

Notes

• Just as game theory is a great abstraction for modelling competitive and cooperative dynamics, AU landscape is great for thinking about consequences: it automatically excludes irrelevant details about the world state. We can think about the effects of events without needing a specific utility function or ontology to evaluate them. In multi-agent systems, we can straightforwardly predict the impact the agents have on each other and the world.
• “Objective impact to a location” means that agents whose plans route through the location tend to be objectively impacted.
• The landscape is not the territory: AU is calculated with respect to an agent's beliefs [LW · GW], not necessarily with respect to what really "could" or will happen.

---

1. The possibility isomorphism is new to my work, as are all other results shared in this post. This apparent lack of basic theory regarding MDPs is strange; even stranger, this absence was actually pointed out in two published papers!

   I find the existing MDP isomorphisms/equivalences to be pretty lacking. The details don't fit in this margin, but perhaps in a paper at some point. If you want to coauthor this (mainly compiling results, finding a venue, and responding to reviews), let me know and I can share what I have so far (extending well beyond the theorems in my recent work on power). ↩︎

2. In fact, you can reconstruct the environment using only a limited subset of possibilities: the non-dominated possibilities. ↩︎

3. As a tensor, the transition function $T$ has size $|A|\cdot |S{|}^2$, while the AU landscape representation only has size $|S{|}^2$. However, if you're just representing $T$ as a transition function, it has size $|A|\cdot |S|$. ↩︎

7 comments

Comments sorted by top scores.