Expected Utility, Geometric Utility, and Other Equivalent Representations

post by StrivingForLegibility · 2024-11-20T23:28:21.826Z · LW · GW · 0 comments

Contents

  Expected Utility Functions
  f-Utility Functions
  Equivalence of Maximization
  Duality
    f-Utility Functions Correspond to Expected Utility Functions
    Expected Utility Functions Correspond to f-Utility Functions
  f as a Bijection of Utility Functions
    f is Injective
    f is Surjective
  Composition
  Conclusions
None
No comments

In Scott Garrabrant's [LW · GW] excellent Geometric Rationality [? · GW] sequence, he points out [LW · GW] an equivalence between modelling an agent as

• Maximizing the expected logarithm of some quantity $V$, $E\left[ln\right(V\left)\right]$
• Maximizing the geometric expectation [LW · GW] of $V$, $G\left[V\right]$

And as we'll show in this post, not only can we prove a geometric version of the VNM utility theorem:

• An agent is VNM-rational if and only if there exists a function $V$ that:
  • Represents the agent's preferences over lotteries
    • $L\prec M$ if and only if $V\left(L\right)<V\left(M\right)$
  • Agrees with the geometric expectation of $V$
    • $V=G\left[V\right]$

Which in and of itself is a cool equivalence result, that $E\left[U\right]$ maximization $⟺$ VNM rationality $⟺$ $G\left[V\right]$ maximization. But it turns out these are just two out of a huge family of expectations we can use, like the harmonic expectation $H$, which each have their own version of the VNM utility theorem. We can model agents as maximizing expected utility, geometric utility, harmonic utility, or whatever representation is most natural for the problem at hand.

Expected Utility Functions

The VNM utility theorem is that an agent satisfies the VNM axioms if and only if there exists a utility function $U:\Delta \Omega \to R$ which:

• Represents that agent's preferences over all lotteries
  • $L\prec M$ if and only if $U\left(L\right)<U\left(M\right)$
• Agrees with its expected value
  • $U=E\left[U\right]$

Where $\Delta \Omega$ is a probability distribution over outcomes $\omega \in \Omega$.

The first property is easy to preserve. Given any strictly increasing function $f$,

$f\left(U\right(L\left)\right)<f\left(U\right(M\left)\right)$ if and only if $U\left(L\right)<U\left(M\right)$

So $f\circ U$ also represents our agent's preferences. But it's only affine transformations $f\left(U\right)=aU+b$ that preserve the second property that $f\left(U\right)=E\left[f\right(U\left)\right]$. And it's only increasing affine transformations that preserve both properties at once.

f-Utility Functions

But what if we were interested in other ways of aggregating utilities? One of the central points of Scott's Geometric Rationality [? · GW] sequence is that in many cases, the geometric expectation $G$ is a more natural way to aggregate utility values $V$ into a single representative number $G\left[V\right]$.

We can represent the same preferences using the expected logarithm $E\left[ln\right(V\left)\right]$, but this can feel arbitrary, and having to take a logarithm is a hint that these quantities $V$ are most naturally combined by multiplying them together. The expectation operator $E$ can emulate a weighted product, but the geometric expectation operator $G$ is a weighted product, and we can model an agent as maximizing $G\left[V\right]$ without ever bringing $E$ into the picture.

And as scottviteri [LW · GW] asks [LW(p) · GW(p)]: 

If arithmetic and geometric means are so good, why not the harmonic mean? https://en.wikipedia.org/wiki/Pythagorean_means. What would a "harmonic rationality" look like?

They also link to a very useful concept I'd never seen before: the power mean. Which generalizes many different types of average into one family, parameterized by a power $p$. Set $p=1$ and you've got the arithmetic mean $E$. Set $p=0$ and you've got the geometric mean $G$. And if you set $p=-1$ you've got the Harmonic mean $H$.

It's great! And I started to see if I could generalize my result to other values of $p$. What is the equivalent of $G\left[V\right]=e^{E\left[ln\right(V\left)\right]}$ for $H$? Well, scottviteri [LW · GW] set me down the path towards learning about an even broader generalization of the idea of a mean, which captures the power mean as a special case: the quasi-arithmetic mean or $f$-mean, since it's parameterized by a function $f$.

For our baseline definition, $f:I\to R$ will be a continuous, strictly increasing function that maps an interval $I$ of the real numbers to the real numbers $R$. We're also going to be interested in a weighted average, and in particular a probability weighted average over outcomes $\omega \in \Omega$. We'll use the notation $\omega \sim P$ to denote sampling $\omega$ from the probability distribution $P$.

Here's the definition of the $f$-expectation $M_f$:

$M_f\left[V\right]=f^{-1}\left(E\right[f\circ V\left]\right)$

$M_{f,\omega \sim P}\left[V\right(\omega \left)\right]=f^{-1}\left(E_{\omega \sim P}\right[f\left(V\right(\omega \left)\right)\left]\right)$

Which for finite sets of outcomes looks like:

$M_{f,\omega \sim P}\left[V\right(\omega \left)\right]=f^{-1}\left(\sum _{\omega \in \Omega }P\right(\omega \left)f\right(V\left(\omega \right)\left)\right)$

So for example:

• If $I=R$ and $f\left(V\right)=V$ (or any increasing affine transformation $f\left(V\right)=aV+b$, where $a>0$), then $M_f$ is the arithmetic expectation $M_f=E$.
• If $I=R^{>0}$, the positive real numbers, and $f\left(V\right)=ln\left(V\right)$ (or any logarithm $f\left(V\right)=lo{g}_b\left(V\right)$ where $b>0$ and $b\ne$1), then $M_f$ is the geometric expectation [LW · GW] $M_f=G$.
  • We can also extend $f$ to include $f\left(0\right)=\underset{V\to 0}{lim}f\left(V\right)=-\infty$, to cover applications like Kelly betting [LW · GW]. This will still be a strictly increasing bijection, and I expect it will work with any result that relies on $f$ being continuous.
• If $I=R^{>0}$, and $f\left(V\right)=\frac{-1}{V}$, then $M_f$ is the harmonic expectation $M_f=H$.
  • Using $\frac{1}{V}$ would also compute the harmonic expectation, but $\frac{-1}{V}$ is strictly increasing. And that lets us frame our agent as always maximizing a utility function.
• If $I=R^{>0}$, and $f\left(V\right)=V^p$, then $M_f$ is the power expectation $M_f=M_p$ using the power $p$.

An $f$-utility function $V$ represents an agent's preferences:

$L\prec M$ if and only if $V\left(L\right)<V\left(M\right)$

And agrees with the $f$-expectation of $V$

$V=M_f\left[V\right]$

It turns out that for every $f$-utility function $V$, there is a corresponding expected utility function $U$, and vice versa. We'll prove this more rigorously in the next sections, but it turns out that these are equivalent ways of representing the same preferences.

Equivalence of Maximization

Here's the core insight that powers the rest of this equivalence result, which Scott [LW · GW] articulates here [LW · GW]: 

Maximization is invariant under applying a monotonic function... So every time we maximize an expectation of a logarithm, this was equivalent to just maximizing the geometric expectation.

If we think of an agent as maximizing over $\pi \in \Pi$, something under their control like their action or policy, we can write this as:

$\underset{\pi \in \Pi }{\arg max}G\left[V\right]=\underset{\pi \in \Pi }{\arg max}{e}^{E\left[ln\right(V\left)\right]}$

$\underset{\pi \in \Pi }{\arg max}G\left[V\right]=\underset{\pi \in \Pi }{\arg max}E\left[ln\right(V\left)\right]$

So for every geometric utility function $V$, there is a corresponding expected utility function $U=ln\left(V\right)$ which gives the same result if maximized. 

This equivalence can be generalized to all $f$-utility functions, and it follows from exactly the same reasoning. We have a strictly increasing function $f$, and so $f^{-1}$ must be strictly increasing as well.

• If $V_1<V_2⟹f\left(V_1\right)<f\left(V_2\right)$
• Then $f^{-1}\left(U_1\right)<f^{-1}\left(U_2\right)⟹U_1<U_2$

And so $\arg max$ will ignore either one. Let's use that to simplify the expression for maximizing the $f$-expectation of $V$:

$\underset{\pi \in \Pi }{\arg max}{M}_f\left[V\right]=\underset{\pi \in \Pi }{\arg max}{f}^{-1}\left(E\right[f\circ V\left]\right)$

$\underset{\pi \in \Pi }{\arg max}{M}_f\left[V\right]=\underset{\pi \in \Pi }{\arg max}E[f\circ V]$

And this suggests a substitution that will turn out to be extremely useful:

$U=f\circ V$

$\underset{\pi \in \Pi }{\arg max}{M}_f\left[V\right]=\underset{\pi \in \Pi }{\arg max}E\left[U\right]$

There is a function $U$ whose expectation we can maximize, and this is equivalent to maximizing the $f$-expectation of our $f$-utility function $M_f\left[V\right]$. And we'll show that $U$ is indeed an expected utility function! Similarly, we can apply $f^{-1}$ to both sides to get a suggestion (which turns out to work) for how to turn an expected utility function into an equivalent $f$-utility function.

$f\circ V=U$

$f^{-1}\circ f\circ V=f^{-1}\circ U$

$V=f^{-1}\circ U$

Duality

f-Utility Functions Correspond to Expected Utility Functions

It turns out that for every expected utility function $U$, there is a corresponding $f$-utility function $V$, and vice versa. And this duality is given by $f$ and $f^{-1}$. 

We'll start by showing how to go from $U$ to $V$. Given an expected utility function $U$, we'll define $V$ to be

$V=f^{-1}\circ U$

We know from the VNM expected utility theorem that

$U=E\left[U\right]$

Let's plug both of those into the definition of $M_f$

$M_f\left[V\right]=f^{-1}\left(E\right[f\circ V\left]\right)$

$M_f\left[V\right]=f^{-1}\left(E\right[f\circ f^{-1}\circ U\left]\right)$

$M_f\left[V\right]=f^{-1}\left(E\right[U\left]\right)$

$M_f\left[V\right]=f^{-1}\left(U\right)$

$M_f\left[V\right]=V$

So $V$ agrees with $M_f\left[V\right]$. And since $f$ is strictly increasing, and $U$ represents an agent's preferences, so does $V$.

$L\prec M$ if and only if $V\left(L\right)<V\left(M\right)$

Which means $V$ is an $f$-utility function!

This gives us one half of the VNM theorem for $f$-utility functions. If an agent is VNM rational, their preferences can be represented using an $f$-utility function $V$.

Expected Utility Functions Correspond to f-Utility Functions

We can complete the duality by going the other way, starting from an $f$-utility function $V$ and showing there is a unique corresponding expected utility function $U$. We'll define $U$ to be:

$U=f\circ V$

And we'll plug that and the fact that $V=M_f\left[V\right]$ into the definition of $M_f\left[V\right]$.

$M_f\left[V\right]=f^{-1}\left(E\right[f\circ V\left]\right)$

$M_f\left[V\right]=f^{-1}\left(E\right[U\left]\right)$

$V=f^{-1}\left(E\right[U\left]\right)$

$f\circ V=f\left(f^{-1}\right(E\left[U\right]\left)\right)$

$f\circ V=E\left[U\right]$

$U=E\left[U\right]$

And that's it! Starting from an $f$-utility function $V$, we can apply a strictly increasing function $f$ to get an expected utility function $U$ which represents the same preferences and agrees with $E\left[U\right]$.

This also gives us the other half of the VNM theorem for $f$-utility functions. If an agent's preferences can be represented using an $f$-utility function $V$, they can be represented using an expected utility function $U$, and that agent must therefore be VNM-rational.

f as a Bijection of Utility Functions

So for every $f$-utility function $V$, we can apply $f$ and get an expected utility function $U$. And the same is true in reverse when applying $f^{-1}$. Does this translation process have any collisions in either direction? Are there multiple $f$-utility functions $V$ and $W$ that correspond to the same expected utility function $U$, or vice versa?

It turns out that $f$ creates a one-to-one correspondence between $f$-utility functions and expected utility functions. And a consequence of that is that all of these languages are equally expressive: there are no preferences we can model using an $f$-utility function that we can't model using an expected utility function, and vice versa.

Another way to frame this duality is to say that our translation function $f:I\to R$ is a bijection between its domain $I$ and its image $f\left(I\right)$. And this induces a structure-preserving bijection between utility functions $U:\Delta \Omega \to R$ and $f$-utility functions $V:\Delta \Omega \to I$.

$V=f^{-1}\circ U$

$U=f\circ V$

To show this, we can show that $f$ is injective and surjective between these sets of utility functions.

f is Injective

An injective function, also known as a one-to-one function, maps distinct elements in its domain to distinct elements in its codomain. In other words, injective functions don't have any collisions. So in this case, we want to show that given two distinct $f$-utility functions $V$ and $W$, $f\circ V$ and $f\circ W$ must also be distinct.

$V\ne W⟹f\circ V\ne f\circ W$

Since $V$ and $W$ are distinct $f$-utility functions, they must disagree about some input $\omega$.

$V\left(\omega \right)\ne W\left(\omega \right)$

And since $f$ is strictly increasing, it can't map these different values in $I$ to the same value in $R$.

$f\left(V\right(\omega \left)\right)\ne \left(W\right(\omega \left)\right)$

And thus $f\circ V$ must be a distinct function from $f\circ W$.

f is Surjective

A surjective function maps every element in its domain onto an element of its codomain, and these functions are also called "onto" functions. So in this case, we want to show that given an expected utility function $U$, there is an $f$-utility function $V$ such that $f\circ V=U$. And this is exactly the $f$-utility function that $f^{-1}$ picks out.

$V=f^{-1}\circ U$

$f\circ V=f\circ f^{-1}\circ U$

$f\circ V=U$

And that's it! $f$ induces a one-to-one correspondence between expected utility functions $U$ and $f$-utility functions $V$. We can freely translate between these languages and maximization will treat them all equivalently.

Composition

I also want to quickly show two facts about how $f$-expectations combine together:

• The $f$-expectation of $f$-expectations is another $f$-expectation
  • $M_f\left[M_f\right[V\left]\right]=M_f\left[V\right]$
• The weights combine multiplicatively, as we'd expect from conditional probabilities
  • Analogous to $P(A\wedge B)=P\left(A\right)P\left(B|A\right)$

All of this is going to reduce to an expectation of expectations, so let's handle that first. Let's say we have a family of $n$ probability distributions $P_i\left(\omega \right)$ and expected utility functions $U_i$. And then we sample $i\in [1..n]$ from a probability distribution I'll suspiciously call $\psi$.

$E_{i\sim \psi }\left[E_{\omega \sim P_i\left(\omega \right)}\right[U_i\left(\omega \right)\left]\right]=E_{i\sim \psi }\left[\sum _{\omega \in \Omega }{P}_i\right(\omega \left)U_i\right(\omega \left)\right]$

$E_{i\sim \psi }\left[E_{\omega \sim P_i\left(\omega \right)}\right[U_i\left(\omega \right)\left]\right]=\sum _{i\in [1..n]}{\psi }_i\left(\sum _{\omega \in \Omega }{P}_i\right(\omega \left)U_i\right(\omega \left)\right)$

$E_{i\sim \psi }\left[E_{\omega \sim P_i\left(\omega \right)}\right[U_i\left(\omega \right)\left]\right]=\sum _{i\in [1..n]}\sum _{\omega \in \Omega }{\psi }_i{P}_i\left(\omega \right)U_i\left(\omega \right)$

$E_{i\sim \psi }\left[E_{\omega \sim P_i\left(\omega \right)}\right[U_i\left(\omega \right)\left]\right]=\sum _{(i,\omega )\in [1..n]\times \Omega }{\psi }_i{P}_i\left(\omega \right)U_i\left(\omega \right)$

Taking the expectation over $i$ of the expectation over $P_i\left(\omega \right)$ is equivalent to taking the expectation over pairs $(i,\omega )$.^[1]

$P(i,\omega )={\psi }_i{P}_i\left(\omega \right)$

$E_{i\sim \psi }\left[E_{\omega \sim P_i\left(\omega \right)}\right[U_i\left(\omega \right)\left]\right]=E_{(i,\omega )\sim P(i,\omega )}\left[U_i\right(\omega \left)\right]$

$E\left[E\right[U\left]\right]=E\left[U\right]$

This is one way to frame Harsanyi aggregation [? · GW]. Sample an agent according to a probability distribution $\psi$, then evaluate their expected utility using that agent's beliefs. The Harsanyi score is the expectation of expected utility, and the fact that nested expectations can be collapsed is exactly why aggregating this way satisfies the VNM axioms. The Harsanyi aggregate is VNM rational with respect to the conditional probability distribution $P(i,\omega )$.

Knowing that, the general result for $f$-expectations is even easier:

$M_{f,i\sim \psi }\left[M_{f,\omega \sim P_i\left(\omega \right)}\right[V_i\left]\right]=f^{-1}\left(E_{i\sim \psi }\right[f\circ M_{f,\omega \sim P_i\left(\omega \right)}\left[V_i\right]\left]\right)$

$M_{f,i\sim \psi }\left[M_{f,\omega \sim P_i\left(\omega \right)}\right[V_i\left]\right]=f^{-1}\left(E_{i\sim \psi }\right[f\circ f^{-1}\left(E_{\omega \sim P_i}\right[f\circ V_i\left]\right)\left]\right)$

$M_{f,i\sim \psi }\left[M_{f,\omega \sim P_i\left(\omega \right)}\right[V_i\left]\right]=f^{-1}\left(E_{i\sim \psi }\right[E_{\omega \sim P_i}[f\circ V_i]\left]\right)$

$M_{f,i\sim \psi }\left[M_{f,\omega \sim P_i\left(\omega \right)}\right[V_i\left]\right]=f^{-1}\left(E_{(i,\omega )\sim P(i,\omega )}\right[f\circ V_i\left]\right])$

$M_{f,i\sim \psi }\left[M_{f,\omega \sim P_i\left(\omega \right)}\right[V_i\left]\right]=M_{f,(i,\omega )\sim P(i,\omega )}\left[V_i\right]]$

$M_f\left[M_f\right[V\left]\right]=M_f\left[V\right]]$

So for example, Scott motivates [LW · GW] the idea of Kelly betting as the result of negotiating between different counterfactual versions of the same agent. In that framing, $G\left[V\right]$ naturally captures "Nash bargaining [LW · GW] but weighted by probability." If we geometrically aggregate these geometric expected utilities $G\left[G\right[V\left]\right]$, the result is the same as one big geometrically aggregate over all counterfactual versions of all agents, weighted by $P(i,\omega )={\psi }_i{P}_i\left(\omega \right)$. And the fact that we can model this aggregate as maximizing $G\left[V\right]$ means that it's VNM-rational as well!

This is a very cool framing, and there are some phenomena that I think are easier to understand this way than using the expected utility lens. But since Geometric Rationality is Not VNM Rational [LW · GW], we know that the $G\left[G\right[V\left]\right]$ model won't have all the cool features we want from a theory of geometric rationality, like actively preferring to randomize our actions in some situations.

Conclusions

With all of that under our belt, we can now reiterate the $f$-Utility Theorem for VNM Agents. Which is that an agent is VNM-rational if and only if there exists a function $V:\Delta \Omega \to I$ that:

• Represents the agent's preferences over lotteries
  • $L\prec M$ if and only if $V\left(L\right)<V\left(M\right)$
• Agrees with the $f$-expectation of $V$
  • $V=M_f\left[V\right]$

We can model a VNM-rational agent as maximizing expected utility $E\left[U\right]$, geometric expected utility $G\left[V\right]$, harmonic expected utility $H\left[V\right]$, or any other $f$-expectation $M_f\left[V\right]$ that's convenient for analysis. We can translate these into expected utility functions, but we can also work with them in their native language.

We can also think of this equivalence as an impossibility result. We may want to model preferences that violate the VNM axioms, like a group preference for coin flips in cases where that's more fair than guaranteeing any agent their most preferred outcome. And the equivalence of all these representations means that none of them can model such preferences.

One approach that does work is mixing these expectations together. Our best model of geometric rationality to my knowledge is $G\left[E\right[U\left]\right]$; the geometric expectation of expected utility. See Geometric Rationality is Not VNM Rational [LW · GW] for more details, but the way I'd frame it in this sequence is that the inner expectation means that the set of feasible joint utilities $F$ is always convex.

And maximizing the geometric expectation, aka the geometric aggregate, always picks a Pareto optimum, which is unique as long as all agents have positive weight.

Check out the main Geometric Utilitarianism [LW · GW] post for more details, but I think of these equivalence results as telling us what we can do while staying within the VNM paradigm, and what it would take to go beyond it.

1. ^{^}

   We went through the proof for discrete probability distributions here, but $E\left[E\right[U\left]\right]=E\left[U\right]$ holds for all probability distributions.

0 comments

Comments sorted by top scores.