Best utility normalisation method to date?

post by Stuart_Armstrong · 2019-09-02T18:24:29.318Z · LW · GW · 7 comments

Contents

  Random dictator policy
  Normalising to the random dictator policy
  Normalising a distribution over utilities
  Properties
  Problems? Double counting, or the rich get richer
None
7 comments

For some time, me and others have been looking at ways of normalising utility functions [LW · GW], so that we can answer questions like:

• Suppose that you are uncertain between maximising $U_1$ and $U_2$, what do you do?

...without having to worry about normalising $U_1$ or $U_2$ (since utility functions are only defined up to positive affine transformations).

I've long liked the mean-max normalisation; in this view, what matters is the difference between a utility's optimal policy, and a random policy. So, in a sense, each utility function has a equal shot of moving the outcome away from an expected random policy, and towards themselves.

The intuition still seems good to me, but the "random policy" is a bit of a problem. First of all, it's not all that well defined - are we talking about a policy that just spits out random outputs, or one that picks randomly among outcomes? Suppose there are three options, option A (if A is output), option B' (if B' is output), or do nothing (any other output), should we really say that A happens twice as often as B' (since typing out A randomly is twice as likely that typing out B'?).

Relatedly, if we add another option C, which is completely equivalent to A for all possible utilities, then this redefines the random policy. There's also a problem with branching - what if option A now leads to twenty choices later, while B leads to no further choices, are we talking about twenty-one equivalent choices, or twenty equivalent choices and one other one as likely as all of them put together? Also, the concept has some problem with infinite option sets.

A more fundamental problem is that the random policy includes options that neither $U_1$ nor $U_2$ would ever consider sensible.

Random dictator policy

These problems can be solved by switching instead to the random dictator policy as the default, rather than a random policy.

Assume we are hesitating between utility functions $U_1$, $U_2$, ... $U_n$, with ${\pi }_i^{\ast }$ the optimal policy for utility $U_i$. Then the random dictator policy is just ${\pi }_{rd}$ which picks a ${\pi }_i^{\ast }$ at random and then follows that. So

• ${\pi }_{rd}=\frac{1}{n}{\pi }_1^{\ast }+\frac{1}{n}{\pi }_2^{\ast }+\dots +\frac{1}{n}{\pi }_n^{\ast }$.

Normalising to the random dictator policy

This ${\pi }_{rd}$ is an excellent candidate for replacing the random policy in the normalisation. It is well defined, it would never choose options that all utilities object to, and it doesn't care about how options are labelled or about how to count them.

Therefore we can present the random dictator normalisation: if you are hesitating between utility functions $U_1$, $U_2$, ... $U_n$, then normalise each one to ${\hat{U}}_i$ as follows:

• ${\hat{U}}_i=\frac{U_i}{E_{{\pi }_i^{\ast }}\left[U_i\right]-E_{{\pi }_{rd}}\left[U_i\right]}$,

where $E_{{\pi }_i^{\ast }}\left[U_i\right]$ is the expected utility of $U_i$ given optimal policy, and $E_{{\pi }_{rd}}\left[U_i\right]$ is its expected utility given the random dictator policy.

Our overall utility to maximise then becomes:

• $U=\frac{1}{n}({\hat{U}}_1+{\hat{U}}_2+\dots +{\hat{U}}_n)$.

Note that that normalisation has a singularity when $E_{{\pi }_i^{\ast }}\left[U_i\right]=E_{{\pi }_{rd}}\left[U_i\right]$. But realise what that means: it means that the random dictator policy is optimal for $U_i$. That means that every single ${\pi }_j^{\ast }$ is optimal for $U_i$. So, though the explosion in the normalisation means that we must pick an optimal policy for $U_i$, this set is actually quite large, and we can use the normalisations of the other $U_j$ to pick from among it (so maximising $U_i$ becomes a lexicographic preference for us).

Normalising a distribution over utilities

Now suppose that there is a distribution over the utilities - we're not equally sure of each $U_i$, instead we assign a probability $p_i$ to them. Then the random dictator policy is defined quite obviously as:

• ${\pi }_{rd}=p_1{\pi }_1^{\ast }+p_2{\pi }_2^{\ast }+\dots +p_n{\pi }_n^{\ast }$.

And the normalisation can proceed as before, generating the ${\hat{U}}_i$, and maximising the normalised sum:

• $U=p_1{\hat{U}}_1+p_2{\hat{U}}_2+\dots +p_n{\hat{U}}_n$.

Properties

The random dictator normalisation has all the good properties of the mean-max normalisation in this post [LW · GW], namely that the utility is continuous in the data and that it respects indistinguishable choices. It is also invariant under cloning (ie adding another option that is completely equivalent to one of the options already there), which the mean-max normalisation does not.

But note that, unlike all the normalisations in that post, it is not a case of normalising each $U_i$ without looking at the other $U_j$, and only then combining them. Each normalisation of $U_i$ takes the other $U_j$ into account, because of the definition of the random dictator policy.

Problems? Double counting, or the rich get richer

Suppose we are hesitating between utilities $U_1$ (with $9/10$ probability) and $U_2$ (with $1/10$) probability.

Then ${\pi }_{rd}=\left(9/10\right){\pi }_1^{\ast }+\left(1/10\right){\pi }_1^{\ast }$ is the random dictator policy, and is likely to be closer to optimal for $U_1$ than for $U_2$.

Because of this, we expect $U_1$ to get "boosted" more by the normalisation process than $U_2$ does (since the normalisation is the inverse of the difference between ${\pi }_{rd}$ and the optimal policies).

But then when we take the weighted sum, this advantage is compounded, because the boosted ${\hat{U}}_1$ is weighted $9/10$ versus $1/10$ for the relatively unboosted ${\hat{U}}_2$. It seems that the weight of $U_1$ thus gets double-counted.

A similar phenomena happens when we are equally indifferent between utilities $U_1$, $U_2$, ... $U_{10}$, if the $U_1$, ... $U_9$ all roughly agree with each other while $U_{10}$ is completely different: the similarity of the first nine utilities seems to give them a double boost effect.

There are some obvious ways to fix this (maybe use $\sqrt{p_i}$ rather than $p_i$), but they all have problems with continuity, either when $p_i\to 0$, or when $U_i\to U_j$.

I'm not sure how much of a problem this is.

7 comments

Comments sorted by top scores.