Bayesian Utility: Representing Preference by Probability Measures

I finally deciphered this post just now so I'll explain how I'm interpreting it for the convenience of future readers. Basically, we start in a world state with various timelines branching off it - points of the initial probability distribution. Each timeline has a particular utility (how much we like it), and a particular probability (how much we expect it). So you can sum utility times probability for all timelines to get the total expected value of this state of the world we're at right now.

However, we have the option of taking some action, the "event" referenced in the post, which rules out some set of timelines. The remaining set of timelines, the ones we can restrict our future to by performing the action, accounts for some proportion of the total expected value of our current state. That proportion is Q(A), derived from summing the expected value of each timeline in the set and dividing by the expected value of this present state - which is the same as normalizing the present state's expected value to 1.

If we perform the action, those timelines keep their probability weights, but in the absence of the other timelines now ruled out, we re-scale them to sum to 1, in the sense of Bayesian updating (our action is evidence that we're in that set of timelines rather than some other set), by dividing by the total proportion of probability mass they had in our initial state (i.e. their total probability), which is P(A).

So, Q(A)/P(A) essentially is like a "score multiplier". If the action restricts the future to a set of timelines whose proportion of total expected value, from the perspective of the pre-action starting state, is greater than their total probability, this normalized expected value of the action will be greater than 1 - we've improved our position, forced the universe into a world state which gives us a better bet than we had before. On the other hand, of course, it could be less than 1 if we restrict to a set of timelines whose density of value proportion per probability is too low - we've thrown away some potential value that was originally available to us.

The fun thing is that since Q and P both look like probability distributions - ways of weighting timelines as proportions of the whole - we can modify them with linear transformations in such a way that the preference ordering of Q(A)/P(A) remains unchanged. But that's where my currently reached understanding stops. I'll have to analyze the rest of the post to get a better sense of how that transformation works and why it would be useful.

I may be missing your point, but to me, it looks like the summary would be:

If you bundle utility with probability, you can do the same maths, which is nice since it simplifies other things. You cannot prefer certain expected outcomes no matter what your utility function is [neat result, btw].

Since the probability math works, I now call the new thing "probability" and show that you can't find prior "probability" (new definition) without considering the normal definition of probability.

This doesn't change anything about regular probability, or finding priors. It just says that you cannot find out what you instrumentally want apriori without knowing your utility function, which is trivially true.

This seems cool but I have a nagging suspicion that this reduces to greater generality and a handful of sentences if you use conditional expectation of the utility function and the Radon-Nikodym theorem?

Why are we concerned with the expected utility of some subset of the probability space? To find the expected utility of an action, you should sum over the products of the utility of the point with its conditional probability given that you take that action, over all points in the space. In effect, you are only considering actions that reduce the probability of some points to zero, and then renormalizes the probability of the remaining points.

Clever! I would have titled it "Couldness and Shouldness", and inserted some sort of pun about "wouldness" at the end.

I don't quite understand the part about mixing. Did you mean 1 >= alpha > beta >= 0 ? If no, some vectors now have negative coordinates and the polar angle becomes an ambiguous ordering. If yes, that's not the general form: why not use any matrix with nonnegative elements and positive determinant?

And I don't understand the last paragraph at all. If X coordinates of points are given, changing the Y coordinates can reorder the polar angles arbitrarily. Or did you simply mean that composite events stay dependent on simple events?

Sorry if those are stupid questions.

A couple of random thoughts. From the point of view on prior+utility as vectors in probability-shouldness coordinates, it's easy to see that the ability to rescale and shift utilities without changing preference corresponds to transformations to the shouldness component. These transformations don't change the order on vectors' (events') angles, and so even if we allow shouldness to go negative, expected utility will still work as preference. Similarly, if the shouldness is fixed positive, one could allow rescaling and shifting probability, so that it, too, can go negative.

Another transformation: if we swap the roles of probability and shouldness, the resulting prior+utility will have shouldness of the original system as prior and inverse utility of the original system as utility. In this system, expected utility minimization will describe the same optimization as the expected utility maximization in the original system. The same effect could be achieved by flipping the sign on utility (another symmetry), which can also be easily seen from the probability-shouldness diagram.

Applying both transformations, we get the same preference, but with shouldness of the original system as prior. Utility of the transformed system is negated inverted utility of the original representation. This shows that conceptually, probability distribution and shouldness distribution are interchangeable.

Added an example of when it isn't possible to specify arbitrary preference for a given prior, and a philosophical note at the end (related to the "where do the priors come from" debate).

I've critiqued this "value is complex" [http://lesswrong.com/lw/y3/value_is_fragile/] material before. To summarise from my objections there:

The utility function of Deep Blue has 8,000 parts - and contained a lot of information. Throw all that information away, and all you really need to reconstruct Deep Blue is the knowledge that it's aim is to win games of chess. The exact details of the information in the original utility function are not recovered - but the eventual functional outcome would be much the same - a powerful chess computer.

The supposed complexity is actually a bunch of implementation details that can be effectively recreated from the goal - if that should prove to be necessary.

It is not precious information that must be preserved. If anything, attempts to preserve the 8,000 parts of Deep Blue's utility function while improving it would actually have a crippling negative effect on its future development. For example, the "look 9 moves ahead" heuristic is a feature when the program is weak, but a serious bug when it grows stronger.

Similarly with complexity of human values: those are a bunch of implementation details to deal with the problem of limited resources - not some kind of representation of the real target.