Utility Maximization and Complex Values

The model people seem to have when making your argument is that utility has to be a linear function of our values: e.g. if I value pleasure, kittens, and mathematical knowledge, the way to express that in a utility function is something like 100 x pleasure + 50 x kittens + 25 x math. Obviously, if you then discover that you get a kitten for $1, but a pleasure costs $10 and a math costs $20, you'd just keep maximizing kittens forever to the exclusion of everything else, which is a problem.

Usually (outside the context of utility functions, even) the way we formulate the sentiment that each one of these matters is by taking products: e.g. pleasure^3 x kittens^2 x math (exponents allow us to weight the different values). In this case, while in the short term we might discover that kittens are the most cost-efficient way to higher utility, this does not continue to hold. If we have 100 kittens and only 2 maths, 1 additional kitten increases utility by about 2% while 1 additional math increases utility by 50%.

The fully general definition of utility function is defined over possible outcomes, so if "lots of paper-clips but no staples" makes you unhappy and "lots of staples but no paper-clips" makes you just as unhappy while "lots of paper-clips and lots of staples" makes you very happy, then we can just say you assign 0 utility to the former two and 1 utility to the third.

It may be that some agents have 'fungible' utility, where one unit of X has the same value regardless of how much you already have or how much of anything else you have, but these utility functions form tiny fraction of all possible functions, so if you don't think your own function is of this type then it probably isn't.

Nonlinear (decelerating) utility functions are a good way to get complex results. If I continued to like eating ice cream the same all the time, I might just eat ice cream every day. But instead the amount of utility it's worth depends on how much ice cream I've had recently and some other complicated stuff, so I eat ice cream occasionally and read books occasionally and play games with friends occasionally.

maximize utility is more like a shorthand for saying "maximize the extent to which your values are being fulfilled."

The goal isn't to get utility points, but whatever your actual goals/values are. Utilities are just a way of keeping track. Utility functions are just a way of representing your values in ways that allows certain kinds of analysis to be applied.

If your values are such that you value having outcomes that fulfill a bunch of your values, then outcomes that squash the others at the expense of one would probably have diminishing returns, depending on, well, your values.

Does not follow necessary. A larger plethora of values can be of the greatest value.

Just like a fine dinner of many very good dishes is better than only the ice cream you like the most.

I don't say that it must always be so. But it can be constructed that way.

Diminishing marginal utility, just like it does now.

How can this [isolated utility maximization] be countered?

If the values are faithfully noted this is no problem. It strictly is what you'd want.

I think the danger is that the utility maximizer optimizes for (possibly only slightly) wrong values. The complex value space might accidentally admit some such optimization because of e.g. leaving out some obscure value with was previously overlooked.

A utility optimizer shoud not optimize faster than errors in the value function can be found i.e. faster than humans can give collective feedback on it.

That is one reason companies (mentioned in your comment) sometimes produce goods that are in high demand until the secondary effects are noticed which then may take some time to fix e.g. by legislation.

I don't think the speed of mere companes need to be slowed down to fix this (altough one might consider and model this). But more powerful utility maximizers definitely should time smooth their optimization process.

What if it turns out that one of its complex values can be much more effectively realized and optimized than its other values, i.e. has the best cost-value ratio?

If you're not entirely indifferent to costs, then minimising cost is itself already a value, and so it's already in your value system, and so this "new" knowledge about cost-value ratios changes nothing. Problem solved?

what says utility space must be 1D?

for example I would regard hyperbolic discounting and diminishing returns both as an example of 2D utility.

so doesn't much of this reduce to the novelty dimension of a utility calculation?

"Rational expected-utility-maximizing agents get to care about whatever the hell they want." - a good heuristic to bear in mind. There really are an awful lot of orderings on possible worlds, and if value is complex, your utility function* probably isn't linear.

*usual disclaimers apply about not actually having one.

Relevant post here: Kaj Sotala's Applying utility functions to humans considered harmful.

The utility function only specifies the terminal values. If the utility function is difficult to maximize, then the agent will have to create a complex system of instrumental values. From the inside, terminal values and instrumental values feel pretty similar.

In particular, for a human agent, achieving a difficult goal is likely to involve navigating a dynamic social environment. There will therefore be instrumental social goals which act as stepping-stones to the terminal goal. For neurotypicals, this kind of challenge will seem natural and interesting. For those non-neurotypicals who have trouble socialising, working around this limitation becomes an additional sub-goal.

This isn't just theoretical - I'm describing my own experience since choosing to apply instrumental rationality to a goal system in which one value comes out dominating (for me this turns out to be Friendly AI).

What if it turns out that one of its complex values can be much more effectively realized and optimized than its other values, i.e. has the best cost-value ratio? That value might turn out to outweigh all other values.

Can you think of an example? I plugged in some values for myself:

• A preference for tea and cream scones;
• The preferences associated with making babies;

...but they didn't seem to turn out too well in terms of illustrating your idea.

What exactly do you mean by complex value? Do you mean vector values? (e.g. a complex number can be expressed as a 2-vector).

If you want to maximize utility, then yes, you need a way to compare between two different utility values, and you can only do that if your function outputs a scalar value. If it outputs a vector value, you can't do that unless you assign some function to convert that vector into a scalar. That function can be a 2-norm, p-norm, or really, any type of arbitrary function you like

This is a side note, but if you have complicated desires and live in a complicated world, it's probably not possible to maximize your utility anyway. For instance, let's say you want to be queen of the universe, and you live in an infinite universe, with infinitely many agents in it at infinite levels of power. You may need to make some compromises.

I kind of wonder if a super happy wouldn't have a tendency to just leave an orgasmtron and seek out other forms of novelty after a while, all while having a base state of super-happiness/constant pleasure. It seems to me that even if they are both local maxima, superhappy is a higher peak than wirehead.