The "Measuring Stick of Utility" Problem

Very very good question IMO. Thank you for this.

Consider a person who comes to a very clear understanding of the world, such that they are extremely capable of designing things, building things, fundraising, collaborating, and so on. Consider a moment where this person is just about to embark on a project but has not yet acquired any resources, perhaps has not even made any connections with anyone at all, yet is highly likely to succeed in their project when they do embark. Would you say this person has “resources”? If not, there is a kind of continuous trading-in that will take place as this person exchanges their understanding of things for resources, then later for final outcomes. Is there really a line between understanding-of-things, resources, and outcomes? The interesting part is the thing that gives this person power, and that seems to be their understanding-of-things.

I came up with a handful of ideas, but one problem my ideas ran into was that strictly speaking, utility is not monotonically increasing with resources. Rather, utility is monotonically increasing with resources under your control; if a resource is controlled by an enemy, then that decreases your expected utility.

So that raises the question of whether we should define "resources" abstractly/generally, or "resources under your control" specifically. I suspect the former could be derived from the latter by considering something like causality or analogy. And the latter would need to start out with defining "you", which could presumably be done in the style of your "optimization at a distance".

Money is an abstraction. A useful abstraction of low complexity that people can coordinate on. Calling it a resource like energy makes sense to me on that abstraction level. But there are Weird Things About Money [LW · GW]. People are optimizing relative wealth, not absolute wealth, and happiness is proportionate to wealth relative to peers (Envy Dominates Greed). Thus coordinating on this specific resource is sub-optimal globally even if it is efficient for individual participants locally. 

The same should hold for energy as long as you can convert money to energy.

Another way to put it: coherence theorems assume the existence of some resources (e.g. money), and talk about systems which are pareto optimal with respect to those resources - e.g. systems which “don’t throw away money”. Implicitly, we're assuming that the system generally "wants" more resources (instrumentally, not necessarily as an end goal), and we derive the system's "preferences" over everything else (including things which are not resources) from that. The agent "prefers" X over Y if it expends resources to get from Y to X. If the agent reaches a world-state which it could have reached with strictly less resource expenditure in all possible worlds, then it's not an expected utility maximizer - it "threw away money" unnecessarily. We assume that the resources are a measuring stick of utility, and then ask whether the system maximizes any utility function over the given state-space measured by that measuring stick.

That is the best explanation of "what the tool of coherence theorems does and is useful for" that I've ever read, including discussions with you. Thanks for that!

The key properties that make money a useful measuring stick are:

• It's fungible: agents can purchase things they want through money (including other resources).
• Agent preferences over money are monotonically non decreasing as a function of quantity (more money is not worse [it may not always be better: e.g. some agents may have bounded utility functions]. I think you alluded to this by describing money as "additive", but I don't think that's the right property; additive resources aren't a useful measuring stick if the agent doesn't want more of the resource.)

Something that you did not address is how the measuring stick relates to the utility function.

Consider five resources $\{A,B,C,D,E\}$, and functions that measure the utility to the agent of a given quantity of each resource. $(R_A,R_B,R_C,R_D,R_E)(R_X:R\to R)$.

• $R_A\in \Theta \left(\log n\right)$
• $R_B\in \Theta \left(\sqrt{n}\right)$
• $R_C\in \Theta \left(n\right)$
• $R_D\in \Theta \left(n^2\right)$
• $R_E\in \Theta \left(2^n\right)$

Which of the four resources is a measuring stick of utility? Are all of them?

I don't think this is a significant hindrance. If we know how utility of the resource grows as a function of quantity, we could back out a linear measure of utility by applying said function to the quantity of the commodity possessed.

As for the question of how to identify measuring sticks of utility in the wild, I think the two conditions highlighted earlier purchase a lot of mileage:

Any commodity that:

1. Can be traded for known resources (e.g. energy, money [specifically, it's commensurable with every other resource]) or is otherwise fungible for things the agent cares about
   • This fungibility should be in such a manner as to preserve the "monotonically nondecreasing as a function of quantity" property
2. Agent preferences over the commodity are monotonically non decreasing as a function of the quantity of the commodity possessed

Note that the measuring stick for utility does NOT require a common attribute of measuring sticks for length - every dollar is better to have than not, but does not need to be better by the same amount as other dollars.  It's perfectly acceptable to have a ranking of outcomes ( mushroom > pepperoni > anchovy > nothing ), with no numbers attached, and without implying that the pepperoni->mushroom upgrade is worse than two anchovy->pepperoni upgrades.  

Summary

Epistemic status: I may misunderstand parts of the post.

Utility functions are inherently coherent:

• More utility is better
• Twice as much is twice as good.

One problem is that we usually do not know the utility function of a system. Knowing this could be useful, for example, if we want to talk about what an advanced AI is optimizing for.

So how do we figure out the utility of a system? One solution is to have a measuring stick of it, i.e., a resource such that:

• The system finds the resource only instrumentally valuable.
• The system can trade the resource in equal proportions to utility

Possible examples of such a resource:

• Money
• Energy

If we have found such a measuring stick for a certain utility maximizer, then we can use it to determine the utility function by observing which trades the system would make. Additionally, we may measure the extent to which the system is a utility maximizer at all by measuring how coherent its trades with the resource are. 

The measuring stick problem asks: How do we find a measuring stick in the wild, when being handed a universe?

Opinion/Confusions

There are some things that confuse me about this and that make me wonder if this is the right frame. There seem some hidden assumptions on optimization in the post that are not transparent to me personally. Some of the confusions may be answered in the post itself, but they do not… “stick”.

• John seems to assume that there is one resource that is equally valuable for maximizing utility for all systems. Why is that? Resources like energy may seem useful for transforming the whole universe, but maybe my utility function is not about making such large-scale transformations.
  • Take humans as examples: most humans experience strong diminishing returns in almost all (or maybe even all) resources. 
• Possibly John’s answer would be “well, if you don’t want to make large-scale transformations, then you don’t have a utility function, because utility functions are about transforming a large system into a small set of states”. 
  • John’s earlier post on utility maximization being description length minimization seems to support that view. 
  • There is, however, an intuition in myself that finds that post unnatural, especially since the mapping between utility function and probability distribution had a free parameter
• What if a system has a utility function, but it’s simply not about states in the same space that John cares about? 
  • E.g., some entity may care about optimizing trajectories, not states. 
    • In that example, we would end up with a distribution over trajectories instead of states. I would imagine that a different type of resource would help in optimizing that than in optimizing states themselves. 

If the rate of entropy generation was increasing within a physical system, wouldn't that indicate an optimization process. As the physical system went through a narrower and narrower band of state space, that would have to be the result of some optimizer.

 The utility function could be represented by the path that is taken through state space. Each next step would be higher in the optimizer's preferences, or it is "throwing away money". 

I do not know how to reverse engineer a utility function from a path through state space, though. (Could resources be represented by the amount of entropy your willing to generate to move from your current state to a different one?)

Epistemic status: probably wrong; intuitively, I feel like I'm onto something but I'm too uncertain about this framing to be confident in it

I refer to optimizers which can be identified by a measuring stick of utility as agenty optimizers

The measuring stick is optimization power. In particular, in the spirit of this sequence, it is the correlation between local optimization and optimization far away. If I have 4 basic actions available to me and each performs two bits of optimization on the universe, I am maximally powerful (for a structure with 4 basic actions) and most definitely either an agent or constructed by one. I speak and the universe trembles.

One might look at the life on Earth and see that it is unusually structured and unusually purposeful and conclude that it is the work of an agenty optimizer. And they would be wrong.

But if they looked closer, at the pipelines and wires and radio waves on Earth, they might conclude that they were the work of an agenty optimizer because they turn small actions (flipping a switch, pressing a key) into large, distant effects (water does or doesn't arrive at a village, a purchase is confirmed and a bushel of apples is shipped across the planet). And they would be correct.

In this framing, resources under my control are structures which propagate and amplify my outputs out into large, distant effects (they needn't be friendly, per se, they just have to be manipulable). Thus, a dollar (+ Amazon + a computer + ...) is an invaluable resource because, with it, I can cause any of literally millions of distinct objects to move from one part of the world to another by moving my fingers in the right way. And I can do that because the world has been reshaped to bend to my will in a way that clearly indicates agency to anyone who knows how to look.

However, I haven't the slightest idea how to turn this framework into a method for actually identifying agents (or resources) in a universe with weird physics.

Also, I have a sense that there is an important difference between accumulating asymmetric power (allies, secret AI) and creating approximately symmetrically empowering infrastructure (Elicit), which is not captured by this framework. Maybe the former is evidence of instrumental resource accumulation whereas the latter provides specific information about the creator's goals? But both *are* clear signs of agenty optimization, so maybe it's not relevant to this context?

Also possibly of note is that more optimization power is not strictly desirable because having too many choices might overwhelm your computational limitations.

I was going to comment that this older post [LW · GW] might provide insight into this problem, and then realized that you wrote that older post...

Typo:  reverse X and Y in one of these.

The agent "prefers" X over Y if it expends resources to get from X to Y.

Why is a resource central here? Consider (if it helps, also change "upgrade" to "switch"):

Let’s start with the simplest coherence theorem: suppose I’ll pay to upgrade pepperoni pizza to mushroom, pay to upgrade mushroom to anchovy, and pay to upgrade anchovy to pepperoni. This does not bode well for my bank account balance. And the only way to avoid having such circular preferences is if there exists some “consistent preference ordering” of the three toppings - i.e. some ordering such that I will only pay to upgrade to a topping later in the order, never earlier. That ordering can then be specified as a utility function: a function which takes in a topping, and gives the topping’s position in the preference order, so that I will only pay to upgrade to a topping with higher utility.

Surely one can notice the circularity problem without using a measuring stick.

Suppose an agent is thinking about whether to cooperate in a Prisoner's Dilemma. In the counterfactual where it cooperates, it might naturally deduce that other agents like it would also cooperate. Therefore we could hand it a world with weird physics, and see whether in the counterfactual where it cooperates, it can deduce more about that world. Then it has presumably found agents like itself in that world.