Coherence arguments do not entail goal-directed behavior

I think that strictly speaking this post (or at least the main thrust) is true, and proven in the first section. The title is arguably less true: I think of 'coherence arguments' as including things like 'it's not possible for you to agree to give me a limitless number of dollars in return for nothing', which does imply some degree of 'goal-direction'.

I think the post is important, because it constrains the types of valid arguments that can be given for 'freaking out about goal-directedness', for lack of a better term. In my mind, it provokes various follow-up questions:

1. What arguments would imply 'goal-directed' behaviour?
2. With what probability will a random utility maximiser be 'goal-directed'?
3. How often should I think of a system as a utility maximiser in resources, perhaps with a slowly-changing utility function?
4. How 'goal-directed' are humans likely to make systems, given that we are making them in order to accomplish certain tasks that don't look like random utility functions?
5. Is there some kind of 'basin of goal-directedness' that systems fall in if they're even a little goal-directed, causing them to behave poorly?

Off the top of my head, I'm not familiar with compelling responses from the 'freak out about goal-directedness' camp on points 1 through 5, even though as a member of that camp I think that such responses exist. Responses from outside this camp include Rohin's post 'Will humans build goal-directed agents?' [LW · GW]. Another response is Brangus' comment post [LW · GW], although I find its theory of goal-directedness uncompelling.

I think that it's notable that Brangus' post was released soon after this was announced as a contender for Best of LW 2018. I think that if this post were added to the Best of LW 2018 Collection, the 'freak out' camp might produce more of these responses and move the dialogue forward. As such, I think it should be added, both because of the clear argumentation and because of the response it is likely to provoke.

In this essay, Rohin sets out to debunk what ey perceive as a prevalent but erroneous idea in the AI alignment community, namely: "VNM and similar theorems imply goal-directed behavior". This is placed in the context of Rohin's thesis that solving AI alignment is best achieved by designing AI which is not goal-directed. The main argument is: "coherence arguments" imply expected utility maximization, but expected utility maximization does not imply goal-directed behavior. Instead, it is a vacuous constraint, since any agent policy can be regarded as maximizing the expectation of some utility function.

I have mixed feelings about this essay. On the one hand, the core argument that VNM and similar theorems do not imply goal-directed behavior is true. To the extent that some people believed the opposite, correcting this mistake is important. On the other hand, (i) I don't think the claim Rohin is debunking is the claim Eliezer had in mind in those sources Rohin cites (ii) I don't think that the conclusions Rohin draws or at least implies are the right conclusions.

The actual claim that Eliezer was making (or at least my interpretation of it) is, coherence arguments imply that if we assume an agent is goal-directed then it must be an expected utility maximizer, and therefore EU maximization is the correct mathematical model to apply to such agents.

Why do we care about goal-directed agents in the first place? The reason is, on the one hand goal-directed agents are the main source of AI risk, and on the other hand, goal-directed agents are also the most straightforward approach to solving AI risk. Indeed, if we could design powerful agents with the goals we want, these agents would protect us from unaligned AIs and solve all other problems as well (or at least solve them better than we can solve them ourselves). Conversely, if we want to protect ourselves from unaligned AIs, we need to generate very sophisticated long-term plans of action in the physical world, possibly restructuring the world in a rather extreme way to safe-guard it (compare with Bostrom's arguments for mass surveillance). The ability to generate such plans is almost by definition goal-directed behavior.

Now, knowing that goal-directed agents are EU maximizers doesn't buy us much. As Rohin justly observes, without further constraints it is a vacuous claim (although the situation becomes better if we constraint ourselves to instrumental reward functions [AF · GW]). Moreover, the model of reasoning in complex environments that I'm advocating myself (quasi-Bayesian reinforcement learning [AF(p) · GW(p)]) doesn't even look like EU maximization (technically there is a way to interpret it as EU maximization but it underspecifies the behavior). This is a symptom of the fact that the setting and assumptions of VNM and similar theorems are not good enough to study goal-directed behavior. However, I think that it can be an interesting and important line of research, to try and figure out the right setting and assumptions.

This last point is IMO the correct takeaway from Rohin's initial observation. In contrast, I remain skeptical about Rohin's thesis that we should dispense with goal-directedness altogether, for the reason I mentioned before: powerful goal-directed agents seem necessary or at least very desirable to create a defense system from unaligned AI. Moreover, the study of goal-directed agents is important to understand the impact of any powerful AI system on the world, since even a system not designed to be goal-directed can develop such agency (due to reasons like malign hypotheses, mesa-optimization and self-fullfiling prophecies [AF · GW]).

A year later, I continue to agree with this post; I still think its primary argument is sound and important. I'm somewhat sad that I still think it is important; I thought this was an obvious-once-pointed-out point, but I do not think the community actually believes it yet.

I particularly agree with this sentence of Daniel's review [LW(p) · GW(p)]:

I think the post is important, because it constrains the types of valid arguments that can be given for 'freaking out about goal-directedness', for lack of a better term."

"Constraining the types of valid arguments" is exactly the right way to describe the post. Many responses to the post have been of the form "this is missing the point of EU maximization arguments", and yes, the post is deliberately missing that point. The post is not saying that arguments for AI risk are wrong, just that they are based on intuitions and not provable theorems. While I do think that we are likely to build goal-directed agents, I do not think the VNM theorem and similar arguments support that claim: they simply describe how a goal-directed agent should think.

However, talks like AI Alignment: Why It’s Hard, and Where to Start and posts like Coherent decisions imply consistent utilities [? · GW] seem to claim that "VNM and similar theorems" implies "goal-directed agents". While there has been some disagreement [LW(p) · GW(p)] over whether this claim is actually present, it doesn't really matter -- readers come away with that impression. I see this post as correcting that claim; it would have been extremely useful for me to read this post a little over two years ago, and anecdotally I have heard that others have found it useful as well.

I am somewhat worried that if readers who read this post in isolation will get the wrong impression, since it really was meant as part of the sequence. For example, I think Brangus' comment post [LW · GW] is proposing an interpretation of "goal-directedness" that I proposed and argued against in the previous post [? · GW] (see also my response [LW(p) · GW(p)], which mostly quotes the previous post). Similarly, I sometimes hear the counterargument that there will be economic pressures towards goal-directed AI, even though this position is compatible with the post and addressed in the next post [? · GW]. I'm not sure how to solve this though, without just having both the previous and next posts appended to this post. (Part of the problem is that different people have different responses to the post, so it's hard to address all of them without adding a ton of words.) ETA: Perhaps adding the thoughts in this comment [LW(p) · GW(p)]?

(This comment provides more intuition pumps for why it is invalid to argue "math implies AI risk". This is not a controversial point -- the critical review [AF(p) · GW(p)] agrees that this is true -- but I figured it was worth writing down for anyone who might still find it confusing, or feel like my argument in the post is "too clever".)

It should seem really weird to you on a gut level to hear the claim that VNM theorem, and only the VNM theorem, implies that AI systems would kill us all. Like, really? From just the assumption that we can't steal resources from the AI system with certainty ^[1], we can somehow infer that the AI must kill us all? Just by knowing that the AI system calculates the value of uncertain cases by averaging the values of outcomes based on their probabilities ^[2], we can infer that the AI system will take over the world?

But it's even worse than that. Consider the following hopefully-obvious claims:

• The argument for AI risk should still apply if the universe is deterministic.
• The argument for AI risk should still apply if the agent is made more intelligent.

If you believe that, then you should believe that the argument for AI risk should also work in a deterministic universe in which the AI can perfectly predict exactly what the universe does. However, in such a universe, the VNM theorem is nearly contentless -- the AI has no need of probability, and most of the VNM axioms are irrelevant. All you get with the VNM theorem in such a universe is that the AI's ordering over outcomes is transitive: If it chooses A over B and B over C, then it also chooses A over C. Do you really think that just from transitivity you can argue for AI x-risk? Something must have gone wrong somewhere.

I think the way to think about the VNM theorem is to see it as telling you how to compactly describe choice-procedures.

Suppose there are N possible outcomes in the world. Then one way to describe a choice-procedure is to describe, for all N(N-1)/2 pairs of outcomes, which outcome the choice-procedure chooses. This description has size O(N^2).

If you assume that the choice-procedure is transitive (choosing A over B and choosing B over C implies choosing A over C), then you can do better: you can provide a ranking of the options (e.g. B, A, C). This description has size O(N).

The VNM theorem deals with the case where you introduce lotteries over outcomes, e.g. a 50% chance of A, 20% chance of B, and 30% chance of C, and now you have to choose between lotteries. While there were only N outcomes, there are uncountably infinitely many lotteries, so simply writing down what the choice-procedure does in all cases would require an uncountably infinitely large description.

The VNM theorem says that if the choice-procedure satisfies a few intuitive axioms, then you can still have an O(N) size description, called a utility function. This function assigns a number to each of the N outcomes (hence the O(N) size description) ^[3]. Then, to compute what the choice-procedure would say for a pair of lotteries, you simply compute the expected utility for each lottery, and say that the choice-procedure would choose the one that is higher.

Notably, the utility function can be arbitrarily complicated, in the sense that it can assign any number to each outcome, independently of all the other outcomes. People then impose other conditions, like "utility must be monotonically increasing in the amount of money you have", and get stronger conclusions, but these are not implied by the VNM theorem. Ultimately the VNM theorem is a representation theorem telling you how to compactly represent a choice-procedure.

It seems to me that AI risk is pretty straightforwardly about how the choice-procedures that we build rank particular outcomes, as opposed to different lotteries over outcomes. The VNM theorem / axioms say ~nothing about that, so you shouldn't expect it to add anything to the argument for AI risk.

Note that this starts from the assumption of goal-directed behavior and derives that the AI will be an EU maximizer along with the other convergent instrumental subgoals.

The result is actually stronger than that, I think: if the AI is goal-directed at least in part, then that part will (tend to) purge the non-goal directed behaviours and then follow the EU path.

I wonder if we could get theorems as to what kinds of minimal goal directed behaviour will result in the agent becoming a completely goal-directed agent.

Actually, no matter what the policy is, we can view the agent as an EU maximizer. The construction is simple: the agent can be thought as optimizing the utility function U, where U(h, a) = 1 if the policy would take action a given history h, else 0. Here I’m assuming that U is defined over histories that are composed of states/observations and actions.

This is not the type signature for a utility function that matters for the coherence arguments (by which I don't mean VNM - see this comment [AF(p) · GW(p)]). It does often fit the type signature in the way those arguments are formulated/formalised, but intuitively, it's not getting at the point of the theorems. I suggest you consider utility functions defined as functions of the state of the world only, not including the action taken. (Yes I know actions could be logged in the world state, the agent is embedded in the state, etc. - this is all irrelevant for the point I'm trying to make - I'm suggesting to consider the setup where there's a Cartesian boundary, an unknown transition function, and environment states that don't contain a log of actions.) I don't think the above kind of construction works in that setting. I think that's the kind of setting it's better to focus on.

One additional source that I found helpful to look at is the paper "Formalizing Convergent Instrumental Goals" by Tsvi Benson-Tilsen and Nate Soares, which tries to formalize Omohundro's instrumental convergence idea using math. I read the paper quickly and skipped the proofs, so I might have misunderstood something, but here is my current interpretation.

The key assumptions seem to appear in the statement of Theorem 2; these assumptions state that using additional resources will allow the agent to implement a strategy that gives it strictly higher utility (compared to the utility it could achieve if it didn't make use of the additional resources). Therefore, any optimal strategy will make use of those additional resources (killing humans in the process). In the Bit Universe example given in the paper, if the agent doesn't terminally care what happens in some particular region $h$ (I guess they chose this letter because it's supposed to represent where humans are), but $h$ contains resources that can be burned to increase utility in other regions, the agent will burn those resources.

Both Rohin's and Jessica's [LW(p) · GW(p)] twitching robot examples seem to violate these assumptions (if we were to translate them into the formalism used in the paper), because the robot cannot make use of additional resources to obtain a higher utility.

For me, the upshot of looking at this paper is something like:

• MIRI people don't seem to be arguing that expected utility maximization alone implies catastrophe.
• There are some additional conditions that, when taken together with expected utility maximization, seem to give a pretty good argument for catastrophe.
• These additional conditions don't seem to have been argued for (or at least, this specific paper just assumes them).

An agent that constantly twitches could still be a threat if it were trying to maximise the probability that it would actually twitch in the future. For example, if it were to break down, it wouldn't be able to twitch, so it might want to gain control of resources.

I don't suppose you could clarify exactly how this agent that is twitching is defined. In particular, how does it accumulate over time? Do you get 1 utility for each point in time where you twitch and is your total utility the undiscounted sum of these utilities.

(I think I’ve seen it before too, but I can’t remember where.)

Possibly on LessWrong (v1.0), where on a couple of occasions I called it the Texas Sharpshooter Utility Function (to imply that it is a useless concept).

(I promise I'm not being intentionally contrarian.)

I agree with everything after this paragraph.

It can be more interesting when “outcomes” refers to world states instead (that is, snapshots of what the world looks like at a particular time), but utility functions over states/snapshots can’t capture everything we’re interested in, and there’s no reason to take as an assumption that an AI system will have a utility function over states/snapshots.

In particular, I don't doubt that, if we allow utility functions over histories, then saying that something is an expected utility maximizer doesn't tell us much about its behavior.

However, I intuitively think that we should expect AI to have a utility function over world states,^[1] and everything I've read (that I recall) suggests that this is what people are talking about. The Coherent decisions imply consistent utilities Arbital page is all about world states.

In fact, if we're talking about histories, then all of the examples of circular utilities stop being examples of circular utilities. An agent that keeps paying you to go around in circles via $A\to B\to C\to A\cdots$ can be perfectly rational since it corresponds to the utility function for which $U(B,t)>U(A,t),U(C,t)$ for $t\equiv 1\text{mod}3$ but $U(C,t)>U(A,t),U(B,t)$ for $t\equiv 2\text{mod}3$ and so on.

So it feels like you're arguing against something that was never intended. Clearly, EY wasn't thinking about utility functions that are allowed to depend on arbitrary histories when he wrote the Arbital post (or during his "AI alignment, why it's hard & where to start" talk, which makes the same points).^[2]

I may be making a stupid point here since I haven't read the VNM theorem. The problem is that I don't know what theorem exactly you're talking about, and neither you nor Eliezer linked to it, and I got the impression that there are actually a bunch of theorems saying similar things, and it's kind of a mess. I don't understand why you can't just look at the theorem and see whether it talks about world states or histories, but I guess the formalism is too abstract or something?

I'm also surprised that no-one else has made a similar point before. Has Eliezer ever responded to this post?

This post was very surprising to me; it's definitely an update about how to think about modelling intelligences, and opens up the space of possible theories a fair bit.

That said, I feel like I've not seen it reviewed by someone who has broadly the position this post responds to, so I'd really like it to be reviewed by someone else.

This post directly addresses what I think is the biggest conceptual hole in our current understanding of AGI: what type of goals will it have, and why? I think it's been important in pushing people away from unhelpful EU-maximisation framings, and towards more nuanced and useful ways of thinking about goals.

Naturally (as an author on that paper), I agree to some extent with this argument.

I think it's worth pointing out one technical 'caveat': the agent should get utility 0 *on all future timesteps* as soon as it takes an action other than the one specified by the policy. We say the agent gets reward 1: "if and only if its history is an element of the set H", *not* iff "the policy would take action a given history h". Without this caveat, I think the agent might take other actions in order to capture more future utility (e.g. to avoid terminal states). [Side-note (SN): this relates to a question I asked ~10days ago about whether decision theories and/or policies need to specify actions for impossible histories.]

My main point, however, is that I think you could do some steelmanning here and recover most of the arguments you are criticizing (based on complexity arguments). TBC, I think the thesis (i.e. the title) is a correct and HIGHLY valuable point! But I think there are still good arguments for intelligence strongly suggesting some level of "goal-directed behavior". e.g. it's probably physically impossible to implement policies (over histories) that are effectively random, since they look like look-up tables that are larger than the physical universe. So when we build AIs, we are building things that aren't at that extreme end of the spectrum. Eliezer has a nice analogy in a comment on one of Paul's posts (I think), about an agent that behaves like it understands math, except that it thinks 2+2=5. You don't have to believe the extreme version of this view to believe that it's harder to build agents that aren't coherent *in a more intuitively meaningful sense* (i.e. closer to caring about states, which is (I think, e.g. see Hutter's work on state aggregation) equivalent to putting some sort of equivalence relation on histories).

I also want to mention Laurent Orseau's paper: "Agents and Devices: A Relative Definition of Agency", which can be viewed as attempting to distinguish "real" agents from things that merely satisfy coherence via the construction in our paper.

Actually, no matter what the policy is, we can view the agent as an EU maximizer.

There is an even broader argument to be made. For an agent that is represented by a program, no matter what the preferences are, even if inconsistent, we can view it as an EU maximizer that always chooses the output it is programmed to take. (If it is randomized, its preferences are weighted between those options.)

I suspect there are other constructions that are at least slightly less trivial, because this trivial construction has utilities over only the "outcomes" of which action it takes, which is a deontological goal, rather than the external world, which would allow more typically consequentialist goals. Still, it is consistent with definitions of EU maximization.

I'm not sure that the agent that constantly twitches is going to be motivated by coherence theorems anyways. Is the class of agents that care about coherence identical to the class of potentially dangerous goal-directed/explicit-utility-maximizing/insert-euphemism-here agents?