Plans Are Predictions, Not Optimization Targets

For instance, after reading some of Eliezer and Nate’s recent writing, I now think it’s probably a better idea to use corrigibility [? · GW] as an AI alignment target (assuming corrigibility turns out to be a coherent thing at all), as opposed to directly targeting human values

I'd be interested in you elaborating on this update. Specifically, how do you expect these approaches to differ in terms of mechanical interventions in the AGI's structure/training story, and what advantages one has over the other?

I really like this post, this is very influential about how I think about plans, and what to work on. I do think its a bit vague though, and lacking in a certain kind of general formulation. It may be better if there were more examples listed where the technique could be used.

Maybe a slightly better title to the post would be "Plans are prediction, not optimization targets"? I found the "plans are predictions" part of the post to be the most insightful, and the rewording also removes a "should".

While I agree with the practical idea, conveyed in this post, I think the language of the post is inconsistent, and filling it with alignment slang makes the post much less understandable. To demonstrate this, I'll reformulate the post in the ontology of Active Inference [LW · GW].

Our doctor-to-be does not treat the plan primarily as a prediction about the world; they treat it as a way to make the world be.

In Active Inference, a “prediction” and a “way to make the world to be” point to the same thing. Besides, a terminological clarification: in Active Inference, a plan (also policy) is a sequence of actions performed by the agent, interleaved (in discrete-time formulation) with predicted future states: $\pi =\{a_1,s_1,a_2,s_2,...\}$, not merely a prediction, which is a generative model over states alone (in Active Inference, actions are ontologically distinct from states): $P\left(s_{t+1}|s_t\right)$.

Now, when looking for those robust bottlenecks, I’d probably need to come up with a plan. Multiple plans, in fact. The point of “robust bottlenecks” is that they’re bottlenecks to many plans, after all. But those plans would not be optimization targets. I don’t treat the plans as ways to make the world be. Rather, the plans are predictions about how things might go. My “mainline plan”, if I have one, is not the thing I’m optimizing to make happen; rather, it’s my modal expectation for how I expect things to go (conditional on my efforts).

This is exactly what is described by Active Inference. In a simplified, discrete-time formulation, an Active Inference agent endlessly performs the following loop:

1. Receive observations
2. Update one’s model of the world according to the observations (e. g., using Bayesian inference)
3. “Choose a plan” by sampling a softmax distribution, where the logits are negative expected free energies, associated with each possible plan. Doing this exactly is impossible, of course, because the space of possible plans is infinite, so actual agents heuristically approximate this step.
4. Perform the first action from the chosen plan, and go back to step 1.

A complete, formal mathematical description of this loop is given in section 4.4 of Chapter 4 of Active Inference (which you can freely download the chapter on the linked page). In an actual implementation of an Active Inference agent, Fountas et al. used Monte-Carlo tree search as the heuristic to use in step 3 of the above loop.

Note that in the above loop, the plan is chosen again (and the whole space of plans is re-constructed) on every iteration of the loop. This, again, is an abstract ideal: that would be horribly inefficient to discard plans on every step, rather than incrementally update them. Nevertheless, this principle is captured in the adage “plans are worthless, but planning is everything”. And it also seems to me that John Wentworth tries to convey the same idea in the passage quoted above.

However, considering the above, the title “Plans Are Predictions, Not Optimization Targets”, and the suggestion that “robust bottlenecks” should be “optimisation targets” don’t make sense. Plans, if successful, have some outcomes (or goals, also known as prior preferences in Active Inference, which are probability distributions over future world states, in which the probability density indicates the level of preference), and, similarly, “robust bottlenecks” also have outcomes (or themselves are outcomes: it’s not clear to me what terminological convention do you use in the post in this regard). The ultimate goal of the plan (becoming a doctor, ensuring humanity’s value potential is not destroyed in the course of creating a superhuman AI) is still a goal. What you essentially suggest is frontloading the plan with achieving highly universal subgoals (”seeking options” in Nassim Taleb’s lingo). And the current title of the article can be reformulated as “Plans are not optimisation targets, but the outcomes of the plans are optimisation targets”, which doesn’t capture the idea of the post.

“On the path to your goal, identify and try to first achieve subgoals that can be most universally helpful on various different paths towards your goal” would be more accurate, IMO.

"Optimisation target" and "optimise for X" are confusing phrases

(This section is not only about this post I'm commenting on; I haven't decided yet whether to leave it as a comment or to turn it into a separate post.)

Despite considerable exposure to writing on AI alignment, I get confused every time I see the phrase “optimisation target”.

As I wrote in a comment [LW(p) · GW(p)] to “Reward is not an optimisation target”, the word “optimisation” is ambiguous because unclear which “optimisation” it refers to:

Reward probably won’t be a deep RL agent’s primary optimization target

The longer I look at this statement (and its shorter version "Reward is not the optimization target"), the less I understand what it's supposed to mean, considering that "optimisation" might refer to the agent's training process as well as the "test" process (even if they overlap or coincide). It looks to me that your idea can be stated more concretely as "the more intelligent/capable RL agents (either model-based or model-free) become in the process of training using the currently conventional training algorithms, the less they will be susceptible to wireheading, rather than actively seek it"?

Separately, the word "target" invokes a sense of teleology/goal-directedness, which, though perfectly fine in the context of this post, where people's "optimisation targets" are discussed, is confusing when the phrase "optimisation target" is applied to objects or processes which don't have any agency according to the models most people have in their heads, e. g., a DNN training episode.

Similarly, I think the phrase "optimise for X", such as "optimise for graduating", uses alignment slang completely unnecessarily. This phrase is exactly synonymous to "have a goal of graduating" or "tries to graduate", but the latter two don't have the veil of secondary connotation, related to alignment, which people might suppose the phrase "optimise for graduating" has, while, in fact, it doesn't.

My optimization targets are, instead, the robust bottlenecks.

When reality throws a brick through the plans, I want my optimization target to have still been a good target in hindsight. Thus robust bottlenecks: something which is still a bottleneck under lots of different assumptions is more likely to be a bottleneck even under assumptions we haven’t yet realized we should use. The more robust the bottleneck is, the more likely it will be robust to whatever surprise reality actually throws at us.

In practical advice and self-help, a subset of these “robust bottlenecks” are also simply called “basics”: meeting basic physical and psychological needs, keeping up one’s physical condition and energy, taking care of one’s mental health.

It's interesting to think how this fits in with other notions like the Simulators [LW · GW] idea.

When you say "optimization target," it seems like you mean a single point in path-space that the planner aims for, where this point consists of several fixed landmarks along the path which don't adjust to changing circumstances. Such an optimization target could still have some wiggle room (i.e., consist of an entire distribution of possible sub-paths) between these landmarks, correct? So some level of uncertainty must be built into the plan regardless of whether you call it a prediction or an optimization target.

It seems to me that what you're advocating for is equivalent to generating an entire ensemble of optimization targets, each based on a different predictive model of how things will go. Then you break those targets up into their constituent landmarks and look for clusters of landmarks in goal-space from across the entire ensemble of paths. Would your "robust bottlenecks" then refer to the densest of these clusters?

Funny meta: I'm reading this just after finishing your two [? · GW] sequences about Abstraction [? · GW], which I find very exciting! But surprise, your plan changes ! Did I read all that for nothing? Fortunately, I think it's mostly robust, indeed :)

I claim the right move is to target robust bottlenecks: look for subproblems which are bottlenecks to many different approaches/plans/paths, then tackle those subproblems.

This reminds me of Paul Graham's idea of flying upwind.

Suppose you're a college freshman deciding whether to major in math or economics. Well, math will give you more options: you can go into almost any field from math. If you major in math it will be easy to get into grad school in economics, but if you major in economics it will be hard to get into grad school in math.

Flying a glider is a good metaphor here. Because a glider doesn't have an engine, you can't fly into the wind without losing a lot of altitude. If you let yourself get far downwind of good places to land, your options narrow uncomfortably. As a rule you want to stay upwind. So I propose that as a replacement for "don't give up on your dreams." Stay upwind.

— What You'll Wish You'd Known

On the other hand... man, that high-schooler is gonna be shit outta luck if they decide that medicine isn't for them. Or if the classes or residency are too tough. Or if they fail the MCAT. Or .... Point is, reality has a way of throwing bricks through our plans, even when we're not operating in a preparadigmatic field.

This reminds me of a Paul Graham quote (but it's not the same thing):

A friend of mine who is a quite successful doctor complains constantly about her job. When people applying to medical school ask her for advice, she wants to shake them and yell "Don't do it!" (But she never does.) How did she get into this fix? In high school she already wanted to be a doctor. And she is so ambitious and determined that she overcame every obstacle along the way—including, unfortunately, not liking it.

— How to Do What You Love