Superintelligence Can't Solve the Problem of Deciding What You'll Do

If showing you a formal proof that you will do a particular action doesn't result in you doing that action, then the supposed "proof" was simply incorrect.

Yes, that's the point, you can make it necessarily incorrect, your decision to act differently determines the incorrectness of the proof, regardless of its provenance. When the proof was formally correct, your decision turns the whole possible world where this takes place counterfactual. (This is called playing chicken with the universe or the chicken rule, a technique that's occasionally useful for getting an agent to have nicer properties, by not letting the formal system that generates the proofs know too much early on about what the agent is going to decide.)

Manipulation that warps an agent loses fidelity in simulating them. Certainly superintelligence has the power to forget you in various ways, similarly to how a supernova explosion is difficult to survive, so this not happening needs to be part of the premise. A simulator that only considers what you do for a particular contrived input fails to observe your behavior as a whole.

So we need some concepts that say what it means for an agent to not be warped (while retaining interaction with sources of external influence rather than getting completely isolated), for something to remain the intended agent rather than some other phenomenon that is now in its place. This turns out to be a lot like the toolset useful for defining values of an agent. Some relevant concepts are membranes, updatelessness, and coherence of volition. Membranes gesture at the inputs that are allowed to come in contact with you, or information about you that can be used in determining the inputs that are allowed, that can be part of an environment that doesn't warp you and enables further observation. This shouldn't be restricted only to inputs, since the whole deal with AI risk is that AI is not some external invasion that arrives to Earth, it's something we are building ourselves, right here. So the concept of a membrane should also target acausal influences, patterns developing internally from within the agent.

Updatelessness is about a point of view on behavior where you consider its dependence on all possible inputs, not just actions given the inputs that you did apparently observe so far. Decisions should be informed by looking at the map of all possible situations and behaviors that take place there, even if the map has to be imprecise. (And not just situations that are clearly possible, the example in the post with ignoring the proof of what you'll do is about what you do in potentially counterfactual situations.)

Coherence of volition is about the problem of path dependence in how an agent develops. There are many possible observations, many possible thoughts and possible decisions, that lead to different places. The updateless point of view says that the local decisions should be informed by the overall map of this tree of possibilities for reflection, so it would be nice if path dependence doesn't lead to chaotic divergence, if there are coherent values to be found, even if only within smaller clusters of possible paths of reflection that settle into being sufficiently in agreement.

Ability to resist a proof of what your behavior will be even to the point of refuting its formal correctness (by determining its incorrectness with your own decisions and turning the situation counterfactual) seems like a central example of a superintelligence being unable to decide/determine (as opposed to predict) what your decisions are. It's also an innocuous enough input that doesn't obviously have to be filtered by weak agent's membrane.

In any case, to even discuss how a weak agent behaves in a superintelligent world, it's necessary to have some notion of keeping it whole. Extreme manipulation can both warp the weak agent and fail to elicit their behavior for other possible inputs. So this response [LW(p) · GW(p)] to another comment seems relevant.

Another way of stating this, drawing on the point about physical bodies thought of as simulations of some abstract formulation of a person, is to say that an agent by itself is defined by its own isolated abstract computation, which includes all membrane-permissible possible observations and resulting behaviors. Any physical implementation is then a simulation of this abstract computation, which can observe it to some extent, or fail to observe it (when the simulation gets sufficiently distorted). When an agent starts following dictates of external inputs, that corresponds to the abstract computation of the agent running other things within itself, which can be damaging to its future on that path of reflection depending on what those things are. In this framing, normal physical interaction with the external world becomes some kind of acausal interaction between the abstract agent-world (on inputs where the physical world is observed) and the physical world (for its parts that simulate the abstract agent-world).

The central point is more that superintelligence won't be able to help with your decisions, won't be able to decide for you at some fundamental level, no matter how capable it is. It can help instrumentally, but not replace your ability to decide in order to find out what the decisions are, so in some sense it's not able to help at all. I'm trying to capture the sense in which this holds regardless of its ability to precisely predict and determine outcomes in the physical world (if we only look at the state of the future, rather than full trajectories that get the world there).

When a program is given strange input, or if the computer it would be running on is destroyed, that event in the physical world usually doesn't affect the semantics of the program that describes its behavior for all possible inputs. If you are weak and brittle, talking about what you'll decide requires defining what we even mean in principle by a decision that is yours, only then can we ask if it does remain yours in actuality. Or if it does remain yours, even if you are yourself no longer present in actuality. Which is not very useful for keeping it (or yourself) present in actuality, but can be conceptually useful for formulating desiderata towards it being present in actuality.

So there are two claims. First, it's in some sense natural for your decisions to remain yours, if you don't start mindlessly parroting external inputs that dictate your actions, even in the face of superintelligence (in its aspect of capability, but not necessarily aimed in a way that disrupts you). Second, if you are strongly manipulated or otherwise overridden, this should in some sense mean that you are no longer present, that the resulting outcome doesn't capture or simulate what we should define as being you (in order to talk about the decisions that are in principle yours). Thus presence of overpowering manipulation doesn't contradict decisions usually remaining yours, it just requires that you are consequently no longer present to manifest them in actuality when that happens.

This seems like the third comment on the same concern, I've also answered it here [LW(p) · GW(p)] and here [LW(p) · GW(p)], going into more detail on other related things. So there is a missing prerequisite post.

More like operationalization of a binding rule is not something even superintelligence can do for you, even when it gives the correct operationalization. Because if you follow operationalizations merely because they are formally correct and given by a superintelligence, then you follow arbitrary operationalizations, not ones you've decided to follow yourself. How would the superintelligence even know what you decide, if you shirk that responsibility and wait for the superintelligence to tell you?

This post is mostly a reaction to Bostrom's podcasts about his new book. I think making your own decisions is a central example of something that can't be solved by others, no matter how capable, and also this activity is very close to what it means to define values, so plans in the vicinity of external imposition of CEV might be missing the point.

One thought about the rule/exception discussion you've linked (in the next paragraph; this one sets up the framing). Rules/norms are primitives of acausal coordination, especially interesting when they are agents in their own right. They mostly live in other minds, and are occasionally directly incarnated in the world, outside other minds. When a norm lives in many minds, it exerts influence on the world through its decisions made inside its hosts. It has no influence where it has no hosts, and also where the hosts break the norm. Where it does have influence, it speaks in synchrony in many voices, through all of its instances, thus it can have surprising power even when it only weakly compels the minds that host its individual instances.

So there is a distinction between an exception that isn't part of a rule, and an exception that the rule doesn't plan for. Rules are often not very intelligent, so they can fail to plan for most things, and thus suggest stupid decisions that don't take those things into account. This can be patched by adding those things into the rule, hardcoding the knowledge. But not every thing needs to be hardcoded in order for the rule to be able to adequately anticipate it in its decision making. This includes hosts (big agents) making an exception to the rule (not following it in a particular situation): some rules are able to anticipate when specifically that happens, and so don't need those conditions to become part of their formulation.