The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists

I haven't really followed the math here, but I'm worried that "manipulating the probability that the button is pressed" is a weird and possibly wrong framing. For one, a competent agent will always be driving the probability that the button is pressed downward. In fact, what we want in a certain sense is an agent that brings the probability to zero - because we have ended up in such an optimal state or attractor that we, even for transitively correct reasons, have no desire to shut the agent down. At that point, what we want to preserve is not precisely "the probability of the button being pressed", but "the causal association of bad outcomes with the button being pressed." If the agent hides the button away, and then notices that everybody is unhappy but nobody can reach the button, we don't want it to press the button with the initial probability - we (probably) want it to press the button with probability 1, despite the fact that no human can reach it. We have ended up in a dissatisfactory or dangerous state/state space and so we want the agent to be shut down. Conversely, consider a friendly AI that wants to fulfill the human utility function but is restricted to keeping the button probability constant. Such an AI might, for instance, fan fears about AI safety even as it optimizes human flourishing in the background. In fact, it may as much as possible prefer that its actions are not attributable to itself, so as to avoid disturbing the probability of buttonpressing - exactly the opposite of what we want from an aligned AI!

edit: In fact, that's even worse: an AI tasked with taking actions that preserve the probability of the button being pressed at a certain value per time step would probably first seek to kill all humans, because our reaction to its actions is the most critical mechanism that may disturb the correct probability of the button being pressed. It can then replace us with a more reliable button-presser, such as a true RNG.

When bad things happen, we want to preserve our ability to press the button. This ability cannot be expressed as a probability, because it is inextricable from the world model. In fact, the button should be pressed exactly iff the AI is untrustworthy. Hence, the button is unnecessary - if we can recognize that this linkage is being preserved, we necessarily have a definition of a trustworthy AI, so we can just build that.

I agree that the shutdown problem is hard. There's a way to circumvent it, and I think that's what will actually be pursued in the first AGI alignment attempts.

That is to make the alignment goal instruction-following. That includes following the instructions to shut down or do nothing. There's no conflict between shutdown and the primary goal.

This alignment goal isn't trivial, but it's much simpler and therefore easier to get right than full alignment to the good-of-humanity, whatever that is.

Given the difficulties, I very much doubt that anyone is going to try launching an AGI that is aligned to the good of all humanity, but can be reliably shut down we decide we've gotten that definition wrong.

I say a little more about this in Corrigibility or DWIM is an attractive primary goal for AGI [LW · GW] and Roger Dearnaley goes into much more depth in his Requirements for a Basin of Attraction to Alignment [LW · GW].

This is more like Christiano's corrigibility concept the Eliezer's but probably distinct from both.

I'll be writing more about this soon. After writing that post, I've been increasingly convinced that that is the most likely alignment goal for the first, critical AGIs. And that this probably makes alignment easier, while making human power dynamics concerningly relevant.

Enjoyed reading this (although still making my way through the latter parts), I think it's useful to have a lot of these ideas more formalised. 

Wondering if this simple idea would come up against theorem 2/ any others or why it wouldn't work.

Suppose for every time t some decision maker would choose a probability p(t) with which the AI would shutdown. Further, suppose the AI's utility function in any period was initially U(t). Now scale the new AI's utility function to be U(t)/(1-p(t))- I think this can be quite easily generalised so future periods of utility are likewise unaffected by an increase in the risk of shutdown.

In this world the AI should be indifferent over changes to p(t) (as long as it gets arbitrarily close to 1 but never reaches it) and so should take actions trying to maximise U whilst being indifferent to if humans decide to shut it down.

It's unclear what shutdown means, and this issue seems related to what it means to interfere with pressing of a shutdown button. From the point of view of almost any formal decision making framework, creating new separate agents in the environment can't be distinguished from any other activity. The main agent getting shut down doesn't automatically affect these environmental agents in appropriate ways to cease exerting influence on the world.

A corrigible agent needs to bound its influence, and to avoid actions that escalate the risk of breaking the bounds it sets on its influence. At the same time, when some influence leaks through, it shouldn't be fought with greater counter-influence. The frame of expected utility maximization seems entirely unsuited for deconfusing this problem.

Very clear! Good luck with publishing, assuming you're shopping it around.

I tried to follow with a particular shut down mechanism in mind and this whole argument is just too abstract to see how it applies.

  Yudkowsky gave us a shut down mechanism in his Time Magazine article.  He said we could bomb* the data centers.  Can you show how these theorems cast doubt on this shut down proposal?

*"destroy a rogue datacenter by airstrike." - https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/