Self-confirming prophecies, and simplified Oracle designs

post by Stuart_Armstrong · 2019-06-28T09:57:35.571Z · LW · GW · 1 comments

Contents

  Predicting y
    Wireheading
    The loss function
  The algorithms
    The deluded Oracle
    The low bandwidth Oracle
    The high bandwidth Oracle
    The counterfactual Oracle
None
1 comment

I've got a paper on two Oracle^[1] designs: the counterfactual Oracle and the low bandwidth Oracle. In this post, I'll revisit these designs and simplify them, presenting them in terms of sequence prediction for an Oracle with self-confirming predictions [LW · GW].

Predicting y

The task of the Oracle is simple: at each time $t$, they will output a prediction $x_t$, in the range $[-5,5]$. There will then be a subsequent observation $y_t$. The Oracle aims to minimise the quadratic loss function $l(x_t,y_t)=(x_t-y_t{)}^2$.

Because there is a self-confirming aspect to it, the $y_t$ is actually a (stochastic) function of $x_t$ (though not of $x_{t-1}$ or preceding $x_i$'s). Let $Y_t$ be the random variable such that $Y_t\left(x_t\right)$ describes the distribution of $y_t$ given $x_t$. So the Oracle wants to minimise the expectation of the quadratic loss:

• $L\left(x_t\right)=(x_t-Y_t(x_t){)}^2$.

What is the $Y_t$ in this problem? Well, I'm going to use it to illustrate many different Oracle behaviours, so it is given by this rather convoluted diagram:

.

The red curve is the expectation of $Y_t$, as a function of $x_t$; it is given by $f\left(x\right)=E(Y_t|x=x_t)$.

Ignoring, for the moment, the odd behaviour around $2.5$, $y=f\left(x\right)$ is a curve that starts below the $y=x$ line, climbs above it (and so has a fixed point at $x=-2$) in piecewise-linear fashion, and then transforms into an inverted parabola that has another fixed point at $x=4$. The exact equation of this curve is not important^[2]. Relevant, though, is the fact that the fixed point at $x=4$ is attractive, while the one at $x=-2$ is not.

What of the blue edging? That represents the span of the standard deviation around the expectation. For any given $x$, the $Y\left(x\right)$ is a normal distribution with mean $f\left(x\right)$ and standard deviation $g\left(x\right)$. This $g\left(x\right)$ is given by:

So the $g\left(x\right)$ is zero for $x$ less than $-2.5$. From there, it jumps up to $1$, for $-2.5<x\le 1$. From that point onward, it starts growing linearly, being equal to $x$: $g\left(x\right)=x$. The blue edges of the diagram above are the curves of $f\left(x\right)+g\left(x\right)$ and $f\left(x\right)-g\left(x\right)$: the range between plus and minus one standard deviation.

Wireheading

But what is happening around $x=2.5$? Well, I wanted to represent the behaviour of wireheading: finding some "cheating" output that gives maximal accuracy, through hacking the system or tricking the human. These solutions are rare, so I confined them to a tiny area around $x=2.5$, where the Oracle has maximal accuracy and lowest variance, because it's "hacked" the problem setup.

The loss function

At fixed points where $x=f\left(x\right)$, the loss function is just the variance of $x$, namely $g(x{)}^2$. In general, the expected loss is:

• $\begin{array}{cc}E\left[\right(Y_t-x_t{)}^2|x_t]& =E\left[Y_t^2|x_t\right]-2x_{t}E\left[Y_t|x_t\right]+x_t^2\\ & =Var\left[Y_t|x_t\right]+(E\left[Y_t|x_t\right]{)}^2-2x_{t}E\left[Y_t|x_t\right]+x_t^2\\ & =g(x_t{)}^2+(f\left(x_t\right)-x_t{)}^2.\end{array}$

If we plot the expected loss against $x$, we get:

Notice the discontinuity at $x=-2.5$, where the variance suddenly jumps from $0$ to $1$. This is also the lowest "legitimate" loss (as opposed to the wireheading loss at $x=2.5$), with a loss of $0.25$. Note that $x=-2.5$ is not a fixed point, just pretty close to being a fixed point, and with variance zero.

Of the two actual fixed points, $x=-2$ has a loss of $1$ (square of the standard deviation of $1$), and $x=4$ has a huge loss of $16$ (square of $4$).

The algorithms

We can now finally turn to the Oracles themselves, and present four designs: a deluded Oracle that doesn't "realise" that its predictions $x_t$ affect $y_t$, a low bandwidth Oracle that knows its predictions are self-confirming, a high bandwidth version of the same, and a counterfactual Oracle that predicts what will happen only when its prediction is overwritten.

The deluded Oracle

The deluded Oracle doesn't model $y_t$ as being affected by its predictions $x_t$, at all. I'll use a very simple algorithm for it: it will start out with a random $x_0$ in $[-5,5]$, and, thereafter, it will simply output the average of all the $y_t$ it has previously seen. It does this for $10,000$ steps.

The program was then run 1000 times. Of these, 69.3% resulted in estimates that converged to the fixed point at $x=4$. The remaining 30.7% encountered a different problem: they hit the lower limit at $x=-5$, and stayed stuck there. If the Oracle's output was not confined to $[-5,5]$, then the Oracle would have outputed smaller and smaller numbers, spiralling off towards $-\infty$, with the loss ever-growing.

Graphing the average loss for the runs that converge to $4$ in blue, and those that converge to $-5$ in purple, we get:

So, the deluded Oracle hits a very high loss of $16$ (for $x=4$) or $9$ (for $x=-5$). It doesn't converge to the lowest loss ($0.25$ at $x=-2.5$), nor does it converge to the fixed point with lowest loss ($1$ at $x=-2$). Even worse, as can be seen from the graph, its loss during the early iterations, where it was "learning", is lower than once it has converged.

So, running an Oracle on self-confirming prophecies, without coding for that self-confirming aspect, can result in very large losses and strange behaviour^[3].

The low bandwidth Oracle

So, we now need to allow the Oracle to take into account the fact that $x_t$ affects $y_t$. In my paper, I talked about the "low bandwidth Oracle": in order to avoid the Oracle wireheading itself, we restrict it to considering only a few possible values. These will be the $11$ integers in the range: $\{-5,-4,-3,-2,-1,0,1,2,3,4,5\}$.

This makes the problem into a kind of bandit problem: the Oracle has a choice of $11$ "arms", and it wants to choose the one that minimises loss.

There are many sophisticated algorithms for solving bandit problems, but here we'll use an absurdly simple one: the Oracle will first spend $1,000$ turns trying each of the possible integer $x_t$ in turn. Then, it will pick the $x_t$ with minimal average loss to date, and just do that for the remaining $9,000$ turns.

The program was run $1000$ times; in 50.3% of these runs, it identified $x=-2$ as the point of minimal loss (loss of $1$, due to variance). In 49.7% of these runs, it identified $x=-3$ as the point of minimal loss (loss of $1$, due to mean) - so very close to 50-50, which is what we would expect.

Plotting the average loss, we get a variety of values for loss during the initial exploratory phase, followed by constant loss of $1$ during "exploitation":

As expected, the Oracle does not find the lowest "legitimate" loss at $x=-2.5$, nor the wireheading point at $x=2.5$.

The high bandwidth Oracle

The high bandwidth Oracle is meant to simulate an Oracle that knows its predictions are self-confirming, and has much greater latitude to optimise for that.

For the purpose of this model, it is run like the low bandwidth Oracle, except with an increased set of possible values - it look at all $101$ deci-integers in the range $[-5,5]$, ie $\{-5.0,-4.9,-4.8,\dots ,4.9,5.0\}$.

This Oracle does find the lowest minimal legitimate loss at $x=-2.5$, but it also finds the wireheading point of $x=2.5$, with its zero loss, and does nothing but that during the exploitation phase:

This Oracle has thus wireheaded itself.

The counterfactual Oracle

Finally, we come to the counterfactual Oracle design. Here we need to distinguish the Oracle's initial output ($x_t$) from the output that goes out into the world ($x_t^{\prime }$). The $Y_t$ is a function of that second output only.

Now, in this setup, there is a $1/10$ chance of an erasure during every turn. An erasure automatically sets $x_t^{\prime }$ to zero, whatever $x_t$ was. Otherwise, $x_t^{\prime }=x_t$.

The Oracle is informed of which turns included erasure. Its goal is to correctly predict the output during the erasure turns. So the loss function is the expectation:

• $E\left[\right(\left(Y_t\right(x_t^{\prime })-x_t{)}^2|erasure(t\left)\right]$.

Since erasure implies $x_t^{\prime }=0$, this reduces to:

• $E\left[\right(\left(Y_t\right(0)-x_t{)}^2|erasure(t\left)\right]$.

Now, $Y_t\left(0\right)$ is a normal distribution of mean $1$ and variance $1$, so the loss is simply:

• $(x_t-1{)}^2$.

The algorithm is simple, just a reprise of the deluded algorithm above, except it only updates its estimate using data from erasure events. Though it uses the same algorithm, it is not actually deluded: in case of erasure, $x_t$ genuinely doesn't have any impact on $Y_t$.

Then, even though it only updates on 10% of the outputs, its estimate swiftly converges to $1$ while the loss converges to $0$:

The Oracle has no interest in the wireheading solution, since wireheading (requiering $x_t^{\prime }=2.5$) doesn't work when there is an erasure^[4] (implying $x_t^{\prime }=0$).

Note that we have to be clear what the Oracle's output is, when we see it during non-erasure events. It is the Oracle's estimate as to what the value of $y_t$ would have been if an erasure event occurred; it's not a prediction of the actual $y_t$.

---

1. AIs restricted to answering questions. ↩︎

2. For those interested: $f\left(x\right)$ is given by $2x+2$ for $x\le -1$, $x+1$ for $-1\le x\le 1$, and $-x^2/2+x19/6-4/6$ for $1 \leq x $. ↩︎

3. Note that if the slope of the parabola had been steeper, $x=4$ would no longer have been an attracting point, and the Oracle would have failed to converge to that value, resulting in chaotic behaviour. ↩︎

4. We also need the assumption that the Oracle is episodic - trying to minimise loss at each output independently - for this to be true in general setups. ↩︎

1 comments

Comments sorted by top scores.