I'm not really sure what you mean when you say "something goes wrong" (in relation to the prize). I've been thinking about all this in a very descriptive way, ie, I want to understand what happens generally, not force a particular outcome. So I'm a little out-of-touch with the "goes wrong" framing at the moment. There are a lot of different things which could happen. Which constitute "going wrong"?

• Becoming non-myopic; ie, using strategies which get lower prediction loss long-term rather than on a per-question basis.
• (Note this doesn't necessarily mean planning to do so, in an inner-optimizer way.)
• Making self-fulfilling prophecies in order to strategically minimize prediction loss on individual questions (while possibly remaining myopic).
• Having a tendency for self-fulfilling prophecies at all (not necessarily strategically minimizing loss).
• Having a tendency for self-fulfilling prophecies, but not necessarily the ones which society has currently converged to (eg, disrupting existing equilibria about money being valuable because everyone expects things to stay that way).
• Strategically minimizing prediction loss in any way other than by giving better answers in an intuitive sense.
• Manipulating the world strategically in any way, toward any end.
• Catastrophic risk by any means (not necessarily due to strategic manipulation).

In particular, inner misalignment seems like something you aren't including in your "going wrong"? (Since it seems like an easy answer to your challenge.)

I note that the recursive-decomposition type system you describe is very different from most modern ML, and different from the "basically gradient descent" sort of thing I was imagining in the story. (We might naturally suppose that Predict-O-Matic has some "secret sauce" though.)

If you aren't already convinced, here's another explanation for why I don't think the Predict-O-Matic will make self-fulfilling prophecies by default.

In Abram's story, the engineer says: "The answer to a question isn't really separate from the expected observation. So 'probability of observation depending on that prediction' would translate to 'probability of an event given that event', which just has to be one."

In other words, if the Predict-O-Matic knows it will predict P = A, it assigns probability 1 to the proposition that it will predict P = A.

Right, basically by definition. The word 'given' was intended in the Bayesian sense, ie, conditional probability.

I contend that Predict-O-Matic doesn't know it will predict P = A at the relevant time. It would require time travel -- to know whether it will predict P = A, it will have to have made a prediction already, and but it's still formulating its prediction as it thinks about what it will predict.

It's quite possible that the Predict-O-Matic has become relatively predictable-by-itself, so that it generally has good (not perfect) guesses about what it is about to predict. I don't mean that it is in an equilibrium with itself; its predictions may be shifting in predictable directions. If these shifts become large enough, or if its predictability goes second-order (it predicts that it'll predict its own output, and thus pre-anticipates the direction of shift recursively) it has to stop knowing its own output in so much detail (it's changing too fast to learn about). But it can possibly know a lot about its output.

I definitely agree with most of the stuff in the 'answering a question by having the answer' section. Whether a system explicitly makes the prediction into a fixed point is a critical question, which will determine which way some of these issues go.

• If the system does, then there are explicit 'handles' to optimize the world by selecting which self-fulfilling prophecies to make true. We are effectively forced to deal with the issue (if only by random selection).
• If the system doesn't, then we lack such handles, but the system still has to do something in the face of such situations. It may converge to self-fulfilling stuff. It may not, and so, produce 'inconsistent' outputs forever. This will depend on features of the learning algorithm as well as features of the situation it finds itself in.

It seems a bit like you might be equating the second option with "does not produce self-fulfilling prophecies", which I think would be a mistake.

One possibility is that it's able to find a useful outside view model such as "the Predict-O-Matic has a history of making negative self-fulfilling prophecies". This could lead to the Predict-O-Matic making a negative prophecy ("the Predict-O-Matic will continue to make negative prophecies which result in terrible outcomes"), but this prophecy wouldn't be selected for being self-fulfilling. And we might usefully ask the Predict-O-Matic whether the terrible self-fulfilling prophecies will continue conditional on us taking Action A.

Maybe I misunderstood what you mean by dualism, but I don't think that's true. Say the Predict-O-Matic has an outside view model (of itself) like "The metal box on your desk (the Predict-O-Matic) will make a self-fullfilling prophecy that maximizes the number of paperclips". Then you ask it how likely it is that your digital records will survive for 100 years. It notices that that depends significantly on how much effort you make to secure them. It notices that that significantly depends on what the metal box on your desk tells you. It uses it's low-model resolution of what the box says. To work that out, it checks which outputs would be self-fulfilling, and then which of these leads to the most paperclips. The more unsecure your digital records are, the more you will invest in paper, and the more paperclips you will need. Therefore the metal box will tell you the lowest self-fulfilling propability for your question. Since that number is *self-fulfilling*, it is in fact the correct answer, and the Predict-O-Matic will answer with it.

I think this avoids your argument that

I contend that Predict-O-Matic doesn't know it will predict P = A at the relevant time. It would require time travel -- to know whether it will predict P = A, it will have to have made a prediction already, and but it's still formulating its prediction as it thinks about what it will predict.

because it doesn't have to simulate itself in detail to know what the metal box (it) will do. The low-resolution model provides a shortcut around that, but it will be accurate despite the low resolution, because by believing it is simple, it becomes simple.

Can you usefully ask for conditionals? Maybe. The answer to the conditional depends on what worlds you are likely to take Action A in. It might be that in most worlds where you do A, you do it because of a prediction from the metal box, and since we know those maximize paperclips, there's a good chance the action will fail to prevent it in those cricumstances. But if that's not the case, for example because it's certain you won't ask the box any more questions between this one and the event it tries to predict.

It might be possible to avoid any problems of this sort by only ever asking questions of the type "Will X happen if I do Y now (with no time to receive new info between hearing the prediction and doing the action)?", because by backwards induction the correct answer will not depend on what you actually do. This doesn't avoid the scenarios on the original where multiple people act on their Predict-O-Matics, but I suspect these aren't solvable without coordination.

What's going on when you try to model yourself thinking about the answer to this question?

If a system is analyzing (itself analyzing (itself analyzing (...))) , not realizing it's doing so, I suspect that it will come up with some best guess answer, but that answer will be ill-determined and dependent on implementation details. Thus a better approach would be to avoid asking self-unaware systems any question that requires that type of analysis!

For example, you can ask "Please output the least improbable scenario, according to your predictive world-model, wherein a cure for Alzheimer's is invented by a group of people with no access to any AI oracles!" Or even ask it to do counterfactual reasoning about what might happen in a world in which there are no AIs whatsoever. (copied from my post here [? · GW]). This type of question is nice for other reasons too—we're asking the system to guess what normal humans might plausibly do in the natural course of events, and thus we'll more typically get normal-human-type solutions to our problems as opposed to bizarre alien human-unfriendly solutions.

If dualism holds for Abram's prediction AI, the "Predict-O-Matic", its world model may happen to include this thing called the Predict-O-Matic which seems to make accurate predictions -- but it's not special in any way and isn't being modeled any differently than anything else in the world. Again, I think this is a pretty reasonable guess for the Predict-O-Matic's default behavior. I suspect other behavior would require special code which attempts to pinpoint the Predict-O-Matic in its own world model and give it special treatment (an "ego").

I don't think this is right. In particular, I think we should expect ML to be biased towards simple functions such that if there's a simple and obvious compression, then you should expect ML to take it. In particular, having an "ego" which identifies itself with its model of itself significantly reduces description length by not having to duplicate a bunch of information about its own decision-making process.

If dualism holds for Abram’s prediction AI, the “Predict-O-Matic”, its world model may happen to include this thing called the Predict-O-Matic which seems to make accurate predictions—but it’s not special in any way and isn’t being modeled any differently than anything else in the world. Again, I think this is a pretty reasonable guess for the Predict-O-Matic’s default behavior. I suspect other behavior would require special code which attempts to pinpoint the Predict-O-Matic in its own world model and give it special treatment (an “ego”).

I don't see why we should expect this. We're told that the Predict-O-Matic is being trained with something like sgd, and sgd doesn't really care about whether the model it's implementing is dualist or non-dualist; it just tries to find a model that generates a lot of reward. In particular, this seems wrong to me:

The Predict-O-Matic doesn't care about looking bad, and there's nothing contradictory about it predicting that it won't make the very prediction it makes, or something like that.

If the Predict-O-Matic has a model that makes bad prediction (i.e. looks bad), that model will be selected against. And if it accidentally stumbled upon a model that could correctly think about it's own behaviour in a non-dualist fashion, and find fixed points, that model would be selected for (since its predictions come true). So at least in the limit of search and exploration, we should expect sgd to end up with a model that finds fixed points, if we train it in a situation where its predictions affect the future.

If we only train it on data where it can't affect the data that it's evaluated against, and then freeze the model, I agree that it probably won't exhibit this kind of behaviour; is that the scenario that you're thinking about?

Two remarks.

Remark 1: Here's a simple model of self-fulfilling prophecies.

First, we need to decide how Predict-O-Matic outputs its predictions. In principle, it could (i) produce the maximum likelihood outcome (ii) produce the entire distribution over outcomes (iii) sample an outcome of the distribution. But, since Predict-O-Matic is supposed to produce predictions for large volume data (e.g. the inauguration speech of the next US president, or the film that will win the Oscar in 2048), the most sensible option is (iii). Option (i) can produce an outcome that is maximum likelihood but is extremely untypical (since every individual outcome has very low probability), so it is not very useful. Option (ii) requires somehow producing an exponentially large vector of numbers, so it's infeasible. More sophisticated variants are possible, but I don't think any of them avoids the problem.

If the Predict-O-Matic is a Bayesian inference algorithm, an interesting dynamic will result. On each round, some hypothesis will be sampled out of the current belief state. If this hypothesis is a self-fulfilling prophecy, sampling it will cause its likelihood to go up. We get positive feedback: the higher the probability Predict-O-Matic assigns to the hypothesis, the more often it is sampled, the more evidence in favor of the hypothesis is produced, the higher its probability becomes. So, if it starts out as sufficiently probable a priori, the belief state will converge there.

Of course realistic learning algorithms are not Bayesian inference, but they have to approximate Bayesian inference in some sense. At the least, there has to be some large space of hypotheses s.t. if one of them is true, the algorithm will converge there. Any algorithm with this property probably displays the dynamics above.

Now, to the simple model. In this model we have just two outcomes: A and B (so it's not large volume data, but that doesn't matter). On each round a prediction is made, after which some outcome occurs. The true environments works as follows: if prediction "A" is made, on this round A happens with probability 99% and B with probability 1%. If prediction "B" is made, on this round B happens with probability 100%. Of course Predict-O-Matic is not aware that predictions can influence outcomes. Instead, we will assume Predict-O-Matic is doing Bayesian inference with a prior over hypotheses, each of which assumes that the environment is IID. In other words, it is learning a single parameter $p$ which is the probability A will occur on any given round.

Claim: If the prior is s.t. any interval in $p$-space is assigned positive probability, then Predict-O-Matic will converge to predicting B with frequency 1.

Sketch of proof: If Predict-O-Matic converges to predicting B with frequency $\ge f<1$ then the environment converges to producing outcome B with frequency $f^{\prime }=0.01(1-f)+f>f$, implying that Predict-O-Matic converges to predicting B with frequency $\ge f^{\prime }$.

Remark 2: Some of the hypotheses in the prior might be intelligent agents in their own right, with their own utility functions. Such an agent can intentionally produce correct predictions to increase its probability in the belief state, until a "treacherous turn" point when it produces a prediction designed to have irreversible consequences in the outside world in favor of the agent. If it is not a self-fulfilling prophecy, this treacherous prediction will cause Predict-O-Matic to update against the agentic hypothesis, but it might be too late. If it is a self-fulfilling prophecy, it will only make this hypothesis even stronger.

Moreover, there is a mechanism that systematically produces such agentic hypotheses. Namely, a sufficiently powerful predictor is likely to run into "simulation hypotheses" i.e. hypotheses that claim the universe is a simulation by some other agent. As Christiano argued before, that opens an attack vector for powerful agents across the multiverse to manipulate Predict-O-Matic into making whatever predictions they want (assuming Predict-O-Matic is sufficiently powerful to guess what predictions those agents would want it to make).

I disagree that self-unawareness / dualism should be the default assumption, for reasons I explained in this comment [LW(p) · GW(p)]. In fact I think that making a system that knowably remains self-unaware through arbitrary increases in knowledge and capability would be a giant leap towards solving AI alignment. I have vague speculative ideas for how that might be done with a type-checking proof, again see that comment I linked.

I think that the concept of a reflexive oracle would be really useful to bring into this discussion if it hasn't already been brought up. From the abstract:

In this paper, we introduce a "reflective" type of oracle, which is able to answer questions about the outputs of oracle machines with access to the same oracle. These oracles avoid diagonalization by answering some queries randomly. We show that machines with access to a reflective oracle can be used to define rational agents using causal decision theory. These agents model their environment as a probabilistic oracle machine, which may contain other agents as a non-distinguished part.

In other words, if the Predict-O-Matic knows it will predict P = A, it assigns probability 1 to the proposition that it will predict P = A.

It's a predictor - it produces probabilities (or expected value?). There's also some rules about probability that it might follow - like if asked to guess the probability it rains next wednesday, it will give the same answer as if asked to guess the probability it will give when asked tomorrow.