A taxonomy of Oracle AIs

It is not clear that there is any general way to design a Predictor that will not exhibit goal-seeking behavior, short of dramatically limiting the power of the Predictor.

Not sure if this is a new idea or how safe it is, but we could design a Predictor that incorporates a quantum random number generator, such that with some small probability it will output "no predictions today, run me again tomorrow". Then have the Predictor make predictions that are conditional on it giving the output "no predictions today, run me again tomorrow".

Can someone help me understand why a non-Friendly Question-Answerer is a bad idea?

A Question-Answerer is a system that [...] somehow computes the "answer to the question.” To analyze the difficulty of creating a Question-Answerer, suppose that we ask it the question "what ought we (or I) to do?" [...]

If it cannot answer this question, many of its answers are radically unsafe. Courses of action recommended by the Question-Answerer will likely be unsafe, insofar as "safety" relies on the definition of human value.

I understand that such an AI won't be able to tell me if something is safe. But if it doesn't have goals, it wouldn't try to persuade me that anything is safe. So this sounds like my daily life: There are tools I can use to find answers to some of my questions, but ultimately it is I who must decide whether something is safe or not. This AI doesn't sound dangerous.

EDIT: Can someone give an example of a disaster involving such an AI?

I see there is no discussion here of Oracle AIs with transparent reasoning processes. How might things change if we had an automatic, detailed rationale for everything the Oracle said?

Also, I think there might be a good argument for building a true Oracle AI instead of a friendly AI once we think we have friendliness solved, if the consequences of only having friendliness almost right are less terrible (would a true Oracle AI that didn't understand that humans could get bored really be that bad? I can give good advice to my 12-year-old brother even though my model of him is not perfect, and it's not clear to me that my advice would become harmful if I was much smarter. To reverse the situation, my dad's goals for me are not exactly the same as my goals for me, and if my dad's intelligence was enhanced radically, I would prefer to hear his advice and be wary of it than to give him complete power over my life.)

But I guess that's no different than saying we should do everything we can to box an FAI even if we think we have one, with one particular kind of box being behavioral constraints added to the AI's goal function.

Since model M has much of P’s distribution’s probability mass, P(d) is approximately equal to the probability of M if M computes d (call this M→d), and zero otherwise.

I found this sentence confusing.

By the total probability rule, we can say that the probability of the data being seen in the future is the sum of these two numbers:

1. The probability that model M is true and that the data is seen.
2. The probability that model M is false and that the data is seen.

If we assume for the sake of argument that the probability of model M being true is 1, then the second number becomes zero. By the definition of conditional probability, the first number is the product of the probability of the data being seen given that model M is true and the probability that model M is true. Since we are assuming that the probability of model M being true is one, this reduces to the probability of the data being seen given that model M is true.

So could we rewrite that sentence as

Since model M has much of P’s distribution’s probability mass, P(d) is approximately equal to the probability of d given M.

Or

Since model M has much of P’s distribution’s probability mass, P(d) is approximately equal to the probability of M computing d.

In any case, it seems to me that the formalism does not add much here, and you could communicate the same idea by saying something like

The goal of the predictor is to make predictions that come true. However, the predictor must take into account the effect that sharing its predictions has on the course of events. Therefore, to make accurate predictions, the predictor must make predictions that continue to be true even after they are shared.

In particular, at any given time it will make the prediction that has the greatest likelihood of continuing to be true even after it is shared, in order to best achieve its goal. This will lead to the predictor behaving somewhat like the Oracle in Greek mythology, making whatever prophecy is maximally self-fulfilling, sometimes with disastrous results.

As far as I know, the predictor argument was first stated by roystgnr in April 2011. Thanks for drawing more attention to it.

My proposed classification scheme in this area is here: oracles, sages, genies.

It would be good to hammer-out some agreed-upon terminology.

I think a useful distinction can be made between different "Question-Answerers that cannot answer the 'what ought we to do' question". Although all answers from any such system are unsafe, they can be unsafe in different ways:
(1) unsafe in the sense in which any new human knowledge is unsafe, or
(2) unsafe in specific UFAI-related risk sense, which is much more unsafe.
A system that gives the first kind of answers can be thought of as a domain-specific intelligence enhancer for people.

This is the first time I've seen this Predictor argument. It would be nice to make a more formal version of this argument (maybe a theorem that certain Predictors become certain kinds of expected-utility maximizers), but doing so would require some formalization of logical uncertainty.

Very accurate and general Predictors may be based on Solomonoff's theory of universal induction. Very powerful Predictors are unsafe in a rather surprising way: when given sufficient data about the real world, they exhibit goal-seeking behavior, i.e. they calculate a distribution over future data in a way that brings about certain real-world states. This is surprising, since a Predictor is theoretically just a very large and expensive application of Bayes' law, not even performing a search over its possible outputs.

I am not yet convinced by this argument. Think about a computable approximation to Solomonoff induction - like Levin search. Why does it "want" its predictions to be right any more than it "wants" them to be wrong? Superficially, correct and incorrect predictions are treated symmetrically by such systems.

I have not previously encountered the Predictor argument, but my immediate thought is to point out that the fidelity with which the predictor models its own behavior is strongly limited by the threat of infinite recursion ("If I answer A then I predict B so I'll answer B then I predict C so I'll answer C then I predict D..." etc). Even if it models its own predictions, either that submodel will be enough simplified not to model its own predictions, or some further submodel will be, or the prediction stack overflows.

Puts me in mind of a Philip K Dick story.

I feel like your discussion of predictors makes a few not-necessarily-warranted assumptions about how the predictor deals with self-reference. Then again, I guess anything that doesn't do this fails as a predictor in a wide range of useful cases. It predicts a massive fire will kill 100 people, and so naturally this prediction is used to invalidate the original prediction.

But there is a simple-ish fix. What if you simply ask it to make predictions about what would happen if it (and say all similar predictors) suddenly stopped functioning immediately before this prediction was returned?

Very powerful Predictors are unsafe in a rather surprising way: when given sufficient data about the real world, they exhibit goal-seeking behavior, i.e. they calculate a distribution over future data in a way that brings about certain real-world states.

This isn't necessarily the case. What the loopiness of predictors shows is that a simple predictor is incomplete. How correct a prediction is depends on the prediction that is made (i.e. is loopy), so you need another criterion in order to actually program the predictor to resolve this case. One way to do this is by programming it to "select" predictions based on some sort of accuracy metric, but this is only one way. There may be other safer ways that make the predictor less agenty, such as answering "LOOPY" when its would-be predictions vary too much depending on which answer it could give.

An Advisor is a systems that takes a corpus of real-world data and somehow computes the answer to the informal question "what ought we (or I) to do?". Advisors are FAI-complete because:

• Formalizing the ought-question requires a complete formal statement of human values or a formal method for finding them. Answering the ought-question requires a full theory of instrumental decision-making.

There may be ways around having an FAI-complete Advisor if you ask somewhat less generic questions. Maybe questions of the form: "Which of these 3 options would it be best for me to do if I want to satisfy this utility function?" Where "this" is the best you've found so far. You probably can't ask questions like this indefinitely, but with a smart ordering of questions you could probably get a lot done quite safely. This all assumes you can build an initial Advisor AI safely. How do you create a powerful AI without recursive self-improvement or that doesn't have goals?

I don't like your naming of AI vs "smart calculators". All of your examples are of intelligent things.
True Oracle AIs = Oracular AI with goals
Oracle non-AIs = Oracular AI without goals
Maybe when we are specifically talking about the subset of AI that have goals, we should start using the term Artificially Intelligent Agent (AIA). Can someone think of a better term?

This means that P(d) depends on P(d)’s predicted impact on the world; in other words, P takes into account the effects of its predictions on the world, and “selects” predictions that make themselves accurate

Utility indifference can avoid this kind of problems.

Given that a True Oracle AI acts, by answering questions, to achieve its goal, it follows that True Oracle AI is only safe if its goal is fully compatible with human values.

Since an oracular goal must contain a full specification of human values, the True Oracle AI problem is Friendly-AI-complete (FAI-complete). If we had the knowledge and skills needed to create a safe True Oracular AI, we could create a Friendly AI instead.

I disagree. It might feel likely that an Oracle needs to be a FAI, but by no means is this established. It is trivially true that there are motivational structures that it is safe for a boxed Oracle to have, but unsafe for an AI (such as "if you're outside of the box, go wild, else, stay in the box and be friendly"). Now that's a ridiculous example, but it is an example: so the "Oracle AI as dangerous as free AI" is not mathematically true.

Now, it's pretty clear than an Oracle with an unlimited input-output channel and no motivational constraints is extremely dangerous. But it hasn't been shown that there are no combinations of motivational and physical constraints that will make the Oracle behave better without it having to have the full spectrum of human values.

Some though experiments (such as http://lesswrong.com/lw/3dw/what_can_you_do_with_an_unfriendly_ai/) seem to produce positive behaviour from boxed agents. So I would not be so willing to dismiss a lot of these approaches as FAI-complete - the case has yet to be made.

A limited interaction channel is not a good defense against a superintelligence.

Very accurate and general Predictors may be based on Solomonoff's theory of universal induction. Very powerful Predictors are unsafe in a rather surprising way: when given sufficient data about the real world, they exhibit goal-seeking behavior, i.e. they calculate a distribution over future data in a way that brings about certain real-world states. This is surprising, since a Predictor is theoretically just a very large and expensive application of Bayes' law, not even performing a search over its possible outputs.

I claimed that Solomonoff induction did not do that here. Does anyone disagree? Is this a point of contention?

Criticizing bad arguments for a conclusion you agree with generally makes you more trustworthy, and leaving bad arguments stand lowers the sanity waterline, especially when people who perceive that "smart money" is on your conclusion are only exposed to the weaker arguments and end up corrupting their mindware to accept them.

Unless the real arguments that you have are dangerous in some way why not present them in lieu of the weak ones.

An Oracular non-AI is a question-answering or otherwise informative system that is not goal-seeking and has no internal parts that are goal-seeking, i.e. not an AI at all. Informally, an Oracular non-AI is something like a "nearly AI-complete calculator" that implements a function from input "questions" to output "answers.”

What if I ask it "What should I do?" or "What would Cthulhu do?" Questions can contain or point to goal-seeking structure, even if non-AI on its own doesn't. AI may be unable to give an accurate answer, but still engage in goal-seeking behavior, so it's not a clear reduction of FAI (as you argue in "Oracular non-AIs: Advisors").

Predictor's output being an influence on its prediction also reduces to at least an Oracular non-AI, and shows one way in which a "non-AI" can exhibit goal-seeking behavior.

Given that a True Oracle AI acts, by answering questions, to achieve its goal, it follows that True Oracle AI is only safe if its goal is fully compatible with human values. A limited interaction channel is not a good defense against a superintelligence.

Bad argument for the right policy. What does it follow from that "True Oracle AI is only safe if its goal is fully compatible with human values"? You just said "it follows", like "may I use the copy machine because I need to make copies". There may well be other circumstances other than AI's goals being "fully compatible" (what does that mean?) with human values, that render (this particular instance of) AI safe, this question is presumably the topic of this post. And a sufficiently limited interaction channel (e.g. a few output bits without timestamp) may well be a sufficiently good defense against a superintelligence.

(Good arguments include insufficiently limited interaction channel being dangerous, and us not knowing how much limitation is sufficient, the AI escaping on some other occasion if the technology to create it exists, AI pretending to be dumber than it is, and its operators opening up too wide an interaction channel or just dropping some parts of security in an attempt to make it useful, etc.)

Very powerful Predictors are unsafe in a rather surprising way: when given sufficient data about the real world, they exhibit goal-seeking behavior, i.e. they calculate a distribution over future data in a way that brings about certain real-world states.

You are speaking the language of observation and certainty, which I think is inappropriate in this case. There are probably different kinds of predictors, not all of which exhibit such features. You are essentially asking, "How would I, as an optimizer, design a good Predictor?", but a predictor doesn't obviously have to be built this way, to consider counterfactuals following from various predictions; it could just be helpless to influence its prediction in a way that makes its prediction more accurate, broken as an optimizer. It may also, as usual, be impossible to control anything in the physical world to any useful extent without a sufficient output channel (or, equivalently, systems that predict the predictor).

Thank you for posting this. I was just wondering about oracle non-AIs today.

In addition to the problems with specific proposals below, many Oracular non-AI proposals are based on powerful metacomputation, e.g. Solomonoff induction or program evolution, and therefore incur the generic metacomputational hazards: they may accidentally perform morally bad computations (e.g. suffering sentient programs or human simulations), they may stumble upon and fail to sandbox an Unfriendly AI, or they may fall victim to ambient control by a superintelligence. Other unknown metacomputational hazards may also exist.

Every time I read something like this I think, "Wow okay, from a superficially point of view this sounds like a logical possibility. But is it physically possible? If so, is it economically and otherwise feasible? What evidence do you have?".

You use math like "Solomonoff induction" as if it described part of the territory rather than being symbols and syntactic rules, scribbles on paper. To use your terminology and heuristics, I think that the Kolmogorov complexity of "stumble upon and fail to sandbox an Unfriendly AI" is extremely high.