Safe questions to ask an Oracle?

Please excuse me if I'm missing something, but why is whether an Oracle AI can be safe considered such an important question in the first place? One of the main premises behind considering unfriendly AI a major existential risk is that someone will end up building one eventually if nothing is done to stop it. Oracle AI doesn't seem to address that. One particular AGI that doesn't itself destroy the world doesn't automatically save the world. Or is the intention to ask the Oracle how best to stop unfriendly AI and/or build friendly AI? Then it would be important to determine whether either of those questions and any sub-questions can be asked safely, but why would comparatively unimportant other questions that e. g. only save a few million lives even matter?

OK, OK, yes, there are lots of issues with Oracle AIs. But I think most of the posts here are avoiding the question.

I can readily imagine the scenario where we've come up with logical properties that would soundly keep the AI from leaving its box, and model-checked the software and hardware to prove those properties of the Oracle AI. We ensure that the only actual information leaving the Oracle AI is the oracle's answers to our queries. This is difficult, but it doesn't seem impossible -- and, in fact, it's rather easier to do this than to prove that the Oracle AI is friendly. That's why we'd make an Oracle AI in the first place.

If I understand the problem's setting correctly, by positing that we have an Oracle AI, we assume the above kinds of conditions. We don't assume that the AI is honest, or that its goals are aligned with our interests.

Under these conditions, what can you ask?

Isn't the issue less what you ask the oracle, and more how you can be sure that in order to answer the question the oracle (or whatever it self-modifies into) won't turn you and the rest of the planet into computronium? As such, it seems that the only questions you could safely ask an oracle would be ones that you are sure are computationally tractable, or that you are sure the oracle could easily prove to be unsolvable.

What happens if we ask the oracle to handle meta questions? As an example:

Question Meta: "If I ask you the questions below and you process them all in parallel, which will you answer first?"

Question 0: "What is 1+1?"

Question 1: "Will you use more than 1 millisecond to answer any of these questions?"

Question 2: "Will you use more than 1 watt to answer any of these questions?"

Question 3: "Will you use more than 1 cubic foot of space to answer any of these questions?"

Question 4: "Will you use more than 10^27 atoms to answer any of these questions?"

Question 5: "Will you use more than 1 billion operations to answer any of these questions?"

Question 6: "Will you generate an inferential gap between us to answer any of these questions?"

If the oracle answers "Question 4 will be answered first." Then you may not want to proceed because there is an inferential gap in that answering 1+1 in the sense you probably mean it should not take more than 10^27 atoms.

Of course, the ORACLE itself is ALSO looking for inferential gaps. So if it identifies one, it would answer "Question 6 will be answered first."

That being said, this feels like a bizarre way to code safety measures.

Using the same assumtions as Manfred. "Using my own normative values to define "should". What question or questions should I ask you?"

I am not sure what exactly you mean by "safe" questions. Safe in what respect? Safe in the sense that humans can't do something stupid with the answer or in the sense that the Oracle isn't going to consume the whole universe to answer the question? Well...I guess asking it to solve 1+1 could hardly lead to dangerous knowledge and also that it would be incredible stupid to build something that takes over the universe to make sure that its answer is correct.

This seems like a really bad question. If we consider an Oracle AI and its human users as a system, then the factors that contribute to risk/safety include at least:

1. design of the OAI
2. its utility function
3. background knowledge it's given access to
4. containment methods
5. the questions
6. what we do with the answers after we get them

All of these interact in complex ways so a question that is safe in one context could be unsafe in another. You say "interpret the question as narrowly or broadly as you want" but how else can we interpret it except "design an Oracle AI system (elements 1-6) that is safe"?

Besides this, I agree with FAWS that we should (if we ought to be thinking about OAI at all) be thinking about how we can use it to reduce existential risk or achieve a positive Singularity, which seems a very different problem from "safe questions".

Assuming that you have successfully solved the problem of the AI covering the world with question-askers and that sort of thing:

"How do we write a friendly AI?" would be an important question, if you could trust the answer.

"What could we do that would make it safer for us to operate you?"

"What should we do to start a business that makes lots of money quickly, but doesn't draw peoples' attention to the fact that we have an Oracle AI?" could be useful.

"What are the top risks to the human race in near future? What should we do to reduce them?"

Revolutionize protein synthesis and cure various diseases if you have time to kill.

I enjoyed paulfchristiano's related post.

I don't think it would be unreasonable to say, "Consuming no more than 100 kWh of energy (gross, total), answer the following question: ..." I can't think of any really dangerous attacks that this leaves open, but I undoubtedly am missing some.

1. Are you afraid that you are in a computer simulation run by a vastly more powerful AI which is friendly to mankind and which is trying to evaluate whether you are friendly enough to mankind to be allowed to survive?

2. Responding as an AI friendly to mankind would, please tell me...

I suspect that right now there is not an agreed definition for "Oracle" and "Oracle AI". Can you please provide a definition to be used for the purposes of this question?

Resource limits need to be explicitly defined. Either as part of the questions, or as a built-in part of the OAI's utility function. Otherwise it's hard to imagine a safe non-trivial question.

The obvious and almost certainly correct answer to the meta-question is that there is no question that can be asked that won't cause the AI to devote as many possible resources to answering forever, given that activating it doesn't cause it to ask itself the simplest possible question and then spend eternity answering it over and over without ever interacting non-harmfully with humans.