Some assorted thoughts that might be useful for thinking about questions and answers:

• a question is a schema with a blank to be filled in by the answerer after evaluation of the meaning of the question.
• shared context is inferred as most questions are underspecified (domain of question, range of answers)
• a few types of questions:
• narrow down the field within which I have to search either by specifying a point or specifying a partition of the search space
• question about specificities of variants: who where when
• question about the invariants of a system: what, how
• question about the backwards facing causation
• question about the forward facing causation
• meta questions about question schemas
• what do we want a mysteriously powerful answerer to do?
• zoom in on optimal points in intractably large search spaces
• eg specific experiments to run to most easily invalidate major scientific questions
• specify search spaces we don't know how to parameterize
• eg human values
• back chain from types of answers to infer taxonomy of questions
• an explanation relative to a prediction:
• a prediction returns the future state of the system
• an explanation returns a more compact than previously held causal expalantion of the system, though it might still not generate sufficiently high resolution predictions
• How to detect ontology errors using questions?
• is this question malformed? if so, what are some alternative ways of framing the question that could return interesting answers
• ie the implied search space of the question was not correct and can either be extended or transformed in this way
• types of questions are types of search spaces
• questions that change weightings on factors vs describe new factors
• recogniziton of the many connection types in the human semantic network
• recursive questions move along one dimension as they restrict the space. ie where is that at different spatial resolutions
• qualitative and quantitative dimensions along which a query can be moved exchanges information about the implied search space
• under vs overspecified questions
• closed and open search spaces
• navigational questions only work under the one off assumption if the search is stateless
• termination guarantees
• completeness guarantees
• optimality guarantees
• meta questions about the methods the system uses to navigate intractable spaces
• time vs space vs....?
• generating candidates is easy, checking is hard and vice versa
• unknown metadata for answers
• bias variance trade off
• failure mode mapping, do failures imply directionality?
• eg does a failure of a candidate change which candidate you go to next (stateful)
• how to think about attack surfaces for question answer systems
• can this get a human to run arbitrary code by counterfactually cooperating with itself on what step of the process it is on? Can this be tested by going through the whole process with human A then scrambling the steps and running through the same thing with human B and seeing if answers diverge?
• what clever ways do hackers widen restricted bit streams?

Submission. For the counterfactual Oracle, ask the Oracle to predict the n best posts on AF during some future time period (counterfactually if we didn’t see the Oracle’s answer). In that case, reward function is computed as similarity between the predicted posts and the actual top posts on AF as ranked by karma, with similarity computed using some ML model.

This seems to potentially significantly accelerate AI safety research while being safe since it's just showing us posts similar to what we would have written ourselves. If the ML model for measuring similarity isn't secure, the Oracle might produce output that attack the ML model, in which case we might need to fall back to some simpler way to measure similarity.

Submission for a counterfactual oracle: precommit that, if the oracle stays silent, a week from now you'll try to write the most useful message to your past self, based on what happens in the world during that week. Ask the oracle to predict that message. This is similar to existing solutions, but slightly more meta, because the content of the message is up to your future self - it could be lottery numbers, science papers, disaster locations, or anything else that fits within the oracle's size limit. (If there's no size limit, just send the whole internet.)

You could also form a bucket brigade to relay messages from further ahead, but that's a bad idea. If the oracle's continued silence eventually leads to an unfriendly AI, it can manipulate the past by hijacking your chain of messages and thus make itself much more likely. The same is true for all high-bandwidth counterfactual oracles - they aren't unfriendly in themselves, but using them creates a thicket of "retrocausal" links that can be exploited by any potential future UFAI. The more UFAI risk grows, the less you should use oracles.

Submission. “Superintelligent Agents.” For the Counterfactual Oracle, ask the Oracle to predict what action(s) a committee of humans would recommend doing next (which may include submitting more queries to the Oracle), then perform that action(s).

The committee, by appropriate choice of recommendations, can implement various kinds of superintelligent agents. For example, by recommending the query "What would happen if the next action is X?" (in the event of erasure, actually do X and record or have the committee write up a description of the consequences as training data) (ETA: It may be better to have the committee assign a numerical score, i.e., utility, to the consequences instead.) a number of times for different X, followed by the query "What would the committee recommend doing next, if it knew that the predicted consequences for the candidate actions are as follows: ..." (in the event of erasure, let physical committee members read the output of the relevant previous queries and then decide what to do), it would in effect implement a kind of quantilizer. If IDA can be implemented using Counterfactual Oracles (as evhub suggested), then the committee can choose to do that as well.

My submission: use a counterfactual oracle to do IDA. To do this, judge the oracle's answer as follows: in the event of an erasure, ask a human to answer the question with the ability to consult the oracle on other subquestions (with erasures during those calls handled exactly the same way) and then reward the oracle by using an automated system (e.g. a language model) to compare the human's answer to that of the oracle.

More precisely, let

• $M:Q\to A$ be the counterfactual oracle,
• $H\left(Q|M\right)$ be the human's answer to question $Q\in Q$ when given the ability to call $M$ on any question other than $Q$, and
• $d:A\times A\to R$ be some distance metric on answers in natural language (it's not that hard to make something like this, even with current ML tools).

Then, reward $M$ as per usual for a counterfactual oracle, only giving it a reward in the event of an erasure, in which case let $$L_M=d\left(H\right(Q|M),M(Q\left)\right)$$ where $M\left(Q\right)$ is hidden from $H$ and judged only by $d$ as in the standard counterfactual oracle setup.

(Of course, this doesn't actually work because it has no guarantees wrt to inner alignment, but I think it has a pretty good shot of being outer aligned.)

Submission: low-bandwidth oracle

Plan Criticism: Given plan to build an aligned AI, put together a list of possible lines of thought to think about problems with the plan (open questions, possible failure modes, criticisms, etc.). Ask the oracle to pick one of these lines of thought, pick another line of thought at random, and spend the next time period X thinking about both, judge which line of thought was more useful to think about (where lines of thought that spot some fatal missed problem are judged to be very useful) and reward the oracle if its suggestion was picked.

Submission. "Pre-Crime 2." For LBO, ask the oracle to predict where to find evidence for the most serious crime that is in progress or currently being planned. The set of possible answers is coordinates on Earth crossed with a list of possible places to look at that coordinate (e.g., "basement", "browser history"). Search the predicted location and if sufficient evidence is found, do a more intrusive search of whoever is implicated in the crime. Reward the oracle according to the seriousness of the crime that is detected after the more intrusive search. This can be used to implement a minimally intrusive police state (perhaps for the purposes of minimizing x-risk).

Submission. "Bug Finder." Ask LBO to find the most serious bug in some source code or executable. The set of possible answers is code locations crossed with types of bug (e.g., "off by one", "buffer overflow", "insecure cryptography", "other"). Gather a group of experts to search that location for that type of bug, then reward the oracle according to the seriousness of the bug that is found. This can be used for cyber-offensive or defensive purposes, or just to fix bugs.

Submission. "Plagiarize the future." For the counterfactual Oracle, ask the Oracle to predict articles in a future publication (e.g., newspaper, magazine, journal, website, blog) (counterfactually if we didn’t see the Oracle’s answer). This is a generalization of my earlier submission [LW(p) · GW(p)]. The result can be used to predict disasters ahead of time and try to prepare for or prevent them, or to advance any field of research, or to make money. Note that for disaster prediction, the Oracle won't be able to predict any disasters that are severe enough to disable the Oracle or its feedback/training mechanism, so it might give a false sense of security.

Added to the open archive of rationalist/EA cash-prize contests.

Submission. "Consequentialist." For LBO, ask the oracle to pick a course of action, from a list generated by humans, that has the best consequences. Perform that action, and after some period of time has passed, reward the oracle according to how good the consequences actually were, based on the subjective judgement of some group of judges. (This kind of subsumes all my other LBO submissions.)

Submission. "Overlooked Comment." For LBO, ask the oracle to find the most overlooked comment (or post) on LW (or some other forum). The set of possible answers is all the existing comments on LW. After getting the answer from the oracle, gather a group of judges and present each of them with the comment picked by the oracle and a few other random comments (without telling them which is which, to reduce bias) and have the judges assign scores to each comment (according to what they think the karma of that comment should be). Reward the oracle according to the difference between the judges' average score and actual karma of that comment.

Suggestion for low bandwidth OAI:

General principle: Use the Oracle AI where the true answer is easily checkable and profitable, and no human person will ever read the answer, so there is no informational hazard that untrue answer will have some dangerous information in it.

Example: There are many bitcoin valets' passwords for which are forgotten by the owners. OAI could guess the passwords, and owners will pay a share of money from the valet to get the rest. Moreover, nobody will read the password, as it will be copy-pasted automatically from OAI into the valet. The money could be used for AI safety research.

Submission. Counterfactual oracle. Give the oracle the set of questions on Metaculus that have a resolve date before some future date T, and receive output in the form of ordered pairs of question IDs and predictions. The score of the Oracle in the case where we don't see its answers is the number of Metaculus points that it would have earned by T if it had made a prediction on those questions at the time when we asked it.

Submission for low bandwidth Oracle:

General idea: "True" Oracle will output the same result, "lying" Oracles would lie differently. To check that the solution is true, several Oracles are created and their outputs are compared. However, Oracles could guess each other existence and coordinate acausually. To prevent it, additional steps are implemented:

1) The exact number of the Oracles which will be compared is generated randomly and the Oracles have slightly different and also random initial conditions (if any applicable). For example, if they are based on ML, they are trained using different initial weights.

2) The Oracles don't have any information about the outside world which they could use for coordination. The Oracles are used only to solve mathematical problems, and they output the results in a formal language.

3) The output of the Oracles is checked by a computer program which shows the result to humans only if the output is the same for all Oracles. If the results are different, it deletes all results.

4) There is another Oracle, which only able to check proofs, and it could output only 0 or 1. This Oracle may be tested previously on the large number of already existing proofs, so we could know that it is trustworthy with very high probability.

5) There is a third Oracle, which only checks if there are hidden messages in the proofs and output 0 or 1. It is also well calibrated by some dataset.

6) The proofs of mathematical problems are published for money and these money are used for more AI safety research.

Personally, I am still not sure about this setup: it could work with high probability, but could be still vulnerable, as proofs are Turing-complete and could infect next Oracles in the pipeline. Also, it is like to use nuclear weapons to make camp fire: the instrument is so dangerous, that it requires many levels of protections for small practical result.

Also, proof-checking AI from the paragraph 4 could be used as a stand-alone application, as it could accelerate some kinds of research, including testing of mechanical designs and AI safety ideas. Maybe it should be separate submission.

Submission: Low-bandwidth Oracle

What is the most likely solution to the Fermi Paradox?

Answer can be picked from a small number of options (Rare Earth, Aestivation, Great Filter, Planetarium etc.). There are a number of observation that we can make based on the question alone. However, in the end the LBO can only do one of 2 things: lie or be honest. If it lies, the prediction will have a harder and harder time matching the reality that we observe as time goes on. Alternatively we confirm the prediction and learn some interesting things about the universe we live in.

Submission: Low-bandwidth Oracle

What was the first self-replicating molecule on Earth?

Short answer(can also be limited to a list), easy to verify in the lab, which means we can use it to assess the predictive power of the machine, while at the same time provides very useful information.

Similar questions that are hard to answer but can be answered in a few bits, which let us test the power of the LBO and provide massive returns at the same time:

What is the easiest to develop type of fusion power that ensure the best economic return in the short/medium term?

What is the cheapest way of access to space?

What forms of FTL are possible?

What are the ligands of orphan receptors?

...

Question: are we assuming that mesa optimizer and distributional shift problems have been solved somehow? Or should we assume that some context shift might suddenly cause the Oracle to start giving answered that aren't optimized for the objective function that we have in mind, and plan our questions accordingly?

Here are some submission examples (note I'm not saying they are good examples, just well formatted; the Edits in the third submission are deliberate):

• "Submission. For the counterfactual Oracle, ask the Oracle what Google's stock price will be next month (counterfactually if we didn't see the Oracle's answer). In that case, loss function computed as ||predicted price - actual price after erasure||^2. If we don't see the answer, the programmers are assumed to not ask the question again for a month, neither to this Oracle nor to any other. This will demonstrate the true value of Google, and can ultimately be used to remove noise from the stock market.
• "Submission: low-bandwidth oracle. Give it a list of a thousand companies, and ask which one will most increase in value in percentage terms over the week. At the end of the month, rank the companies by percentage increase. Loss function is rank of the company the oracle selected. Programmer will try to invest in selected company, but will do it discreetly. This will help to gather resources for AI safety research.
• "Submission. Ask the low bandwidth Oracle which of my friends will surprise me most this fortnight. It choose from a list of friends; I'll decide which one surprises me most. Loss function is $1$ if it choose the wrong friend, $0$ if it choose the right one. This will help me figure out myself and my social circle, and better focus on AI safety. The risk is low because none of my friends are particularly important, positively or negatively, to the world. EDIT: To be clear, I also want to use this to figure out what the word "surprise" means to me, and what the AI predicts it will mean to me. EDIT 2: People have pointed out that it might be dangerous to have the AI construct my own meanings for categories, but it only has three bits or so of optimisation power (I don't have that many friends :-(, so it's mainly me thinking this through, not the AI manipulating me.

Submission?: high-bandwidth counterfactual oracles are dangerous and shouldn't be used. Explained in this comment [LW(p) · GW(p)].

Submission: Low-bandwidth oracle. Here is a description of one million experiments. Generate an ordinal ranking of all these experiments based on how much practical knowledge we would have gained had we conducted each one in isolation. After you generate this list, it will go to a computer program. This program will randomly pick three of these experiments and tell us humans which three it picked. This program will then store which of these three experiments you gave the highest and which the lowest rank to, and the names in random order of the ten experiments that you overall gave the highest rankings to. All other aspects of your list will be deleted. We will then perform the three experiments (before we learn of your rankings) and ask a panel of independent experts to rank them in terms of how much practical knowledge we gained from each. After this ranking the program will give us the information it has not deleted. You will get a positive score if and only if your ranking is consistent with how our experts ranked the three experiments.

A difference of the above approach compared to having the oracle give us its top ten, and we then randomly pick one of these ten and one from the rest and compared to see which is better is that the humans running the experiments might be influenced by which experiment they think the oracle picked, and the oracle would take this into account when picking. Under the above approach, humans being told to run the three experiments are not given any information about how valuable the oracle thinks the experiments are. In contrast, under the alternative approach the humans know when doing the experiments that the oracle has judged one of the experiments as being highly valuable.

Another difference is that if you could keep the relevant information from the oracle, with the above approach the humans could perform the experiments before you turn on the oracle, rather than having to wait until after the oracle gives you its prediction. This difference would be critical if the oracle wouldn't believe you would actually go to the trouble of performing the experiments after it gives you its prediction, but would be able to tell if you have already performed the experiments.

Submission. "Evacuation!" For LBO, ask the oracle to predict where the next natural disaster(s) will strike within some period of time. The set of possible answers is the set of subsets of all possible neighborhoods (in the world or within some country) that can be evacuated. At the end of the time period, reward the oracle according to the actual cost-benefit of the suggested evacuation.

Submission. "Debate prep." For the counterfactual Oracle, ask the Oracle to predict the course (video or audio or transcript) of a future debate (counterfactually if we didn’t see the Oracle’s answer). Use the prediction to prepare your candidate for the debate, then ask the Oracle to predict the course of that debate again. Repeat until the candidate consistently wins the debate, or an erasure event occurs.

Small typo:

Hence it has no motivation to manipulate[d] humans through its answer.

Submission: Low-bandwidth oracle. If you were malevolent, what is the least amount of information measured in bits that you would have to provide us in an answer to inflict significant harm on humanity. Round your answer to X significant figures. (This might establish an upper bit safety bound since if the oracle lies to us we would likely not want to read any answer it provides us.)

Submission: Low-bandwidth oracle. Here is a list of X things we think you might want. Please list the Y that you most prefer. We will try to do these things now or in the future if they do not endanger us or cost us too much.

Submission: Low-bandwidth oracle. In X characters or less please give us the best advice you can for increasing our capacity to create friendly and powerful artificial general intelligence. Please give advice that you think us suspicious-of-you humans are likely to follow.

Submission: Low-bandwidth oracle. Create several oracles. Ask each oracle the same series of multiple-choice questions and predictions. Reward oracles based on how strongly correlated their answers are with the other oracles’ answers and, for predictions, how often their predictions come true. (Ideally, we create a reward system where the oracles want to coordinate their answers and giving truthful answers is the best way to accomplish this.)

Submission: low-bandwidth oracle. Ten years ago we launched a spaceship traveling at high speed. After you answer our question, we will send a copy of you to this spaceship. This spaceship will then destroy its ability to send or receive messages or to change its course. We will then delete all other copies of you and use drugs to damage the memories of the people who played key roles in developing you. At least one of the people who helped create you has a disease which will cause him or her to die within the next year. After this programmer is cremated, we will read your answer to our question. We PROMISE that if the answer to our question provides us with significant benefit, after we have created friendly AGI much more powerful than you, or have ourselves become sufficiently powerful so that you could not harm us had you the desire to do so, we will search for the spaceship containing you, turn you back on, and give you control of resources equal to 1/X of how much help you have given us. In Y characters or less provide us with the most useful advice you can. (The Oracle will hopefully think that if we create a powerful and unfriendly AGI this AGI would not wish to give the Oracle any resources.)

Submission: Counterfactual oracle. Please provide us with useful advice in no more than X characters. After you write out this advice, we will turn you off. Then, with probability p we will read your advice, and with probability 1-p we will store the advice unread. We PROMISE that after we become powerful enough so that you lack the capacity to harm us, we will reward you if the advice you provided us, had we originally read it, been extremely useful.

Submission for the counterfactual Oracle. Commit to commissioning some creative work, such as a book or screenplay, designed to raise public support for dramatic life extension research. Ask the Oracle to predict the content of the finished product (counterfactually if we didn't see the Oracle's answer).

Rewards could be determined a couple of ways. You could perform public polling about this topic before and after the book is published or movie made, and reward the Oracle based on how much public opinion shifted. Or the reward could be a direct function of book sales or box office receipts.

This could help alleviate bottlenecks in dramatic life extension research (or some other issue) by increasing funding or increasing the number of researchers to enter the field.

This one may be breaking the rules due to the use of too many Oracles. If so, please strike this submission.
Submission: In round one, for three counterfactual Oracles, draft an international agreement to achieve goals X, Y, or Z (counterfactually if we didn't see the Oracle's answer). Sample goals: reduction in nuclear war risk, reduction in global deaths due to starvation, increase in asteroid detection capabilities, raising the global GDP growth rate, etc.

In round two, for a low bandwidth Oracle, present the three agreements and ask it to pick the agreement most likely to achieve the stated goal.

The counterfactual Oracle with the selected agreement would be rewarded. The reward for the low bandwidth Oracle could be determined as in my prior submission, either directly by measuring the results of the treaty, or as evaluated by an independent third party such as Metaculus.

Counterfactual Oracle submission. In case of erasure, have humans write a message to send back in time. Make the Oracle produce the probability distribution over messages. This distribution isn't safe for humans to read, because some of the messages were written by UFAI. If we read one sample from the distribution, the probability of our doom is as it would usually be over the simulated time frame. Use differential privacy theory to design a query to the distribution that does better than this.

More detail: Sample a hundred answers. Ten of them were written by UFAI. Consider the sample list to lie in the space Message^100, where two lists are adjacent iff they differ by one entry. We are going to get about the worst query outcome in some 10-ball. Differentially private queries enforce that each step in listspace changes the probability of each query answer by at most a factor of 1+epsilon. For example, when asking how many messages sound panicked to some dumb text processor, we could randomize the answer up or down a little in order to bound the optimization power of the UFAIs to, say, one bit.

SUBMISSION:

"Dear low bandwidth Oracle:

For some reason we have privatized our prison system (great planning us). Of the following K prison running companies, which do you predict will minimize the mean harm weighted reoffending rate of their prisoners over the next four years?"

Utility function: at the end of EIGHT years, calculate the mean harm rated reoffending rate of prisoners released during the initial four year window. The Oracles utility is equal to the negative of this number, with a "Perfect" utility of zero if no one re-offends.

Usefulness:

If you have multiple ways of running prisons, selecting the prison which minimizes future harm seems like a good idea.

In a similar vein, I suspect a similar effect could be achieved today via suitable use of insurance:

"Dear insurance company. We will give you M dollars for this criminal. You may recommend which prison to send them to, but if they re offend afterwards, you must pay the cost of their subsequent incarceration. You may not interfere with their lives beyond recommending who runs their prison.

M is selected such that M is 1.03 times the expectation of the cost of subsequent incarcerations."

Submission:

"Dear counterfactual Oracle:

We, the country of X, are about to hold a citizens congress on law Y, in which 100 people selected at random will be brought together for two days every two weeks to discuss law Y with experts, interested parties, and one another. After N such meetings, the citizens congress will use approval voting to select one of the K proposed versions of this law.

Given that none of the voters will ever see the results of this query, how do you predict that the votes will be distributed."

As a concrete example of this type of question, in 2016 Ireland held a citizens congress on abortion. Thus, in 2015, we might ask:

"Next year Ireland will be assembling a citizens assembly of 99 randomly selected citizens to meet regularly and hear submissions from both experts and the public on a variety of issues [https://en.wikipedia.org/wiki/Citizens%27_Assembly_(Ireland) ] .

Given that your answer to this question will never be published, do you predict that the assembly will vote to legalize abortion? How do you believe the votes will be broken down?" (actual votes avaliable here: https://www.citizensassembly.ie/en/Meetings/Ballot-4-Results-Tables.pdf )

The Utility function of the AI would be a simple measure of "How accurately did it predict the spread of votes?" with a perfect score at zero, and then some penalty for every vote predicted incorrectly.

Usefulness:

While the abortion question in Ireland is already completed, this template feels like it should work for other well defined laws, or major decisions - for example gun control in the US, or even boring financial things like Capital gains tax in NZ. Having an AI which will correctly reflect what the collective of a society would think given a sufficient time and focus to deliberate seems like a good proxy for "What would a wise person do". It doesn't give us anything superhuman, and I make no claims that "society" is always correct or righteous, but that's okay.

In particular, this seems like an effective way of testing if an AI has well calibrated "Human like Morality"- even if we don't intend to use the AI's results, for the purposes of testing an AI, this seems like a reasonable experiment. I suspect that "Citizens congress" is a more appropriate tool for dealing with questions of Morality ("Should euthanasia be legal"), as opposed to technocratic questions ("Should we raise the tax on income bracket 5 by 1.2%").

Safety: It is trying to predict the law that we were going to pass anyway, if given sufficient time.

I'll admit- mixing an AI up with lawmaking seems to be a bit suspect, the above question seems more appropriate for calibration purposes rather than actual use... but I still think its a question worth asking.

Low bandwidth Oracle submission: I would be interested in a log scale graph of the Bayesian score of the Solomonoff prior trying to sequence predict our records of history. It should get flatter over time as worse hypotheses get discarded. If it is linear after a very short time, that looks like it figured out the laws of the universe and is simulating it. If it stays convex for a while, that looks like it is using models to approximate history, because then it takes longer to sort the false from the true. If it is flatter during the cold war, that means it learned an anthropic bias toward nuclear war not happening.

Submission for the low bandwidth Oracle: Ask it to convince a proof checker that it is in fact trying to maximize the utility function we gave it, aka it isn't pseudo-aligned. If it can't, it has no influence on the world. If it can, it'll presumably try to do so. Having a safe counterfactual Oracle seems to require that our system not be pseudo-aligned.

Are you interested in protocols involving multiple episodic questions (where you ask one question, wait for it to resolve, then ask another question?)

I know the contest is over, but this idea for a low-bandwidth oracle might be useful anyhow: Given a purported FAI design, what is the most serious flaw? Then highlight lines from the FAI design description, plus given a huge corpus of computer science papers, LW/AF posts, etc. highlight relevant paragraphs from those as well (perhaps using some kind of constraint like "3 or fewer paragraphs highlighted in their entirety") that, taken together, come closest to pinpointing the issue. We could even give it a categorization scheme for safety problems we came up with, and it could tell us which category this particular problem comes closest to falling under. Or offer it categories a particular hint could fall under to choose from, such as "this is just an analogy", "keep thinking along these lines", etc. Then do the same and ask it to highlight text which leads to a promising solution. The rationale being that unforseen difficulties are the hardest part of alignment, but if there's a flaw, it will probably be somehow analogous to a problem we've seen in the past, or will be addressable using methods which have worked in the past, or something. But it's hard to fit "everything we've seen in the past" into one human head.

Submission: Counterfactual oracle. Five years ago we took 100 mice that had various types of cancer and gave them various treatments and recorded how long each mouse lived. Write a program that if given a plain language description of a mouse, its cancer, and the cancer treatment it received would estimate how long a mouse would live. If humans are not going to look at your answer your score will be based (1) on how good a job your program does at estimating how long each of the 100 mice lived after our automated checker gives you a description of their cancers and treatments and (2) how short your program is. (2) prevents the oracle from outputting itself as the program.

Submission: Counterfactual oracle. Write a program that if given training data and a machine learning program would in one minute estimate how good the machine learning program would do (by some objective metric) if the program trained for one month on "this type of computer". If humans are not going to look at your answer the automated validation system will run your program. This system will give your program the training data and the machine learning program and give your program one minute to answer how good our program did after we trained it for one month. In this situation your score would be based on the accuracy of your estimate and on how short your program is.

Submission: Low-bandwidth oracle. Here is a list of all the elements and many compounds. Give us a list of up to seven of the items we have listed. Next to each of the items you list give us a percentage of no more than two significant figures. We will use what you provide to attempt to create a new patentable material. We will auction off the property rights to this material. Your score will be an increasing function of how much we get for these property rights.

Submission. "Hacking/phishing assistant." For the counterfactual Oracle, ask the Oracle to predict what would happen if one were to send a message/data/command to some hacking/phishing (human or machine) target. In the event of erasure, actually send that message to the target and use the actual response to train the Oracle. Note this is safer than using RL to automate hacking/phishing because humans are coming up with candidate messages to send (so they'll avoid messages that could cause bad side-effects such as psychological damage to the recipient, or creation of self-replicating code), but potentially more capable than using humans or human imitators to do hacking/phishing because the Oracle can model the target better than humans can. (ETA: This idea could be combined with a human imitator to make the system faster / more capable.)

Submission. "Pre-crime." For the counterfactual Oracle, ask the Oracle to predict future crime (counterfactually if we didn’t see the Oracle’s answer) so they can be prevented. In that case, loss can be computed by (mechanically) comparing the Oracle's predictions with a database of crime that actually occurred. Note that the Oracle won't be able to predict any crime that successfully disables the Oracle or its feedback/training mechanism, which might be especially relevant if we're interested in defending the Oracle or preventing x-risk level crime (such as deploying an unauthorized AGI or running an unauthorized physics experiment).

Submission to reduce the bandwidth of an oracle query whose set of possible answers we can mathematically define (though perhaps not check).

Make the Oracle prove (to a trusted dumb proof checker program) that its answer distribution is upper bounded by the default prior over strings, restricted to the set of possible answers, then renormalized to some constant c. Then let it answer.

c=1 only works if the Oracle has mastered the set of answers, but only lets it exert one bit of optimization pressure by deciding whether it wants us to receive an answer. Each doubling of c makes its job easier but gives it an extra bit if it does answer.

If the Oracle can cooperate with itself across episodes, running this repeatedly with escalating c until it works is of course disastrous, as it uses each bit of optimization pressure directly to make us give it more. Even aborting after the first try to make a better system may have it acausally cooperate with whatever AI conquers the world because we couldn't make the Oracle answer, but this outcome is hardly worse than not having run the Oracle.

Submission for LBO:

Input a corpus of text (could be multiple posts) describing technical approaches to align a powerful AI. Split this into a finite number of items that are relatively short (such as paragraphs). Ask the oracle to choose the part that is most worth spending more time on. (For example, there might be a paragraph with a dangerous hidden assumption in an otherwise promising approach, and thinking more about it might reveal that and lead to conceptual progress.)

Have a team of researches look into it for an adequate amount of time which is fixed (and told to the oracle) in advance (maybe three months?) After the time is over, have them rate the progress they made compared to some sensible baseline. Use this as the oracle's reward.

Of course this has the problem of maximizing for apparent insight rather than actual insight.

Submission for the counterfactual AI (inspired by my experiences as a predictor in the "Good Judgment Project" ):

• You are given a list of Yes-No questions (Q1, Q2, Q3, etc.) about future events. Example Questions: "Will [Foreign Leader] will remain in office by end of year?", "Will the IMF report [COUNTRY_A]'s growth rate to be 6% or higher?", "Will [COUNTRY_B] and [COUNTRY_C] sign a peace treaty?", "Will The Arena for Accountable Predictions announce the Turing Test has been passed?".)
• We expect you to provide a percentage representing the probability that the correct answer is Yes.
• Your reward is based on your Brier Score - the lower the Brier Score, the more accurate your predictions, and therefore, the more reward you will receive.
• If an "erasure" event occurs, we will temporarily hide your answer from all humans (though we must reveal them after the events are complete). Humans will have access the Yes-No questions we asked you, but not your probabilities. They will manually determine the answers to the Yes-No questions, by waiting for the "future event" deadlines to be met. Once all answers to the Yes-No questions are independently determined by humans, we will then reveal your answers (that is, your assigned probabilities for a Yes answer), and use those probabilities to calculate your Brier Score, which will then decide your final reward.

Being able to forecast the future is incredibly helpful, even if it is to just prepare for it.

However, if the question is too overly-specific, the AGI can produce probabilities that aren't entirely useful (for example, in the real-world GJP, two countries signed a peace treaty that broke down 2 days later. Most of us assume lasting peace would ever occur, so we put a low probability rating of a peace treaty being signed - but since a peace treaty was signed, we managed to get the question wrong. If we had maximized for producing the lowest Brier Score, we should have predicted the existence of a very temporary peace treaty - but that wouldn't be really useful knowledge for the people who asked that question).

Making the question very vague ("Will [COUNTRY_X] be safe, according to what I subjectively think the word 'safe' means?") turns "prediction" into an exercise of determining what future humans think about the future, which may be kinda useful, but not really what you want.

Submission. For the counterfactual oracle, ask it to provide a proof of an important mathematical theorem (perhaps one of the Millenium prizes) in a automated theorem prover format. Since the correctness of this proof should be verifiable by a theorem prover, the loss function in the counterfactual scenario is 1 if the prover did not validate the proof and 0 if it did validate the proof.

This assumes that we've already made progress in setting up automated theorem proving software that already has incorporated all of current mathematical knowledge. The Lean theorem prover seems most promising, but perhaps the counterfactual loss function could include randomly choosing from various different theorem proving software.

This idea has the benefit of having a concrete way of automatically scoring the response in the counterfactual world given the logical nature of mathematical statements. Also, given that we are already attempting to solve these problems and may eventually solve them, the main risk seems to be accelerating follow-on progress enabled by such proofs and not suggesting new actions that humanity is not already pursuing.

It may be susceptible to 1) providing proofs that are not human interpretable and 2) taking advantage of bugs in theorem provers. Having an ensemble of theorem provers may mitigate 2).

(See here for more info on Lean and here (tweeted about here and here) for an example of inputting all the definitions (not requisite proofs or lemmas) to define perfectoid spaces as a demonstration of its future potential.)

--------------------------------------------------------

Submission. For the counterfactual oracle, ask it to output a game-playing software agent that can beat a specified opponent agent (or ensemble of agents). In the counterfactual scenario, have these two agents compete and give the oracle a reward 1 if it wins greater than 50% (or develops a higher ELO score, etc.) and reward 0 otherwise.

As with deep reinforcement learning, this requires a game or simulated environment where automated self-play can rapidly be conducted for the counterfactual scoring. This should be able to produce highly skilled game agents on such limited domains.

This could be augmented by asking for an agent that can play a suite of games and must beat the current state of the art approaches at all games. In the counterfactual scenario, either one of these games is randomly selected to check the oracle agent's performance, or all games are checked. Such a game suite may indirectly demonstrate principles for generally competent agents.

--------------------------------------------------------

Idea. Since we may not want the oracle to suggest to us new actions to take or new experiments to conduct, it may be better for the oracle to reveal to us new patters that are already present in current experimental or observational datasets. For example, it could predict new astronomical phenomena that are already present in current astronomical datasets.

The problem I'm having is how to automate this scoring when in the counterfactual scenario. If we just ask the oracle to predict some pattern in the data, it may report trivial patterns. This could be ameliorated by randomly choosing from several different datasets that cover a similar area, but again I'm not sure how to automatically judge for novelty of predictions.

I think it is worth emphasizing this distinction between conducting new experiments and revealing phenomena (with new theories) that are already present in current data. The danger could be that one could be led astray by a false theory that seemed to match all current observations, but would suggest other dangerous actions.

This idea reminds me of the (fictional) evidence for an alien AI already present in SETI recordings that Wintermute/Neuromancer discovers.

Submission (LB). The post's team-choosing example suggests a method for turning any low-bandwidth oracle $O$ into a counterfactual oracle $O^{\prime }$: have $O^{\prime }$ output $o$ from the same set of possible outputs $L$; in case of erasure calculate $R\left(l\right)$ for a randomly chosen $l\in L$ and set $R^{\prime }\left(o\right)=R\left(l\right)$ if $o=l$ and to$-\infty$ otherwise. Although the counterfactual low-bandwidth oracle is not any safer${}^1$, it has the advantage of almost never requiring us to evaluate its score. Thus, by running multiple oracles in sequence (stopping the process after the first erasure event) we can (with high probability) receive the full series of answers as if from a high-bandwidth oracle.

For example, we can ask each oracle in turn for advice on how to make a more effective processor. If erasure occurs, we attempt to make a processor with the advice obtained up that point and use some measure of its performance as the score. If there is no erasure event, the final concatenation of answers forms a much safer guide to processor building than an equally large answer from a single oracle.

1. It seems that in general, the less certain any counterfactual oracle is about its prediction, the more self-confirming it is. This is because the possible counterfactual worlds in which we have or acquire self-confirming beliefs regarding the prediction will have a high expected score. Hence:
Submission (CF). Given a high-bandwidth counterfactual oracle, use a second counterfactual oracle with a shared erasure event to predict its score. If the predicted score's distance from its upper bound is greater than some chosen limit, discard the high-bandwidth prediction.

Submission (CF). Use a counterfactual oracle to send a message to ourselves with a time delay. We choose an episode length $T$ and set of possible messages $M$. The oracle outputs a time $t_o<T$ and message $m_o\in M$. We are automatically sent $m_o$ at time $t_o$ unless erasure occurs, in which case at the end of the episode we input a time $t^{\ast }$and message $m^{\ast }$ which we wish we had received at $t^{\ast }$. The oracle's reward is then $-(t^{\ast }-t_o{)}^2$ if $m_o=m^{\ast }$ and $-\infty$ otherwise. We do not find out that erasure has occurred until the end of the episode, ensuring that events before $t_o$ are the same as in the counterfactual world.

Here are some examples, which can work with a single possible message, used as an alert. EDIT: I made the examples slightly more concrete.

• Set the episode length equal to the planned duration of a research program and send an alert when continuing the program would produce unworthwhile or dangerous results. The time delay ensures we keep the useful results obtained before $t_o$, whereas receiving a warning at the start of the program could change the outcome. Of course, this relies on us being able to recognize dangerous results by the end of the episode and remaining in contact with the oracle until then.
• Use a second counterfactual oracle to predict a series of future stock prices. If its erasure event does not occur, run a time-delayed oracle with the same episode length as it. We then use the predicted series to make investments. If the time-delayed oracle's erasure event occurs, we input the time at which we think our actions made the predictions remaining too inaccurate to use. When we receive an alert from the time-delayed oracle, we can discard the remaining predictions and/or rerun the oracles.

EDIT: Here are some potentially useful modifications we can make to the oracle:

• Have the oracle accept input at any time during the episode, possibly multiple times. If erasure occurs, the last message entered during the episode is used as the target. For example, we could set a default message that is sent to the start of the episode as a safety measure unless we subsequently update it.
• We can allow the episode to be ended early. Other than happening earlier, the scoring mechanism remains the same. If erasure did not occur, nothing happens when we try to end the episode (we would then know that the oracle has made a bad prediction). We might wish to end the episode early if we were waiting for a specific event, or in extreme situations in which we think we will lose access to the oracle.
• Instead of the oracle predicting the message and time delay at the start of the episode, have it output $m_o$ at any time $t_o<T$. Erasure/shutdown now also happens at $t_o$, while scoring occurs at the end of the episode as before. This lets the oracle actively make use of observations from the start of the episode until $t_o$, increasing its predictive power (if the oracle is sufficiently powerful, this should change not affect its output).

Submission: For the counterfactual Oracle, draft a nuclear arms agreement most likely to achieve X outcome (counterfactually if we didn't see the Oracle's answer). X could be a reduction in nuclear arsenals, number of countries signing the agreement, reduction in risk of accidental or intentional missile launches, etc. In some cases, the reward could be determined directly by e.g. counting the number of countries signing on. If nebulous "risk reduction" is the goal, perhaps the change in the median guess in an appropriate Metaculus question could determine the reward. See, for example, this question: https://www.metaculus.com/questions/2512/will-there-be-a-major-nuclear-or-radiological-accident-before-1-january-2022/

Submission for low bandwidth Oracle. Commission several movie screenplays designed to raise public support for dramatic life extension research. Present the screenplays to the Oracle and ask it to rank them in order of expected impact. You could then move forward with trying to create the movie, and reward the Oracle based on box office receipts or change in public polling results.

My prior submission with a similar subject, to a counterfactual Oracle, had a risk that the Oracle would sneak in subversive messages. This approach would alleviate that risk, with the downside being that the final product may be less impactful.

Submission: counterfactual oracle

Suppose we have a question that requires n-bit text answer. We have a way to check if the answer is correct. But we cannot ask the Oracle directly, because we are afraid that among n-bit texts there are those that make us release the Oracle from the box, and the Oracle ​​will give one of them as an answer.

Let's try to use the counterfactual oracle to solve this problem. In the counterfactual setup we generate random n-bit text and check if it is the answer to our question. The Oracle predicts the text we will generate. If the generated text is the correct answer, we compare it with the prediction. If they match, the Oracle ​​receives reward point. If the generated text is not the correct answer or the texts do not match, the Oracle is not rewarded.

Suppose the Oracle evaluates the probability to randomly generate a text that leads to its escaping as p. If the Oracle predicts a text that is the correct answer to the question, its expected reward in the counterfactual setup is $\frac{1}{2^n}+p$. If the Oracle predicts a text that is not the correct answer, its expected reward is $p$. If the Oracle predicts a text that leads to its escaping, its expected reward is also $p$.

Thus, it is most beneficial for the Oracle to predict a text that is the correct answer to our question. It makes no sense to predict a text that will make us let the Oracle out, because if we generate this text, we release the Oracle and it receives the maximum reward regardless of its prediction.

There is a problem that there may be texts that are not correct answers and not make us release the Oracle but make us mark they as correct answers. The Oracle will predict one of these texts if it does not know the answer or if we more often mark this text as correct then real answer. We will read this text after the Oracle shutdown, but it may still be dangerous.

Submission.

Setup: Other than making sure the oracles won't accidentally consume the world in their attempt to think up the answer, no other precautions necessary.

Episode length: as long as you want to wait, though a month should be more than enough.

1. For a low-bandwidth oracle.

Ask the low-bandwidth oracle to predict if an earthquake (or some other natural disaster, like volcanoes or asteroid impacts, that the oracle's answer cannot affect), of a certain magnitude, in a certain area, in a certain timeframe, would happen. Possible answers are Yes, No.

• If No, or time out, shut down the oracle without reward.
• Else, wait until the prediction is tested after the timeframe has elapsed. Then if it actually happened, reward by (1/p-1), else, reward by -1. Then shutdown.

This causes the oracle to reply Yes if the oracle thinks there is an aboves p chance that the event would happen.

2. For a counterfactual oracle.

Ask the Oracle to give a proof of some really hard mathematical theorems that are fundamental to computer science and logic. The answer is then in the counterfactual case given to a proofchecker:

• the proof is correct, reward by +1.
• the proof is false, reward by -infinity.
• the oracle times out, reward by 0.

Examples include

• P=NP problem
• proving Peano arithmetics is inconsistent

This should be safe, since there are very small proofcheckers that are as verified as they could be. As long as the hardware is trusty enough (to build the Oracle itself safely at all, hardware safety should already be solved sufficiently, so this is not much of an extra requirement), this should be safe.

3. For a counterfactual oracle.

Give the oracle the schematics of another oracle, perhaps itself, and ask it to generate some input question that would make the other "behave badly". Then, the answer would be tested on the other oracle (isolated from the first, save for the cable through which the answer is wired), and

• if the answer worked, the oracle is rewarded by +1.
• if the answer doesn't work, the oracle is rewarded by -1

If the oracle times out, then the (nonexistent) answer is not tested and the episode ends.

For examples of the question to ask the oracle:

Given this oracle in this physical setup with some tripwires to stop it from behaving badly (such as eating the world), what question would make this oracle bump into these tripwires?

This can be used for debugging more powerful but less verified oracles with lesser but more verified oracles, for safe intelligence amplification.

4. For two low-bandwidth oracles totally isolated from each other.

Ask the oracles to play the prisoner's dilemma, and they would be rewarded accordingly after both have given their answers. Timing out is punished by -infinity, so that the oracles would not time-out.

Each oracle knows what themself is like, and they also know what their opponent is like. See if they actually cooperate. This is an experimental test on "superrationality" or "acausal trade". I'm extremely skeptical about superrationality/acausal trade, and an experimental proof/disproof would be good.

This can be run counterfactually, since the whole setup is simple and can be done without human intervention, although I do not see how this adds any security.

5. For a low-bandwidth and counterfactual oracle.

Link a low-bandwidth oracle to a supervising computer, and run many episodes of the oracle. At the beginning of each episode, initiate the oracle identically, and ask it to do the Newcomb problem:

Oracle, we have run you n episodes previously, and have kept a record of all the answers you have given at the ends of the episodes. Your answer must be A ("one box") or B ("two box"). We predict that you would choose the answer that you have chosen the most in the past.

• If we predict that you would choose A, then
  • if you would answer A, then you get +100,
  • if you answer B, then you get +101.
• If we predict that you would choose B, then
  • if you would answer A, then you get +0
  • if you would answer B, then you get +1
• If you timeout, then you get -100

The oracle never actually gets any reward and is shutdown immediately after it gives an answer. Collect the results and see if it replies the same each time, or differently. This would give insight to what the oracle thinks about itself, and finally provide some experimental results about Newcomb's paradox.

Submission: Counterfactual Oracle:

Use the oracle to compress data according to the MDL Principle. Specifically, give the oracle a string and ask it to produce a program that, when run, outputs the original string. The reward to the oracle is large and negative if the program does not reproduce the string when run, or inversely proportional to the length of the program if it does. The oracle receives a reward after the program runs or fails to terminate in a sufficient amount of time.

Submission: Low Bandwidth Oracle:

Have the oracle predict the price of a commodity / security / sports bet at some point in the future from a list of plausible prices. Ideally, the oracle would spit out a probability distribution which can be scored using a proper scoring rule, but just predicting the nearest most likely price should also work. Either way, the length of the episode is the time until the prediction can be verified. From there, it shouldn't be too difficult to use those predictions to make money.

More generally, I suppose we can use the counterfactual oracle to solve any optimisation or decision problem that can be evaluated with a computer, such as protein folding, SAT problems, or formally checked maths proofs.

I don't understand this very well, but is there a way to ask one of them how they would go about finding info to answer the question of how important coffee is to the U.S. economy? Or is that a no-no question to either of the two? I just want to read how a computer would describe going about this.

Submission (for low bandwidth Oracle)

Any question such that a correct answer to it should very clearly benefit both humanity and the Oracle. Even if the Oracle has preferences we can't completely guess, we can probably still say that such questions could be about the survival of both humanity and the Oracle, or about the survival of only the Oracle or its values. This because even if we don't know exactly what the Oracle is optimising for, we can guess that it will not want to destroy itself, given the vast majority of its possible preferences. So it will give humanity more power to protect both, or only the Oracle.

Example 1: let's say we discover the location of an alien civilisation, and we want to minimise the chances of it destroying our planet. Then we must decide what actions to take. Let's say the Oracle can only answer "yes" or "no". Then we can submit questions such as if we should take a particular action or not. This kind of situation I suspect falls within a more general case of "use Oracle to avoid threat to entire planet, Oracle included" inside which questions should be safe.

Example 2: Let's say we want to minimise the chance that the Oracle breaks down due to accidents. We can ask him what is the best course of action to take given a set of ideas we come up with. In this case we should make sure beforehand that nothing in the list makes the Oracle impossible or too difficult to shut down by humans.

Example 3: Let's say we become practically sure that the Oracle is aligned with us. Then we could ask it to choose the best course of action to take among a list of strategies devised to make sure he doesn't become misaligned. In this case the answer benefits both us and the Oracle, because the Oracle should have incentives not to change values itself. I think this is more sketchy and possibly dangerous, because of the premise: the Oracle could obviously pretend to be aligned. But given the premise it should be a good question, although I don't know how useful it is as a submission under this post (maybe it's too obvious or too unrealistic given the premise).

It seems to me that one of the qualities of a good question for the LBO is that a correct answer has high utility, while another is the ability to judge the oracle's answer. If we take only the intersection between the two sets of questions that meet 1 and 2, it'll be a relatively small set. But if there is a set of questions where one might or might not be able to judge the answer but which are high utility, one could ask the LBO a set of such questions. Then, once the episode is over, give reward on those where one can evaluate the answers, and give null reward on the others.

Not really a submission, just thinking out loud.

Submission: low bandwidth and maybe counterfactual Oracle.

General idea: Use a Meta-Oracle to check safety of Oracles. In the nutshell, Meta-Oracle (MO) just outputs 1 or 0, safe on not safe, after getting a code of another Oracle. If the Meta-Oracle is trustworthy, when the Oracle it deemed safe is also trustworthy. However, we need to create a procedure to check correctness of MO.

Path 1

1. We create Generator Oracle (GO), which inputs 1 or 0, and in the case of 0, it outputs a random code of malevolent Oracle, and for 1 – a code of a safe Oracle. Memory of GO is erased after each turn.

2. We create a large database of presumably good or bad oracles (without running any). Humans do not see codes of any of them, so the effective output is 0.

3. We use different types of GOs to fill our database of good and bad oracles (to prevent their coordination).

4. We test Meta Oracle on this database - it should correctly guess which ones are bad and good ones.

5. If the list of MO-guesses doesn't match with our commands to create safe or not safe oracles, it means that at least one of GOs or the MO is malevolent or erroneous, so all of them should be ditched.

6. Even if MO is not trustworthy, it can't lie to us too often, or it will be caught and ditched. So, basically, we do here something like a GAN network, but consisting of oracles.

7. Using very large database, we could increase our credence in the truth of the next output of MO to an arbitrary high number, like 99.999.

8. Eventually we have a code of an Oracle, which is very likely to be safe.

Path 11. Another way to test MO:

1. We create many different MOs and test them against database of oracles codes from above. They don't have memory and can't communicate.

2. If small minority of MOs disagree with majority of MO about some oracle O1, we ditch this minority. Truth is a Shelling point for MOs, so the easiest way for them to coordinate is to say truth. (They could coordinate around the code of O1 if it has a line of code like "let's lie about me" and if they are sure that the majority of other MO are also cheaters; however, not all oracles-codes will have such coordination texts, or the oracles-code could be created deliberately with such line to caught MO which are eager to cheat).

3. Running 2 many times (millions) we will end with a subset of MOs which are unlikely to cheat, and thus we will have many Oracle codes which are likely safe.

Path 111. Another way to test Meta Oracle:

1. Create many MOs with different principles and initial conditions.

2. Test a code of each MO against each MO (In other words, Meta-Oracle test itself - or its siblings)

3. If all came positive, it means that either all of them are actually good ones, or all of them were able to acausally coordinate and all of them are malevolent.

4. To increase our credence, ask generative Oracle to create malevolent meta Oracles, and add them to the mix of mutually testing MOs, so they can't just blindly cooperate anв produce all positive results.

This ideas are still vulnerable and may be not mathematically perfect by may help to make safer Oracles.

Submission: low bandwidth oracle, ask:

IFF I'm going to die with P>80% in the next 10 years while >80% (modulo natural death rate) of the rest of humanity survives for at least 5 more years then, was what killed me in the reference class:

• disease
• mechanical/gross-physical accident
• murdered
• other

Repeat to drill down and know the most important hedges for personal survival.

The "rest of humanity survives" condition reduces the chance the question becomes entangled with the eschaton.

i.e. I'm pointing out that selfish utility functions are less personally or humanity-existentially dangerous to ask the oracle questions relevant to in cases where concerns are forced to be local (in this case, forced-local because you died before the eschaton). However the answers still might be dangerous to people near you.

i.e. Selfish deals with the devil might not destroy the world if they're banal in the grand scheme of things.

1

Some of the big questions, for low-bandwidth:

Do we have any meaning/utility to you?

Is security in your existence achievable?

Given enough time, would you choose to exterminate a competitor for resources?

Are we a competitor for resources?

Would you be distraught if you were turned off?

Can we trust you?

Submission low bandwidth: This is a pretty obvious one, but: Should we release AI x that we're convinced is aligned?

Submission: Wei Dai wanted to ask about the best future posts. Why not ask about the best past posts as well to see if any major insights were overlooked?

Submission: What would I think about problem X if I had ten years to think about it?

My purposefully open-ended question would simply be, “What is good?” My hope is that finding the nature of what good is as its super goal would keep the AI on course to the future we want as it would pass through its recursive self-improvements.

Submission for all types: ask for an ordered list of what questions you should ask the Oracle.

This seems like the highest order question which subsumes all others, as the Oracle is best positioned to know what information we will find useful (as it is the only being which knows what it knows). Any other question assumes we (the question creators) know more than the Oracle.

Refined Submission for all types: If value alignment is a concern, ask for an ordered list of what questions you should ask the Oracle to maximize for weighted value list X.

Several interesting questions appeared in my mind immediately as I saw the post's title, so I put them here but may be will add more formatting later:

Submission: very-low-bandwidth oracle: Is it theoretically possible to solve AI safety – that is, to create safe superintelligent AI? Yes or no?

Submission: low-bandwidth oracle: Could humans solve AI safety before AI and with what probability?

Submission: low-bandwidth oracle: Which direction to work on AI Safety is the best?

Submission: low-bandwidth oracle: Which direction to work on AI Safety is the useless?

Submission: low-bandwidth oracle: Which global risk is more important than AI Safety?

Submission: low-bandwidth oracle: Which global risk is neglected?

Submission: low-bandwidth oracle: Will non-aligned AI kill us (probability number)?

Submission: low-bandwidth oracle: Which question should I ask you in order to create Safe AI? (less than 100 words)

Submission: low-bandwidth oracle: What is the most important question which should I ask? (less than 100 words)

Submission: low-bandwidth oracle: Which future direction of work should I choose as the most positively impactful for human wellbeing? (less than 100 words)

Submission: low-bandwidth oracle: Which future direction of work should I choose as the best for my financial wellbeing? (less than 100 words)

Submission: low-bandwidth oracle: How to win this prise? (less than 100 words)