This proposal is a minor variation on the HCH type ideas. The main differences seem to be

1. Only a single training example needed through use of hypotheticals.
2. The human can propose something else on the next step.
3. The human can't query copies of their simulation, at least not at first. There are probably all sorts of ways the human can bootstrap, they are writing arbitrary maths expressions.

This leads to a selection of problems largely similar to the HCH problems. 

1.  We need good inner alignment. (And with this, we also need to understand hypotheticals).
2. High fidelity, we don't want a game of Chinese whispers. 
3. The process could never converge, it could get stuck in an endless loop. 
4. Especially if passing the message on to the next cindy is one button press, what happens when one of them stumbles on a really memmetic idea, would the question end up filled with "repost or a ghost will haunt you" nonsense that plagues some websites. You are applying strong memetic selection pressure, and might get really funny jokes instead of AI ideas. 

Having the whole alignment community for 6 months be part of the question answerer is more likely to work than one person for a few hours, but that amplifies other problems. 

This method also has the problem of amplified failure probability. Suppose somewhere down the line, millions of iterations in, cindy goes outside for a walk, and gets hit by a truck. Virtual cindy doesn't return to continue the next layer of the recursion. What then? (Possibly some code just adds "attempt 2" at the top and tries again.) 

Ok, so another million layers in, cindy drops a coffee cup on the keyboard, accidentally typing some rubbish. This gets interpreted by the AI as a mathematical command, and the AI goes on to maximize ???

Chaos theory. Someone else develops a paperclip maximizer many iterations in, and the paperclip maximizer realizes it's in a simulation, hacks into the answer channel and returns "make as many paperclips as possible" to the AI. 

And then there is the standard mindcrime concern. Where are all these virtual cindies going once we are done with them? We can probably just tell the AI in english that our utility function is likely to dislike deleting virtual humans. So all the virtual humans get saved on disk, and then can live in the utopia. Hey, we need loads of people to fill up the dyson sphere anyway.

I am not confident that your "make it complicated and personal data" approach at the root really stops all the aliens doing weird acausal stuff. The multiverse is big. Somewhere out there there is a cindy producing any bitstream that looks like this personal data, and somewhere out there are aliens faking the whole scenario for every possible stream of similar data. You probably need the internal counterfactual design to be resistant to acausal tampering.  

I think this is brilliant as a direction to think in, but I'm object-level skeptical. I could be missing important details.

Summary of what I think I understand

1. A superintelligent AI is built^[1] to optimise for $QACI(A,B,\text{“what should the utility function be?"})$.
2. That function effectively tells the AI to figure out: "If you extrapolate from the assumption that uniquely-identifiable-$B$ was actually $\text{“what should the utility function be?"}$ (ceteris paribus), what would uniquely-identifiable-$A$ have been?" And then take its own best guess about that as its objective function.
3. In practice, that may look something like this: The AI starts looking for referents of $A$ and $B$, and learns that the only knowable instances are in the past. Assuming it does counterfactual reasoning somewhat like humans, it will then try to reconstruct/simulate the situation in as much heuristically relevant detail as possible. Finally, it runs the experiment forwards from the earliest possible time [LW · GW] the counterfactual assumption can be inserted (i.e. when Cindy ran the program that produced $B$).
   1. (Depending on how it's built, the AI has already acquired a probability distribution over what its utility function could be, and this includes some expectancy over Cindy's values. Therefore, it plausibly tries to minimise side-effects.)
4. At some point in this process, it notices that the contents of $A$ are extremely sensitive to what went on in Cindy's brain at time T. So brainscanning her is obvious for accuracy and repeatability.
5. In the simulation, Cindy sees with joy that $B$ is $\text{“what should the utility function be?"}$ so she gets to work. Not willing to risk delaying more than 24-ish hours (for reasons), she finally writes into $A$: $QACI(A,B,\text{“Hi! Cindy-2 here. These are my notes: [...]. You can do it!"})$.
6. As long as that is the AI's best guess, it is now motivated to repeat the experiment with the new message. This allows successive Cindys to work^[2] on the problem until one of them declares success and writes a function plainly into $A$.

Implementation details

• The AI might be uncertain about how precise its simulations are, in which case it could want to run a series of these experiments with varying seeds before adopting whatever function the plurality converges to. The uncertainty compounds, however, so simulation-batches which output answers in fewer iterations (for whatever reason) will weigh more.
• I'm not sure $QACI$ will be interpreted as transitive between simulations by default. I think it depends on preferences regarding degrees of freedom in the logic used to interpret $QACI\left(QACI\right(\left)\right)$, if both the inner and outer function depend on mutually exclusive counterfactuals over the same state of reality (or variable). Each step is locally coherent, but you could get stuck in a repeating loop.
• We can't take for granted that arbitrary superintelligences will have human heuristics for what counts as "correct" counterfactual reasoning. It seems shaky to rely on it. (I notice you discuss this in the comments.)

Why I don't think it works

• It does not seem to do anything to inner alignment afaict, and it seems too demanding and leaky to solve outer alignment.
• I don't see how to feasibly translate QACI() into actual code that causes an AI to use it as a target for all its goal-seeking abilities.
• Even if it were made into a loss function, you can't train a transformer on it without relevant training data.
• If the plan is to first train a transformer-ish AI on normal data and only later swap its objective function (assuming it were made into one), then the network will already have encoded proxies for its old function, and its influence will (optimistically) see long-tailed exponential decay with training time.
• If instead the plan is to first train an instruction-executing language model with sufficient understanding of human-symbolic math or English, this seems risky for traditional reasons but might be the most feasible way to try to implement it. I think this direction is worth exploring.
• A mathematically precise (though I disagree the term is applicable here) objective function doesn't matter when you have a neural net trying its best to translate it into imprecise proxies which actually work in its environment.

1. ^{^}

   I recommend going all-in on building one. I suspect this is the bottleneck, and going full speed at your current course is often the best way to discover that you need to find a better course--or, indeed, win. Uncertainty does not imply slowing down [LW · GW].

2. ^{^}

   This ends up looking like a sort of serial processor constrained by a sharp information bottleneck between iterations.

I have a pretty strong heuristic that clever schemes like this one are pretty doomed. The proposal seems to lack security mindset, as Eliezer would put it. 

The most immediate/simple concrete objection I have is that no one has any idea how to create aligned-AI-part-2.exe? I don't think figuring out what we'd do if we knew how to make a program like that is really the difficult part here.

only one round of initial question-answer still seems very bad to me. it's very hard to get branch coverage of a brain. random data definitely won't do it.

I'm a little suspicious about this step

in order to get more compute and data, AI₁ very carefully hacks the internet, takes over the world, maybe prints nanobots and turns large uninhabited parts of the world into compute, and starts using its newfound access to real-world data and computing power to make better guesses as to what utility function E would eventually return.

What reasons do we have to believe AI_1 will be careful enough?

If we have techniques that are powerful enough to carry through this step correctly, chances are we have already solved the alignment problem. The Alignment problem is mostly about getting the AI to do anything at all without destroying the world, not about figuring out how to write down the one true perfect utility function.

As I said over on your Discord, this feels like it has a shard of hope, and the kind of thing that could plausibly work if we could hand AIs utility functions.

I'd be interested to see the explicit breakdown of the true names you need for this proposal.

It seems like if you can motivate an AI to do this very specific thing, you already solved the important bits somewhere else.

its formal goal [LW · GW]: to maximize whichever utility function (as a piece of math) would be returned by the (possibly computationally exponentially expensive) mathematical expression E which the world would've contained instead of the answer, if in the world, instances of question were replaced with just the string "what should the utility function be?" followed by spaces to pad to 1 gigabyte.

How do you think about the under-definedness of counterfactuals?

EG, if counterfactuals are weird, this proposal probably does something weird, as it has to condition on increasingly weird counterfactuals.

One assumption that stands out to me as a little questionable is the idea that Cindy will, with infinite simulated time to think, eventually manage to come up with a solution to the alignment problem. (This is compounded by the fact that she's regularly brain-wiped and can only preserve insights by cramming them into the 1 gigabyte of scratch paper afforded to her.)