Self-modification is the correct justification for updateless decision theory

I wish I could upvote this more than once. An agent not yet capable of 100% precommitments can't be expected to make them, but you can imagine it slowly self-modifying to install stronger and stronger internal pre-commitment mechanisms, since even probabilistic precommitments are better than nothing (like Joe at least tried to do in Newcomb's problem happened to me).

Then it would fare more optimally on average for scenarios that involve predicting its behavior after those installments. I think that's the most one can hope for if one does not start with precommitment mechanisms. You can't stop Omega (in Judea Pearl's timeless sense of causality, not classically) from simulating a bad decision you would have made one year ago, but you can stop him from simulating a bad decision you would make in the future.

I'll take it :)

(Sorry about the late reply. I'm not sure how I missed this post.)

Suppose you're right and we do want to build an AI that would not press the button in this scenario. How do we go about it?

1. We can't program "the umpteenth digit of pi is odd" into the AI as an axiom, because we don't know this scenario will occur yet.
2. We also can't just tell the AI "I am conscious and I have observed Alpha Centari as not purple", because presumably when Omega was predicting the AI's decision a million years ago, it was predicting the AI's output when given "I am conscious and I have observed Alpha Centari as not purple" as part of its input.
3. What we can do is, give the AI an utility function that does not terminally value beings who are living in a universe with a purple Alpha Centauri.

Do you agree with the above reasoning? If so, we can go on to talk about whether doing 3 is a good idea or not. Or do you have some other method in mind?

BTW, I find it helpful to write down such problems as world programs so I can see the whole structure at a glance. This is not essential to the discussion, but if you don't mind I'll reproduce it here for my own future reference.

def P():
    if IsEven(Pi(10^100)):
        if OmegaPredict(S, "Here's a button... Alpha Centauri does not look purple.") = "press":
            MakeAlphaCentauriPurple()
        else:
            DestroyEarth()
    else:
        LetUniverseRun(10^6 years)
        if S("Here's a button... Alpha Centauri does not look purple.") = "press":
            DestroyEarth()

    LetUniverseRun(forever)

Then, assuming our AI can't compute Pi(10^100), we have:

• U("press") = .5 * U(universe runs forever with Alpha Centauri purple) + 0.5 * U(universe runs for 10^6 years then Earth is destroyed)
• U("not press") = .5 * U(Earth is destroyed right away) + 0.5 * U(universe runs forever)

And clearly U("not press") > U("press") if U(universe runs forever with Alpha Centauri purple) = U(Earth is destroyed right away) = 0.

The solution is for the AI to compute that digit of pi and press the button iff it's even. :-)

Then you have an AI that satisfies a certain kind of philosopher, wins big in a certain logically impossible world, and destroys humanity.

And if we assume that it's better to have Earth exist one million years longer, this is the correct thing to do, no question about it, right? If you're going to take a bet which decides the fate of entire human civilization, you want to take the best bet, which in this case(we assume) was to risk to live only for million years instead risking of exploding right away.

Unless, of course, in the counterfactual you know you would've pressed the button even though you now don't. Rigging the lottery is a sneaky way out of the problem.

Let's take for granted that earth existing for million years before exploding is better than it exploding right away. Then, of course you should press the button. If you don't do that, that'd mean that every time this sort of thing happens, you fare, on average, worse than someone who doesn't update like you suggest. You should take the best bet, even acausally, especially if you're gambling with the fate of the entire world.

If there's some trick here why you should not press the button, I'm not seeing it.

"you" could be a UDT agent. So does this example show a divergence between UDT and XDT applied to UDT?

I'm wondering where this particular bit of insanity (from my perspective) is coming from. I assume that if Omega would have destroyed the solar system (changed from just the earth because trading off 1million years of human history vs the rest of the SS didn't seem to be the point of the thought experiment) a million years ago if AI would not press the button and made the button also destroy the solar system you'd want the AI to press the button. Why should a 50% chance to be lucky change anything?

Would you also want the AI not to press the button if the "lucky" digit stayed constant, i. e. if Omega left the solar system alone in either case if the digit was even, destroyed the solar system a million years ago if the digit was not even and the AI would not press the button and made the button destroy the solar system if the digit was not even and the AI would press the button? If not, why do you expect the choice of the AI to affect the digit of pi? Ignorance does not have magic powers. You can't make any inference on the digit of pi from your existence because you don't have any more information than the hypothetical you whose actions determine your existence. Or do you rely on the fact that the hypothetical you isn't "really" (because it doesn't exist) conscious? In that case you probably also think you can safely two-box if the boxes are transparent, you see money in both boxes and Omega told you it doesn't use any conscious simulations. (you can't, btw, because consciousness doesn't have magic powers either).

I agree. The way I think of it is this:

If you find yourself deciding contrary to UDT you may infer that either

1. your designer was stupider or crazier than you, or

2. your designer was smarter or less crazy than you.

In the first case, your designer failed to realize something that you would have realized, had you been in your designer's place. In the second case, your designer knew something that you would not have known in your designer's place.

For example, if you find yourself turning down Omega in Pascal's Mugging [ETA: I meant the Counterfactual Mugging], you may infer that either

1. your designer failed to design you in a way that you would have expected would maximize your utility, or

2. your designer somehow predicted, with sufficient confidence, how Omega's coin would land, and so designed you to do what is best in that case.

Of course, as a present-day human, you already knew that (1) is the case, not (2). You were designed by a stupid process of natural selection. You already knew that your designer didn't take into account the possibility of being confronted by something like Omega. It's true that other humans contributed some modest amount to your design, through education and the art of rationality. However, even these additions were made by people who were, for the most part, crazier or stupider than you are now.

So, finding that you are declining Omega, contrary to UDT, isn't even necessarily bad news. It probably just means that your designers lucked out more than they deserved, which would be good news.

But that doesn't change the fact that your designers were stupid, and that you ought to want to do better. You ought to modify yourself and any of your progeny to act more in accord with UDT in the future.

Just to be explicit: The first AI type, which is "time consistent", would probably press the button. The second AI type, which does worse in impossible worlds, and better in world we don't know are impossible, would not press the button.

do you think the AI should press the button if it was the first digit of pi that Omega calculated?

If a UDT agent is presented with a counterfactual mugging based on uncertainty of a logical proposition, it should attempt to resolve the logical uncertainty and act accordingly. But, in the Least Convenient Possible World, Omega says, "I chose some logical proposition that I assigned .5 probability of being true, then computed if it is in fact true, and if it is, then I ..."