Posts
Comments
Actually come to think of it, an even better analogy than a switched up newcomb's problem is a switched up parfit's hitchhiker. The human vs. human version works, not perfectly by any means, but at least to some extent, because humans are imperfect liars. You can't simulate another human's brain in perfect detail, but sometimes you can be a step ahead of them.
If the hitchhiker is omega, you can't. This is a bad thing for both you and omega, but it's not something either of you can change. Omega could self-modify to become Omega+, who's just like omega except that he never lies, but he would have no way of proving to you that he had done so. Maybe omega will get lucky, and you'll convince yourself through some flawed and convoluted reasoning that he has an incentive to do this, but he actually doesn't, because there's no possible way it will impact your decision.
Consider this. Omega promises to give you $500 if you take him into town, you agree, when you get to town he calls you a chump and runs away. What is your reaction? Do you think to yourself "DOES NOT COMPUTE"?
Omega got everything he wanted, so presumably his actions were rational. Why did your model not predict this?
You're right about the payoff matrix, I guess newcomb's problem doesn't have a payoff matrix at all, since there's no payoff defined for the person filling the boxes.
What do you mean by "prove a fact about their decision theory"? Do you mean that you're proving "a rational AI would use decision theory X and therefore use strategy Y", or do you only mean "GIVEN that an AI uses decision theory X, they would use strategy Y"?
There seems to be a belief floating around this site that an AI could end up using any old kind of decision theory, depending on how it was programmed. Do you subscribe to this?
The "horrible strategy", newcomb's problem, and TDT games in general only make sense if player 2 (the player who acts second) can be simulated by player 1 in enough detail that player 1 can make a choice which is CONDITIONAL on player 2's choice. A choice which, to reiterate, they have not yet actually made, and which they have an incentive to make differently than they are expected to.
The difficulty of achieving this may best be illustrated with an example. Here player 1, a human, and player 2, omega, are playing newcomb's problem.
Player 1: "Omega is super-intelligent and super-rational. He uses updateless decision theory. Therefore, he will pick the winning strategy rather than the winning local move. Ergo, he will only take the opaque box. So, I should put money in both boxes. A = A."
Player 2: "Chump." [takes both boxes]
But of course, this wouldn't REALLY happen, because if omega reasoned locally like this, we'd somehow be able to predict that, right? and so we wouldn't put money in the box, right? And so, being rational, he wouldn't want to act like that, because then he'd get less money. So, he'd definitely one-box. Whew, glad we reasoned through that. Let's put the money in both boxes now.
Player 2: "Chump." [takes both boxes]
The problem is, he can keep doing this no matter how fancy our reasoning gets, because at the end of the day, WE CAN'T SIMULATE HIS THINKING. It's not enough to do some handwavy reasoning about decision theories and payoff matrixes and stuff, in order to do a UDT bargain, we have to actually be able to actually simulate his brain. To not just see his thinking on the horizon, as it were, but to be A STEP AHEAD. And this we cannot do.
I take it your implication is that you could play the game with a superintelligent entity somewhere far in spacetime. If this is your plan, how exactly are you going to get the results back? Not really a test if you don't get results.
No, it's not. You might be able to guess that a superintelligence would like negentropy and be ambivalant toward long walks on the beach, but this kind of "simulating" would never, ever, ever, ever allow you to beat it at paper scissor rock. Predicting which square of a payoff matrix it will pick, when it is to the interest of the AI to pick a different square than you think it will, is a problem of the latter type.
This is a general purpose argument against all reasoning relating to superintelligences, and aids your argument no more than mine.
The only way you can "bargain" with somebody from the past is if you can be PRESENT in the past in a simulation they're running. That's how newcomb's problem works, omega is simulating you at time 0, and via that simulation, you have two-way interaction.
In this scenario, YOU have to be simulating THE AI at time 0, in your human imagination. This is not possible. The fact that the payoff matrix is simple does not make the opponent's reasoning simple, and in order for "bargaining" to happen, their reasoning has to be not only simple, but, crucially, tied to your actions. The situation is, exactly as I described, newcomb's problem with you preparing the boxes and omega picking.
Oh really? You have an omega sitting around you can test game theory problems with? Omniscient super-intelligent being, maybe in your garage or something?
Seriously though, for the decision of the person who picks the box to influence the person who puts in the money, the person who puts in the money has to be able to simulate the thinking of the person who picks the box. That means you have to simulate the thinking of Omega. Given that omega is smart enough to simulate YOUR thinking in perfect detail, this is patently impossible.
The only reason for omega to two-box is if your decision is conditional on his decision, and much as he might wish it was, no amount of super-intelligence or super-rationality on his part is going to give you that magical insight into his mind. He knows whether you put the money in the box, and he knows that which box he picks has no influence on it.
This "horrible strategy" is basically built on a kind of farcical newcomb's problem where both boxes are transparent, you're the one filling them with money, and omega is the one picking. It doesn't work at all.
If people would stop cranking their hypothetical scenarios up to 11 in the interest of "seriousness", maybe they could stop hyperventilating long enough to notice these things...