The True Epistemic Prisoner's Dilemma

As Omega led the creationist to the other room, you would (I know I certainly would) make a convulsive effort to convince him of the truth of evolution.

I could do that, but it seems simpler to make a convulsive effort to convince him that Omega, who clearly is no good Christian, almost certainly believes in the truth of evolution.

(Of course this is not relevant, but seemed worth pointing out. Cleverness is usually a dangerous thing, but in this case it seems worth dusting off.)

It is this then which I call the true epistemic prisoner's dilemma. If I tell you a story about two doctors, even if I tell you to put yourself in the shoes of one, and not the other, it is easy for you to take yourself outside them, see the symmetry and say "the doctors should cooperate". I hope I have now broken some of that emotional symmetry.

As Omega lead the creationist to the other room, you would (I know I certainly would) make a convulsive effort to convince him of the truth of evolution.

It seems like it would be wiser to forgo the arguments for evolution and spend your time talking about cooperating.

But of course, you would fail. And the door would shut, and you would grit your teeth, and curse 2000 years of screamingly bad epistemic hygiene, and weep bitterly for the people who might die in a few hours because of your counterpart's ignorance. And then -- I hope -- you would cooperate.

By the way, while we are adding direct emotional weight to this example, the real villain here is Omega. In all honesty, the Young Earth Creationist cannot be blamed for sending untold numbers to their death because of a bad belief. The bad belief has nothing to do with the asteroid and any moral link between the two should be placed on Omega.

And then -- I hope -- you would cooperate.

Why do you hope I'd let a billion people die (from a proposed quantification in another comment)?

This is actually rather different from a classic PD, to the extent that Cooperate (cooperate) is not the collectively desirable outcome.

Payoffs: You(Creationist): Defect(D): 1 Billion live D(C): 3 Billion live C(D): 0 live C(C): 2 Billion live

Under the traditional PD, D(C) is best for you, but worst for him. Under this PD, D(C) is best for both of you. He wants you to defect and he wants to cooperate; he just doesn't know it. Valuing his utility does not save this like it does the traditional PD. Assuming he's vaguely rational, he will end up happier if you choose to defect, regardless of his choice. Furthermore, he thinks you will be happier if he defects, so he has absolutely no reason to cooperate.

If only by cooperating can you guarantee his cooperation, you should do so. However the PD generally assumes such prior commitments are not possible. And unlike the traditional PD, C(C) does not lead to the best possible collective outcome. Thus, you should try your hardest to convince him to cooperate, then you should defect. He'll thank you for it when another billion people don't die.

The medical situation is more confusing because I don't think it's realistic. I sincerely doubt you would have two vaguely rational doctors who would both put 99% confidence on a diagnosis knowing that another doctor was at least 99% confident that that diagnosis was incorrect. Thus, you should both amend your estimates substantially downwards, and thus should probably cooperate. If you take the hypothetical at face value, it seems like you both should defect, even though again D(C) would be the optimal solution from your perspective.

The real problem I'm having with some of these comments is that they assume my decision to defect or cooperate affects his decision, which does not seem to be a part of the hypothetical. Frankly I don't see how people can come to this conclusion in this context, given that it's a 1-shot game with a different collective payoff matrix than the traditional PD.

I think you've all seen enough PDs that I can leave the numbers as an exercise

Actually, since this is an unusual setup, I think it's worth spelling out:

To the atheist, Omega gives two choices, and forces him to choose between D and C:

D. Omega saves 1 billion people if the Earth is old.
C. Omega saves 2 billion people if the Earth is young.

To the creationist, Omega gives two choices, and forces him to choose between D and C:

D. Omega saves an extra 1 billion people if the Earth is young.
C. Omega saves an extra 2 billion people if the Earth is old.

And then -- I hope -- you would cooperate.

No, I certainly wouldn't. I would however lie to the creationist and suggest that we both cooperate. I'd then defect, which, regardless of what he does, is still the best move. If I choose C then my action saves no lives at all, since the Earth isn't young.

My position on one-shot PDs remains that cooperation is only worthwhile in odd situations where the players' actions are linked somehow, such that my cooperating makes it more likely that he will cooperate; e.g. if we're artificial agents running the same algorithm.

you would cooperate

As I understand it, to the extent that it makes sense to cooperate, the thing that cooperates is not you, but some sub-algorithm implemented in both you and your opponent. Is that right? If so, then maybe by phrasing it in this way we can avoid philosophers balking.

I will point out to the defectors that the scenario described is no more plausible than creationism (after all it involves a deity behaving even more capriciously than the creationist one). If we postulate that your fictional self is believing in the scenario, surely your fictional self should no longer be quite so certain of the falsehood of creationism?

Given the stakes, it seems to me the most rational thing to do here is to try to convince the other person that you should both cooperate, and then defect.

The difference between this dilemma and Newcomb is that Newcomb's Omega predicts perfectly which box you'll take, whereas the Creationist cannot predict whether you'll defect or not.

The only way you can lose is if you screw up so badly at trying to convincing him to cooperate (i.e. you're a terrible liar or bad at communicating in general and confuse him), that instead he's convinced he should defect now. So the biggest factor when deciding whether to cooperate or defect should be your ability to convince.

The Standard PD is set up so there are only two agents and only their choices and values matter. I tend to think of rationality in these dilemmas as being largely a matter of reputation, even when the situation is circumscribed and described as one-shot. Hofstadter's concept of super-rationality is part of how I think about this. If I have a reputation as someone who cooperates when that's the game-theoretically optimal thing to do, then it's more likely that whoever I've been partnered with will expect that from me, and cooperate if he understands why that strategy works.

Since it would buttress that reputation, I keep hoping that rationalists, generally, would come to embrace some interpretation of super-rationality, but I keep seeing self-professed rationalists whose choices seem short-sightedly instrumentalist to me.

But this seems to be a completely different situation. Rather than attempting to cooperate with someone who I should assume to be my partner, and who has my interests at heart, I'm asked to play a game with someone who doesn't reason the way I do, and who explicitly mistrusts my reasoning. In addition, the payoff isn't to me and the other player, the payoff is to a huge number of uninvolved other people. MBlume seems to want me to think of it in terms of something valuable in my preference ranking, but he's actually set it up so that it's not a prisoner's dilemma, it's a hostage situation in which I have a clearly superior choice, and an opportunity to try to convince someone whose reasoning is alien to my own.

I defect. I do my best to convince my friend that the stakes are too high to justify declaring his belief in god. So you can get me to defect, but only by setting up a situation in which my allies aren't sitting on the other side of the bargaining table.

The young Earth creationist is right, because the whole earth was created in a simulation by Omega that took about 5000 years to run.

You can't win with someone that much smarter than you. I don't see how this means anything but 'it's good to have infinite power, computational and otherwise.'

the atheist will choose between each of them receiving $5000 if the earth is less than 1 million years old or each receiving $10000 if the earth is more than 1 million years old

Isn't this backwards? The dilemma occurs if payoff(unbelieved statement) > payoff(believed statement).

And then -- I hope -- you would cooperate.

This is to value your own "rationality" over that which is to be protected: the billion lives at stake. (We may add: such a "rationality" fetish isn't really rational at all.) Why give us even more to weep about?

My thinking is, if you are stupid (or ignorant, or irrational, or whatever) enough to be a creationist, you are probably also stupid enough not to know the high-order strategy for the prisoner's dilemma, and therefore cooperating with you is useless. You'll make your decision about whether or not to cooperate based on whatever stupid criteria you have, but they probably won't involve an accurate prediction of my decision algorithm, because you are stupid. I can't influence you by cooperating, so I defect and save some lives.