Counterfactual Mugging

Imagine that one day you come home to see your neighbors milling about your house and the Publisher's Clearinghouse (PHC) van just pulling away. You know that PHC has been running a new schtick recently of selling $100 lottery tickets to win $10,000 instead of just giving money away. In fact, you've used that very contest as a teachable moment with your kids to explain how once the first ticket of the 100 printed was sold, scratched, and determined not to be the winner -- that the average expected value of the remaining tickets was greater than their cost and they were therefore increasingly worth buying. Now, it's weeks later, most of the tickets have been sold, scratched, and not winners and they came to your house. In fact, there were only two tickets remaining. And you weren't home. Fortunately, your neighbor and best friend Bob asked if he could buy the ticket for you. Sensing a great human interest story (and lots of publicity), PHC said yes. Unfortunately, Bob picked the wrong ticket. After all your neighbors disperse and Bob and you are alone, Bob says that he'd really appreciate it if he could get his hundred dollars back. Is he mugging you? Or, do you give it to him?

The counterfactual anti-mugging: One day No-mega appears. No-mega is completely trustworthy etc. No-mega describes the counterfactual mugging to you, and predicts what you would have done in that situation not having met No-mega, if Omega had asked you for $100.

If you would have given Omega the $100, No-mega gives you nothing. If you would not have given Omega $100, No-mega gives you $10000. No-mega doesn't ask you any questions or offer you any choices. Do you get the money? Would an ideal rationalist get the money?

Okay, next scenario: you have a magic box with a number p inscribed on it. When you open it, either No-mega comes out (probability p) and performs a counterfactual anti-mugging, or Omega comes out (probability 1-p), flips a fair coin and proceeds to either ask for $100, give you $10000, or give you nothing, as in the counterfactual mugging.

Before you open the box, you have a chance to precommit. What do you do?

Philosopher Kenny Easwaran reported in 2007 that:

Josh von Korff, a physics grad student here at Berkeley, and versions of Newcomb’s problem. He shared my general intuition that one should choose only one box in the standard version of Newcomb’s problem, but that one should smoke in the smoking lesion example. However, he took this intuition seriously enough that he was able to come up with a decision-theoretic protocol that actually seems to make these recommendations. It ends up making some other really strange predictions, but it seems interesting to consider, and also ends up resembling something Kantian!

The basic idea is that right now, I should plan all my future decisions in such a way that they maximize my expected utility right now, and stick to those decisions. In some sense, this policy obviously has the highest expectation overall, because of how it’s designed.

Korff also reinvents counterfactual mugging:

Here’s another situation that Josh described that started to make things seem a little more weird. In Ancient Greece, while wandering on the road, every day one either encounters a beggar or a god. If one encounters a beggar, then one can choose to either give the beggar a penny or not. But if one encounters a god, then the god will give one a gold coin iff, had there been a beggar instead, one would have given a penny. On encountering a beggar, it now seems intuitive that (speaking only out of self-interest), one shouldn’t give the penny. But (assuming that gods and beggars are randomly encountered with some middling probability distribution) the decision protocol outlined above recommends giving the penny anyway.

In a sense, what’s happening here is that I’m giving the penny in the actual world, so that my closest counterpart that runs into a god will receive a gold coin. It seems very odd to behave like this, but from the point of view before I know whether or not I’ll encounter a god, this seems to be the best overall plan. But as Josh points out, if this was the only way people got food, then people would see that the generous were doing well, and generosity would spread quickly.

And he looks into generalizing to the algorithmic version:

If we now imagine a multi-agent situation, we can get even stronger (and perhaps stranger) results. If two agents are playing in a prisoner’s dilemma, and they have common knowledge that they are both following this decision protocol, then it looks like they should both cooperate. In general, if this decision protocol is somehow constitutive of rationality, then rational agents should always act according to a maxim that they can intend (consistently with their goals) to be followed by all rational agents. To get either of these conclusions, one has to condition one’s expectations on the proposition that other agents following this procedure will arrive at the same choices.

Korff is now an Asst. Prof. at Georgie State.

My two bits: Omega's request is unreasonable.

Precommitting is something that you can only do before the coin is flipped. That's what the "pre" means. Omega's game rewards a precommitment, but Omega is asking for a commitment.

Precommitting is a rational thing to do because before the coin toss, the result is unknown and unknowable, even by Omega (I assume that's what "fair coin" means). This is a completely different course of action than committing after the coin toss is known! The utility computation for precommitment is not and should not be the same as the one for commitment.

In the example, you have access to information that pre-you doesn't (the outcome of the flip). If rationalists are supposed to update on new information, then it is irrational for you to behave like pre-you.

We're assuming Omega is trustworthy? I'd give it the $100, of course.

Hi,

My name is Omega. You may have heard of me.

Anyway, I have just tossed a fair coin, and given that the coin came up tails, I'm gonna have to ask each of you to give me $100. Whatever you do in this situation, nothing else will happen differently in reality as a result. Naturally you don't want to give up your $100. But see, if the coin came up heads instead of tails, I'd have given you each $10000, but only to those that would agree to give me $100 if the coin came up tails.

You know, if Omega is truly doing a full simulation of my cognitive algorithm, then it seems my interactions with him should be dominated by my desire for him to stop it, since he is effectively creating and murdering copies of me.

I guess I'm a bit tired of "God was unable to make the show today so the part of Omniscient being will be played by Omega" puzzles, even if in my mind Omega looks amusingly like the Flying Spaghetti Monster.

Particularly in this case where Omega is being explicitly dishonest - Omega is claiming to be either be sufficiently omniscient to predict my actions, or insufficiently omniscient to predict the result of a 'fair' coin, except that the 'fair' coin is explicitly predetermined to always give the same result . . . except . . .

What's the point of using rationalism to think things through logically if you keep placing yourself into illogical philosophical worlds to test the logic?

I convinced myself to one-box in Newcomb by simply treating it as if the contents of the boxes magically change when I made my decision. Simply draw the decision tree and maximize u-value.

I convinced myself to cooperate in the Prisoner's Dilemma by treating it as if whatever decision I made the other person would magically make too. Simply draw the decision tree and maximize u-value.

It seems that Omega is different because I actually have the information, where in the others I don't.

For example, In Newcomb, if we could see the contents of both boxes, then I should two-box, no? In the Prisoner's Dilemma, if my opponent decides before me and I observe the decision, then I should defect, no?

I suspect that this means that my thought process in Newcomb and the Prisoner's Dilemma is incorrect. That there is a better way to think about them that makes them more like Omega. Am I correct? Does this make sense?

I really fail to see why you're all so fascinated by Newcomb-like problems. When you break causality, all logic based on causality doesn't function any more. If you try to model it mathematically, you will get inconsistent model always.

I'm very torn on this problem. Every time I think I've got it figured out and start typing out my reasons why, I change my mind, and throw away my 6+ paragraph explanation and start over, arguing the opposite case, only to change my mind again.

I think the problem has to do with strong conflicts between my rational arguments and my intuition. This problem is a much more interesting koan for me than one hand clapping, or tree in the forest.

I think my answer would be "I would have agreed, had you asked me when the coin chances were .5 and .5. Now that they're 1 and 0, I have no reason to agree."

Seriously, why stick with an agreement you never made? Besides, if Omega can predict me this well he knows how the coin will come up and how I'll react. Why then, should I try to act otherwise. Somehow, I think I just don't get it.

• This problem seems conceptually identical to Kavka's toxin puzzle; we have merely replaced intending to drink the poison/pay $100 with being the sort of person whom Omega would predict would do it.

• Since, as has been pointed out, one needn't be a perfect predictor for the game to work, I think I'll actually try this on some of my friends.

Whether I give Omega the $100 depends entirely on whether there will be multiple iterations of coin-flipping. If there will be multiple iterations, giving Omega the $100 is indeed winning, just like buying a financial instrument that increases in value is winning.

I know that this is a very old post, but I thought that I should add a link to The Counterfactual Prisoner's Dilemma [LW · GW], which is a thought experiment Cousin_it and I independently came up with to demonstrate why you should care about this dilemma.

The setup is as follows:

Omega, a perfect predictor, flips a coin. If if comes up heads, Omega asks you for $100, then pays you $10,000 if it predict you would have paid if it had come up tails and you were told it was tails. If it comes up tails, Omega asks you for $100, then pays you $10,000 if it predicts you would have paid if it had come up heads and you were told it was heads

In this case if you always pay you receive $9900, while if you never pay you receive nothing. Crucially, you perform better regardless of whether the coin comes up heads or tails.

So, is it reasonable to pre-commit to giving the $100 in the counterfactual mugging game? (Pre-commitment is one solution to the Newcomb problem.) On first glance, it seems that a pre-commitment will work.

But now consider "counter-counterfactual mugging". In this game, Omega meets me and scans my brain. If it finds that I've pre-committed to handing over the $s in the counterfactual mugging game, then it empties my bank account. If I haven't pre-committed to doing anything in counterfactual mugging, then it rewards me with $1 million. Damn.

So what should I pre-commit to doing, if anything? Should I somehow try to assess my likelihood of meeting Omega (in some form or other) and guess what sort of parlour game it is likely to play with me, and for what stakes? Has anyone got any idea how to do that assessment, without unduly privileging the games that we happen to have thought of so far? This way madness lies I fear...

The interest with these Omega games is that we don't meet actual Omegas, but do meet each other, and the effects are sometimes rather similar. We do like the thought of friends who'll give us $1000 if we really need it (say in a once-in-a-lifetime emergency, with no likelihood of reciprocity) because they believe we'd do the same for them if they really needed it. We don't want to call that behaviour irrational. Isn't that the real point here?

If I found myself in this kind of scenario then it would imply that I was very wrong about how I reason about anthropics in an ensemble universe (as with Pascal's mugging or any sort of situation where an agent has enough computing power to take control of that much of my measure such that I find myself in a contrived philosophical experiment). In fact, I would be so surprised to find myself in such a situation that I would question the reasoning that led me to think one boxing was the best course of action in the first place, because somewhere along the way my model became very confused. (I'd still one box, but it would seem less obvious after taking into account the huge amount of previously unexpected structural uncertainty my model of the world suddenly has to deal with.)

Normally, you can assume your thought processes are uncorrelated with whats out there. Newcomb-like problems however, do have the state of the outside universe correlated with your actual thoughts, and this is what throws people off.

If you are unsure if the state of the universe is X or Y (say with p = 1/2 for simplicity), and we can chose either option A or B, we can calculate the expected utility of choosing A vs B by taking 1/2u(A,X)+1/2u(A,Y) and comparing it to 1/2u(B,X)+1/2u(B,Y).

In a newcomb-like problem, where the state of the experiment is actually dependent on your choice, the expected utility comparison should now be ~1u(A,X)+~0u(A,Y) vs ~0u(B,X)+~1u(B,Y).

In this case, it boils down to "Is u(A,X) > u(B,Y)?".

It is not enough for Omega to have a decent record of getting it right, since you could probably do pretty well by reading peoples comments and guessing based on that.

If Omega made its prediction solely based on a comment you made on LessWrong, you should expect that if you choose A the universe will be in the same state as if you choose b- knowing your ultimate decision doesn't tell you anything, since the only relevant evidence is what you said a month ago.

If, however, Omega actually simulates your thought process in sufficient detail to know for sure which choice you made, knowing that you ultimately decide to pick A is strong evidence that omega has set up X, and if you choose B, you better expect to see Y.

The reason that the answer changes is that the state of the box actually does depend on the thoughts themselves- it's just that you thought the same thoughts when omega was simulating you before filling the boxes/flipping the coin.

If you aren't sure whether you're just Omega's simulation, you better one box/pay omega. If we're talking about a wannabe Omega that just makes decent predictions based off comments, then you defect (though if you actually expect a situation like this to come up, you argue that you won't)

There is a caveat: if you are an agent who is constructed to live in the world where Omega tossed its coin to come out tails, so that the state space for which your utility function and prior are defined doesn't contain the areas corresponding to the coin coming up heads, you don't need to give up $100. You only give up $100 as a tribute to the part of your morality specified on the counterfactual area of the state space.

I would one-box on Newcombe, and I believe I would give the $100 here as well (assuming I believed Omega).

With Newcombe, if I want to win, my optimal strategy is to mimic as closely as possible the type of person Omega would predict would take one box. However, I have no way of knowing what would fool Omega: indeed if it is a sufficiently good predictor there may be no such way. Clearly then the way to be "as close as possible" to a one-boxer is to be a one-boxer. A person seeking to optimise their returns will be a person who wants their response to such stimulus to be "take one box". I do want to win, so I do want my response to be that, so it is: I'm capable of locking my decisions (making promises) in ways that forgo short-term gain for longer term benefit.

The situation here is the same, even though I have already lost. It is beneficial for me to be that type of person in general (obscured by the fact that the situation is so unlikely to occur). Were I not the type of person who made the decision to pay out on loss, I would be the type of person that lost $10000 in an equally unlikely circumstance. Locking that response in now as a general response to such occurrances means I'm more likely to benefit than those who don't.

I'm way late to this party, but aren't we ignoring something obvious? Such as imperfect knowledge of how likely Omega is to be right about its prediction of what you would do? If you live in a universe where Omega is a known fact and nobody thinks themselves insane when they meet him, well, then it's the degenerate case where you are 100% certain that Omega predicts correctly. If you lived in such a universe presumably you would know it, and everyone in that world would pre-commit to giving Omega $100, just like in ours pizza-deliverers pre-commit to not carrying more than a small amount of cash with them.

There may be other universes where Omega is known to be right and do what he says he will do 80% of the time. Or ones where there are rumors of an omniscient Omega that always makes good on his word, but you assign them 80% probability of being true. And so on.

Given the $5000 expected payoff and the $50 expected cost for pre committing, you should do it if the probability of Omega being both right and trustworthy is greater than or equal to 0.01.

But, if you, knowing what you know about THIS universe, suddenly found yourself in the presence of some alien entity making the claim Omega makes in the above scenario, what kind of evidence would you demand for this claim before assigning a probability greater than 0.01?

It occurs to me that the dude in the robe and mask pretending to be Omega could up the ante to $1000000, and if I wouldn't believe him more than 0.01% given a $10000 payoff, it probably wouldn't matter to me what he offered as a payoff, because if he has enough delusions and/or chutzpah to make this claim in this universe, there's no reason for him to balk at adding on a few extra decimal places. I'm not sure how to formalize that mathematically, though.

Under my syntacticist cosmology, which is a kind of Tegmarkian/Almondian crossover (with measure flowing along the seemingly 'backward' causal relations), the answer becomes trivially "yes, give Omega the $100" because counterfactual-me exists. In fact, since this-Omega simulates counterfactual-me and counterfactual-Omega simulates this-me, the (backwards) flow of measure ensures that the subjective probabilities of finding myself in real-me and counterfactual-me must be fairly close together; consequently this remains my decision even in the Almondian variety. The purer and more elegant version of syntacticism doesn't place a measure on the Tegmark-space at all, but that makes it difficult to explain the regularity of our universe - without a probability distribution on Tegmark-space, you can't even mathematically approach anthropics. However, in that version counterfactual-me 'exists to the same extent that I do', and so again the answer is trivially "give Omega the $100".

Counterfactual problems can be solved in general by taking one's utilitarian summation over all of syntax-space rather than merely one's own Universe/hubble bubble/Everett branch. The outstanding problem is whether syntax-space should have a measure and if so what its nature is (and whether this measure can be computed).

This is just the one-shot Prisoner's Dilemma. You being split into two different possible worlds, is just like the two prisoners being taken into two different cells.

Therefore, you should give Omega $100 if and only if you would cooperate in the one-shot PD.

I don't see the difficulty. No, you don't win by giving Omega $100. Yes, it would have been a winning bet before the flip if, as you specify, the coin is fair. Your PS, in which you say to "assume that in the overwhelming measure of the MWI worlds it gives the same outcome", contradicts the assertion that the coin is fair, and so you have asked us for an answer to an incoherent question.

is the decision to give up $100 when you have no real benefit from it, only counterfactual benefit, an example of winning?

No, it's a clear loss.

The only winning scenario is, "the coin comes down heads and you have an effective commitment to have paid if it came down tails."

By making a binding precommitment, you effectively gamble that the coin will come down heads. If it comes down tails instead, clearly you have lost the gamble. Giving the $100 when you didn't even make the precommitment would just be pointlessly giving away money.

I realise I'm coming to this a little late, but I'm a little unclear about this case. This is my understanding:

When you ask me if I should give Omega the $100, I commit to "yes" because I am the agent who might meet Omega one day, and since I am in fact at the time before the coin has been flipped right now, by the usual expected value calculations the rational choice is to decide to.

So does that mean that if I commit now (eg: by giving myself a monetary incentive to give the $100), and my friend John meets Omega tomorrow who has flipped the coin and it has landed tails, I should tell him that the rational choice is to not give the $100, since he is deciding after the coin toss.

Would anyone be so kind as to tell me if that seems right?

Well, this comes up different ways under different interpretations. If there is a chance that I am being simulated, that is this is part of his determining my choice, then I give him $100. If the coin is quantum, that is there will exist other mes getting the money, I give him $100. If there is a chance that I will encounter similar situations again, I give him $100. If I were informed of the deal beforehand, I give him $100. Given that I am not simulated, given that the coin is deterministic, and given that I will never again encounter Omega, I don't think I give him $100. Seeing as I can treat this entirely in isolation due to these conditions, I have the choice between -$100 and $0, of which two options the second is better. Now, this runs into some problems. If I were informed of it beforehand, I should have precommitted. Seeing as my choices given all information shouldn't change, this presents difficulty. However, due to the uniqueness of this deal, there really does seem to be no benefit to any mes from giving him the money, and so it is purely a loss.

No precommittment, no deal.

Suppose Omega gives you the same choice, but says that if a head had come up, it would have killed you, but only if you {would have refused|will refuse} to give it your lousy $100 {if the coin had come up heads|given that the coin has come up heads}. Not sure what the correct tense is, here.

I believe that I would keep the $100 in your problem, but give it up in mine.

ETA: Can you clarify your postscript? Presumably you don't want the knowledge about the distribution of coin-flip states across future Everett branches to be available for the purposes of the expected utility calculation?

Ok so there's a good chance I'm just being an idiot here, but I feel like a multiple worlds kind of interpretation serves well here. If, as you say, "the coin is deterministic, [and] in the overwhelming measure of the MWI worlds it gives the same outcome," then I don't believe the coin is fair. And if the coin isn't fair, then of course I'm not giving Omega any money. If, on the other hand, the coin is fair, and so I have reason to believe that in roughly half of the worlds the coin landed on the other side and Omega posed the opposite question, then by giving Omega the $100 I'm giving the me in those other worlds $1000 and I'm perfectly happy to do that.

Not sure how to delete, but this was meant to be a reply.

I think that what really does my head in about this problem is, although I may right now be motivated to make a commitment, because of the hope of winning the 10K, nonetheless my commitment cannot rely on that motivation, because when it comes to the crunch, that possibility has evaporated and the associated motivation is gone. I can only make an effective commitment if I have something more persistent - like the suggested $1000 contract with a third party. Without that, I cannot trust my future self to follow through, because the reasons that I would currently like it to follow through will no longer apply.

MBlume stated that if you want to be known as the sort of person who'll do X given Y, then when Y turns up, you'd better do X. That's a good principle - but it too can't apply, unless at the point of being presented with the request for $100, you still care about being known as that sort of person - in other words, you expect a later repetition of the scenario in some form or another. This applies as well to Eliezer's reasoning about how to design a self-modifying decision agent - which will have to make many future decisions of the same kind.

Just wanting the 10K isn't enough to make an effective precommitment. You need some motivation that will persist in the face of no longer having the possibility of the 10K.

Does this particular thought experiment really have any practical application?

I can think of plenty of similar scenarios that are genuinely useful and worth considering, but all of them can be expressed with much simpler and more intuitive scenarios - eg when the offer will/might be repeated, or when you get to choose in advance whether to flip the coin and win 10000/lose 100. But with the scenario as stated - what real phenomenon is there that would reward you for being willing to counterfactually take an otherwise-detrimental action for no reason other than qualifying for the counterfactual reward? Even if we decide the best course of action in this contrived scenario - therefore what?

The only mechanisms I know of by which Omega can accurately predict me without introducing paradoxes is by running something like a simulation, as others have suggested. But I really, truly, only care about the universe I happen to know about, and for the life of me, I can't figure out why I should care about any other. So even if the universe I perceive really is just simulated so that Omega can figure out what I would do in this situation, I don't understand why I should care about "my" utility in some other universe. So, two box, keep my $100.

Edit: I should add that my not caring about other universes is conditional on my having no reason to believe they exist.

I have one minor question about this problem, would I be allowed to say, offer omega $50 instead of the $100 he asked for in exchange for $5000 and the promise that, if it had occured that the coin landed head, it would give me $5000 and ask me for $50, which he (going to refer to all sentinents as he, that way I don't have to waste time typing figuring out whether the person I'm talking about is he,she, or it.) would know to do since Omega would simulate the me when the tail landed tails, and thus the simulated me would offer him this proposition. Which should not be too difficult to accept, given that the cost to Omega is basically zero across all possibilities, unless part of the point of the exercise was to mess with me.

In the event that he rejects this offer, I'm going to give him $100, and then mug him. (Assuming of course that the probability is not 0 that I succeed in the mugging, if he were to say kill me in response to the mugging, I'd simply have the copy that succeeded in mugging him force him to use some of his powers to resurrect my less fortunate copies. Assuming of course that that power is part of his omnipotence, else I wouldn't mug him, no point in betting my life in something that does not generate more of my life (given that if I succeeded I can have him make all copies of me immortal. Of course, if he had this power, there's a might be a chance that his retaliation might have been to simply wipe out all copies of me in all existances, in that case, the probability of success should be computed as a negative value, given I CAN fail more times then I try.) In the case I succeed in the mugging, I'd get at least my $100 back, and the cases I fail, I doubt I'd care about $100.

In the case that neither of the above are possible, I would not give him the $100, given that the diminishing returns of increasing amounts of money might well make the $10000 less utility than 2x instances of $100. (The 2x instances of $100 scale linearly, where each increasing $100 in the $10000 diminishes in value. As in, each instance of $100 would be worth just as much as a prior instance of $100 since it's being distributed among different copies of me, so diminishing returns does not kick in, whereas the $10000 all goes to one instance. It should be obvious that I prefer $50 to 50% chance of $100)

Of course, due to the above, there's a fourth possiblity, one where the iteration of me being offered the choice is being very much affected by the diminishing returns on the value of money. In that case, I would give $100 to omega, since this action would partially smooth out the differing amounts of wealth among multiple copies of me across worlds. Or rather, diminish the number of me who are "poorer," since the copies that are in need of money do not give up $100, but will recieve some regardless, unless that doesn't work out because Omega simulates the exact version, including current finiancial assets, which rather nullifies his capabilities as an interdimentional arbitrageur among copies of me. But at that point, the diminishing returns on money should be such that each additional $100 should be roughly equal in value, since diminishing returns ALSO suffer from diminishing returns, with increasing amounts of diminishing returns diminishing less returns.

In short, the options are, I offer him $50 for a constant $5000 across all outcomes of the coin flip, and he accepts, I give him $100 then I mug him, I do not give him $100 if diminishing returns is not yet itself affected by much by diminishing returns, or I give him $100 if it is.

I am unable to see how this boils down to anything but a moral problem (and therefore with no objective solution).

Compare this to a simple lost bet. Omega tells you about the deal, you agree, and then he flips a coin, which comes out tails. Why exactly would you pay the $100 in this example?

Because someone will punish/ostracise me if I renege (or other external consequences)? Then in the CM case all that matters is what the consequences are for your payment/refusal.

Because I have an absolute/irrational/moral desire to hold to my word? Then the only question is whether your definition of "my word" (or, more generally, your self-imposed moral obligations) includes counterfactual promises. But this is only a matter of choosing the boundaries of your arbitrary moral guidelines. It is hardly more solvable or more interesting than asking if you would consider yourself morally beholden to a promise you made when you were four years old.

Uniqueness raises all sorts of problems for decision theory, because expected utility implicitly assumes many trials. This may just be another example of that general phenomenon.

The Omega is also known to be absolutely honest and trustworthy, no word-twisting, so the facts are really as it says, it really tossed a coin and really would've given you $10000.

How do I know that? I would assign a lower prior probability to that than to me waking up tomorrow with a blue tentacle instead of my right arm; so, it such a situation, I would just believe Omega is bullshitting me.

Precommitting should be, as someone already said, signing a paper with a third party agreeing to give them $1000 in case you fail to give the $100 to Omega. Precommitment means you have no other option. You can't say that you both precommitted to give the $100 AND refused to do it when presented with the case.

Which means, if Omega presents you with the scenario before the coin toss, you precommit (by signing the contract with the third party). If Omega presents you with the scenario after the coin toss AND also tells you it has already come up tails - you haven't precommited, therefore you shouldn't give it $100.

EDIT: Also, some people objected to not giving the $100, because they might be the emulation which Omega uses to predict whether you'd really give money. If you were an emulation, then you would remember precommitting in expectation to get $10,000 with a 50% chance. It makes no sense for Omega to emulate you in a scenario where you don't get a chance to precommit.

How do you verify that "Omega" really is Omega and not a drunk in a bar? I can't think of a way of doing it - so it sounds like a fraud to me.

Why is the Omega asking me, when it already knows my answer? So what happens to Omega/the universe when I say no?

If he asks me the question, I have already answered the question, so I don't need to post this comment. I acted as I did. But i didn't act as I did (Omega hasn't shown up in my part of the universe), so we all know my answer.

Wow, this reddit softtware is pretty neat for a blog.

I'd love to see a post on the best introductory books to logic, and also epistemology. Epistemology, especially, seems to lack good introductory texts.

This is actually a parable on the boundaries of self (think a bit Buddhist here). Let me pose this another way: late last night in the pub, my past self committed to the drunken bet of $100 vs. $200 on the flip of a coin (the other guy was even more drunk than I was). My past self lost, but didn't have the money. This morning, my present self gets a phone call from the person it lost to. Does it honor the bet? Assuming, as in typical in these hypothetical problems, that we can ignore the consequences (else we'd have to assign a cost to them that might well offset the gains, so we'll just assign 0 and don't consider them), a utilitarian approach is that I should default on the bet if I can get away with it. Why should I be responsible for what I said yesterday?

However, as usual in utilitarian dilemmas, the effect that we get in real-life is that we have a conscience - can I live with myself being the kind of person that doesn't honor past commitments? So, most people will, out of one consideration or another, not think twice about paying up the $100.

Of Omega it is said that I can trust it more than I would myself. It knows more about me than I do myself. It would be part of myself if I didn't consider it seperate from myself. If I consider my ego and Omega part of the same all-encompassing self, then honoring the commitment that Omega committed itself to on my behalf should draw the same response as if I had done it myself. Only if I perceive Omega as a separate entity to whom I am not morally obligated can I justify not paying the $100. Only with this individualist viewpoint will I see someone whom I am not obligated to in any way demanding $100 of me.

If you manage to instill your AI with a sense of the "common good", a sense of brotherhood of all intelligent creatures, then it will, given the premises of trust etc., cooperate in this brotherhood - in fact, that is what I believe would be one of the meanings of "friendly".