Don't be Pathologically Mugged!

I'm pretty sure that any time you attach a rider like "if you would choose the option that to your knowledge is best then instead someone pokes you in the eye" rational agents will get poked in the eye. And this isn't a problem, 'cause that sort of rider is far weirder than all those other contrived decision problems.

A decision algorithm that would tend to win in this contrived situation would tend to lose in regular situations, right?

Yes. There is No Free Lunch. For every possible algorithm it is possible to create a problem in which the algorithm fares poorly. An algorithm optimized for any problem which gives a payoff in utility for being irrational will tend lose in regular situations. Also, decisions made based on pathological priors will tend to lose. This includes having inaccurate priors about the likely behaviour of the superintelligence that you are playing with.

Consider the Pathological Decision Challenge.

Omega shows up and presents a Decision Challenge, consisting of some assortment of your favorite decision theory puzzlers. (Newcomb, etc etc etc...)

Unbeknownst to you, however, Omega also has a secret additional test: If the decisions you make are all something OTHER than the normal rational ones, then Omega will pay you some huge superbonus of utilions, vastly dwarfing any cost to loosing all of the individual challenges...

However, Omega also models you and if you would have willingly "failed" HAD YOU KNOWN about the extra challenge above, (but not this extra extra criteria), then you get no bonus for failing everything.

That is not Omega. Omega as presented for the purpose of Newcomblike problems is known, for the sake of the hypothetical, to be trustworthy. He does not deceive us about our utility payoffs. And yes, that does include being technically truthful but leaving off a whole utility-payoff category. If the is not clear to the audience from the description given in the problem definition then the problem definition needs to be more pedantic.

Consider, for example, Vladmir's original definition of counterfactual mugging. He throws in "the Omega is also known to be absolutely honest and trustworthy, no word-twisting, so the facts are really as it says". It should be fairly clear to the reader that 'unbeknownst to you" is to be considered out of scope of the exercise.

If you want a demigod who plays games that happen to involve him making us have an inaccurate knowledge of his arbitrary utility payoffs then you need to invent a new name.

None of Parfit's Hitch-hiker, Prisoner's Dilemma, Newcomb's or Counterfactual Mugging rely on the kind of 'payoff for being irrational' difficulties you present. They are all instances where a decision algorithm that wins will also win in the regular situations that they caricaturize.

Omega really needs to stop killing copies of me.

It's just not right.

The reason this seems wrong is because you're leaving out a crucial step: generating a probability distribution over the set of worlds you could be in, predicated on what you've seen already. When you meet a supposed Omega, you should start off believing that he's a human trying to fool you, which means that you should not pay up in counterfactual mugging and should two-box in Newcomb's problem. It takes an extraordinary amount of evidence to dislodge that conclusion even a little. Only after you have that evidence can you accurately map out the remaining possibilities. Either Omega gives you the outcome based on your decision as he claims, he gives you a fixed outcome regardless of your decision, or he deviates the algorithm he claims to follow in an obvious way (he doesn't pay up when he claims he would), or he deviates from the algorithm he claims to follow in a non-obvious way (such as an easter-egg bonus or penalty given when your behavior matches a particular pattern). Once you have probabilities for each of those, you just plug the scenarios into your utility function, multiply by the probability, sum up the expected utility for each decision algorithm, and follow whichever gives the highest expected utility.

I've had similar concerns but haven't gotten around to writing anything about it here - mostly because I figured it had already been addressed in hundreds of comments that have been posted on these topics, and I didn't want to go through them all.

Here's my basic thoughts on the subject. When we are constructing our decision theory, we need to make sure we setting ourselves up to optimize on the correct domain. If Omega exists and is actually a trickster, specifically good at fooling you into believing he's not Omega, but is rather Epsilon, the younger, smaller, and more honest of the two brothers and with a taste for your money, then designing your decision theory to be susceptible to being counterfactually mugged is a bad idea. Though I suppose I'm splitting hairs, because how would be choose between decision theories? A meta-decision theory? I guess that brings us back to the simultaneously insightful and vague "rational agents win!"

Some situations you have to consider, but not necessarily bind to predetermined actions. If the right balance is to miss out on this one particular situation, that's the solution you assign to the situation, once you consider it. Rigor of the theory is in ability to mechanistically comprehend all of its domain: if there are dark corners where the answer is unclear, a new theory has to resolve that perplexity, one way or the other.