Extremely Counterfactual Mugging or: the gist of Transparent Newcomb

Your objection would be valid if we had a formalized concept of "counterfactual if" distinct from "logical if", but we don't. When looking at the behavior of deterministic programs, I have no idea how to make counterfactual statements that aren't logical statements.

Noting that a world is impossible because the agents will not make the decisions that lead to that world does not mean that you can just make stuff up about that world since "anything is true about a world that doesn't exist".

If event S is empty, then for any Q you make up, it's true that [for all s in S, Q]. This statement also holds if S was defined to be empty if [Not Q], or if Q follows from S being non-empty.

Both these statements are true, so I'd say they are consistent :-)

In particular, the first one is true because "The player would pay if asked" is true.

"The player would pay if asked" is true because "The player will be asked" is false and implies anything.

"The player will be asked" is false by the extra axiom.

Note I'm using ordinary propositional logic here, not some sort of weird "counterfactual logic" that people have in mind and which isn't formalizable anyway. Hence the lack of distinction between "will" and "would".

o=ASK => a=PAY

a=PAY

...Huh? My version of Omega doesn't bother predicting the agent, so you gain nothing by crippling its prediction abilities :-)

ETA: maybe it makes sense to let Omega have a "trembling hand", so it doesn't always do what it resolved to do. In this case I don't know if the problem stays or goes away. Properly interpreting "counterfactual evidence" seems to be tricky.

Yes, you got it right. I love your use of the word "collapse" :-)

My argument seems to indicate that there's no easy way for UDT agents to solve such situations, because the problem statements really are incomplete. Do you see any way to fix that, e.g. in Parfit's Hitchhiker? Because this is quite disconcerting. Eliezer thought he'd solved that one.

I'm not completely sure what your comment means. The result hasn't "changed", it has appeared. Without the extra axiom there's not enough axioms to nail down a single result (and even with it I had to resort to lexicographic chance at one point). That's what incompleteness means here.

If you think that's wrong, try to prove the "correct" result, e.g. that any agent who precommits to not paying won't get the $1000, using only the original axioms and nothing else. Once you write out the proof, we will know for certain that one of us is wrong or the original axioms are inconsistent, which would be even better :-)

Disagree. In e.g. the case of hazing, the person who has hazed me is not a counterfactual me, and his decision is not sufficiently correlated with my own for this approach to apply.

This is very nicely put.

Does this depend on many worlds as talking about "branches" seems to suggest? Consider, e.g.

You could have won or lost this time, but you've learned that you've lost, and your decision to scrape out a little more utility in this case takes away more utility by increasing the chance of losing in future similar situations.

"You could have been in a winning or losing branch, but you've learned that you're in a losing branch, and your decision to scrape out a little more utility within that branch takes away more utility from (symmetric) versions of yourself in (potentially) winning branches."

In the Hitchhiker you're scraping in the winning branch though.

I'm sorry it came off that way, I just found it overly similar to the other various Newcomblike problems, and couldn't see how it was supposed to reveal anything new about optimal decision strategies; paying is the TDT answer and the UDT answer; it's the choice you'd wish you could have precommitted to if you could have precommitted, it's the decision-type that will cause you to actually get $1,000,000, etc. If I'm not mistaken, this problem doesn't address any decision situations not already covered by standard Counterfactual Mugging and Parfit's Hitchhiker.

(Admittedly, I should have just said all that in the first place.)

Well, fine, but then the correct strategy depends on Omega's success rate (and the payoffs). If the reward given to those willing to pay is r, and Omega's demand is d, and Omega's prediction success rate is s, the expected payoff for those who agree to pay is s r + (s - 1) d, which may be both positive or negative. (Refusers trivially get 0.)

To predict, Omega doesn't need to simulate. You can predict that water will boil when put on fire without simulating the movement of 10^23 molecules.

Omega even can't use simulation to arrive at his prediction in this scenario. If Omega demands money from simulated agents who then agree to pay, the simulation violates the formulation of the problem, according to which Omega should reward those agents.

If the problem is reformulated as "Omega demands payment only if the agent would counterfactually disagree to pay, OR in a simulation", then we have a completely different problem. For example, if the agent is sufficiently confident about his own decision algorithm, then after Omega's demand he could assign high probability to being in a simulation. The analysis would be more complicated there.

In short, I am only saying that

1. Omega is trustworthy.
2. Omega can predict the agents behaviour with certainty.
3. Omega tells that it demands money only from agents whom it predicted to reject the demand.
4. Omega demands the money.
5. The agent pays.

are together incompatible statements.

Doesn't that contradict the original assertion?

That is, at that point it sounds like it's no longer "Omega will ask me to pay him $100 if he predicts I wouldn't pay him if he asked," it's "Omega will ask me to pay him $100 if he predicts I wouldn't pay him if he asked OR if I'm a simulation."

Not that any of this is necessary. Willingness to pay Omega depends on having arbitrarily high confidence in his predictions; it's not clear that I could ever arrive at such a high level of confidence, but it doesn't matter.

We're just asking, if it's true, what decision on my part maximizes expected results for entities for which I wish to maximize expected results? Perhaps I could never actually be expected to make that decision because I could never be expected to have sufficient confidence that it's true. That changes nothing.

Also worth noting that if I pay him $100 in this scenario I ought to update my confidence level sharply downward. That is, if I've previously seen N predictions and Omega has been successful in each of them, I have now seen (N+1) predictions and he's been successful in N of them.

Of course, by then I've already paid; there's no longer a choice to make.

(Presumably I should believe, prior to agreeing, that my choosing to pay him will not actually result in my paying him, or something like that... I ought not expect to pay him, given that he's offered me $100, regardless of what choice I make. In which case I might as well choose to pay him. This is absurd, of course, but the whole situation is absurd.)

ETA - I am apparently confused on more fundamental levels than I had previously understood, not least of which is what is being presumed about Omega in these cases. Apparently I am not presumed to be as confident of Omega's predictions as I'd thought, which makes the rest of this comment fairly irrelevant. Oops.

P(o=AWARD) = 0.8*P(a=PAY)+0.2*P(a=REFUSE)

Could you explain the error, rather than just say that it is a common error? How can I agree to pay in a situation which happens only if I was predicted to disagree?

(I don't object to having precommited to agree to pay if Omega makes his request; that would indeed be a correct decision. But then, of course, Omega doesn't appear. The given formulation

Omega asks you to pay him $100. Do you pay?

implies that Omega really appears, which logically excludes the variant of paying. Maybe it is only a matter of formulation.)

I'll echo prase's request. It seems to me that given that he's made the offer and I am confident of his predictions, I ought not expect to pay him. This is true regardless of what decision I make: if I decide to pay him, I ought to expect to fail.

Perhaps I'm only carrying counterfeit bills, or perhaps a windstorm will come up and blow the money out of my hands, or perhaps by wallet has already been stolen, or perhaps I'm about to have a heart attack, or whatever.

Implausible as these things are, they are far more plausible than Omega being wrong. The last thing I should consider likely is that, having decided to pay, I actually will pay.

ETA - I am apparently confused on more fundamental levels than I had previously understood, not least of which is what is being presumed about Omega in these cases. Apparently I am not presumed to be as confident of Omega's predictions as I'd thought, which makes the rest of this comment fairly irrelevant. Oops.

It's "useless" in part because, as you note, it assumes Omega works by simulating the player. But mostly it's just that it subverts the whole point of the problem; Omega is supposed to have your complete trust in its infallibility. To say "maybe it's not real" goes directly against that. The situation in which Omega simulates itself is merely a way of restoring the original intent of infallibility.

This problem is tricky; since the decision-type "pay" is associated with higher rewards, you should pay, but if you are a person Omega asks to pay, you will not pay, as a simple matter of fact. So the wording of the question has to be careful - there is a distinction between counterfactual and reality - some of the people Omega counterfactually asks will pay, none of the people Omega really asks will successfully pay. Therefore what might be seen as mere grammatical structure has a huge impact on the answer - "If asked, would you pay?" vs. "Given that Omega has asked you, will you pay?"