Is Omega Impossible? Can we even ask?

My objection, more succinctly: too long an introduction, not enough argument in the main part. Rewriting it with a paragraph (or three) cut from the intro and an extra paragraph of arguments about the impossibility of Omega in Newcomb would have made it much better, in my opinion.

But glad the discussion was useful!

For the curious, you should be indifferent to one- or two-boxing when Omega predicts your response 50.05% of the time. If Omega is just perceptibly better than chance, one-boxing is still the way to go.

Now I wonder how good humans are at playing Omega.

This seems an overly simplistic view. You need to specify your source of knowledge about correlation of quality of predictions and decision theory prediction target uses.

And even then, you need to be sure that your using an exotic DT will not throw Omega too much off the trail (note that erring in your case will not ruin the nice track record).

I don't say it is impossible to specify, just that your description could be improved.

Interesting and valuable point, brings the issue back to decision theory and away from impossible physics.

As I have said in the past, I would one-box because I think Omega is a con-man. When magicians do this trick the trick is the box SEEMS to be sealed ahead of time, but in fact there is a mechanism for the magician to slip something inside it. In the case of finding a signed card in a sealed envelope, the envelope had a razor slit which the magician could surreptitiously push the card in from. Ultimately, Siegfried and Roy were doing the same trick with tigers in cages. If regular (but talented) humans like Siegfried and Roy could trick thousands of people a day, then Omega can get the million out of the box if I two box, or get it in there if I one box.

Yes, I would want to build an AI clever enough to figure out a probable scam and then clever enough to figure out whether it can profit from that scam by going along with it. No, I wouldn't want that AI to think it had proof that there was a being that could seemingly violate the causal arrow of time merely because it seemed to have done so a number of times on the same order as Siegfried and Roy had managed.

Ultimately, my fear is if you can believe in Omega at face value, you can believe in god, and an FAI that winds up believing something is god when it is actually just a conman is no friend of mine.

If I see Omega getting the answer right 75% of the time, I think "the clever conman makes himself look real by appearing to be constrained by real limits." Does this make me smarter or dumber than we want a powerful AI to be?

I was already downvoted 6 before whining about my own fault.

So?

If Omega need not be infallible than it is certainly gratuitously confusing to me to put such an omega in the argument.

No, it's necessary to prevent other people from ignoring the hypothetical and going 'I trick Omega! ha ha ha I are so clever!' This is as about as interesting as saying, in response to the trolley dilemma, 'I always carry a grenade with me, so instead of choosing between the 5 people and the fat man, I just toss the grenade down and destroy the track! ha ha ha I are so clever!'

I am a very smart potential fellow traveler here, it would seem a defect of the site that its collective behavior is to judge queries such as mine unreasonable and undesirable to be seen.

'I am important! You should treat me nicer!'

If omega has previously been cited to be quite fallible and still have a newcomb's problem, I have not noticed it and I would love to see a link.

Multiple people have already pointed it out here, which should tell you something about how widespread that simple observation is - why on earth do you need a link? (Also, if you are "very smart", it should have been easy to construct the obvious Google query.)

A further thing to note: If Eliezer models other people as either significantly overestimating or significantly understimating the probability he'll one-box against them, both possibilities increase the probability he'll actually two-box against them.

So it all depends on Eliezer's model of other people's model of Eliezer's model of their model. Insert The Princess Bride reference. :-)

Now we are getting somewhere good! Certainty rarely shows up in predictions, especially about the future. Your decision theory may be timeless, but don't confuse the map with the territory, the universe may not be timeless.

Unless you are assigning a numerical, non-zero, non-unity probability to Omega's accuracy, you do not know when to one-box and when to two-box with arbitrary amounts of money in the boxes. And unless your FAI is a chump, it is considering LOTS of details in estimating Omega's accuracy, no doubt including considerations of how much the FAI's own finiteness of knowledge and computation fails to constrain the possibility that Omega is tricking it.

A NASA engineer had been telling Feynman that the liquid rocket motor had a zero probability of exploding on takeoff. Feynman convinced him that this was not an engineering answer. The NASA engineer then smiled and told Feynman the probability of the liquid rocket motor exploding on take off was "epsilon." Feynman replied (and I paraphrase from memory) "Good! Now we are getting somewhere! Now all you have to tell me is what your estimate for the value of epsilon is, and how you arrived at that number."

Any calculation of your estimate of Omega's responsibility which does not include gigantic terms for the evaluation of the probability that Omega is tricking you in a way you haven't figure out yet is likely to fail. I base that on the prevalence and importance of con games in the best natural experiment on intelligence we have: humans.

Surely the same applies to Omega. By hypothesis, Eliezer knows that Omega is reliable, and since Eliezer does not believe in magic, he deduces that Omega's prediction is based on good evidence, even if Omega doesn't say anything about the evidence.

My only reason for being unsure that Eliezer would one-box against me is that there may be some reflexivity issue I haven't thought of, but I don't think this one works.

One issue is that I'm not going around making these offers to everyone, but the only role that that plays in the original problem is to establish Omega's reliability without Newcomb having to explain how Omega does it. But I don't think it matters where the confidence in Omega's reliability comes from, as long as it is there.

Unless he figures you're not an idiot and you already know that, in which case it's better for him to have a rule that says "always one-box on Newcomb-like problems whenever the payoff for doing so exceeds n times the payoff for failed two-boxing" where n is a number (probably between 1.1 and 100) that represents the payment differences. Obviously, if he's playing against something with no ability to predict his actions (e.g. a brick) he's going to two-box no matter what. But a human with theory of mind is definitely not a brick and can predict his action with far better than random accuracy.

Omega problems are not posed for the purpose of being prepared to deal with Omega should you, or an FAI, ever meet him. They are idealised test problems, thought experiments, for probing the strengths and weaknesses of formalised decision theories, especially regarding issues of self-reference and agents modelling themselves and each other. Some of these problems may turn out to be ill-posed, but you have to look at each such problem to decide whether it makes sense or not.

As far as I know, this is the first thread in which it has ever been pointed out that Omega doesn't need to be infallible or even close to infallible in order for the problem to work. [...] As far as I am concerned, a real and non-trivial improvement has been proposed to the statement of Newcomb's problem as a result of my so-called "discussion" posting.

I'll note here that the lesswrong wiki page on Newcomb's problem has a section which says the following:

Irrelevance of Omega's physical impossibility

Sometimes people dismiss Newcomb's problem because a being such as Omega is physically impossible. Actually, the possibility or impossibility of Omega is irrelevant. Consider a skilled human psychologist that can predict other humans' actions with, say, 65% accuracy. Now imagine they start running Newcomb trials with themselves as Omega."

Also this section wasn't recently added, it has been there since November 2010.

In short you're not the first person to introduce to us the idea of Omega being impossible.

A newcomb's problem set with a gratuitous infallible predictor is inferior to a newcomb's problem set with a currently-implementable but imperfect prediction algorithm. Wouldn't you agree?

No, in maths you want to pick the simplest possible thing that embodies the principle you want to study, needless complications are distracting. Throwing in a probabilistic element to something that works fine as a deterministic problem is needless.