Hofstadter's Superrationality

The first essay is by far the best introduction to TDT-like reasoning that I've ever read. In fact this paragraph sums up the whole informal part of the idea:

This solution depends in no way on telepathy or bizarre forms of causality. It’s just that the statement I’ll choose C and then everyone will, though entirely correct, is somewhat misleadingly phrased. It involves the word choice, which is incompatible with the compelling quality of logic. Schoolchildren do not choose what 507 divided by 13 is; they figure it out. Analogously, my letter really did not allow choice; it demanded reasoning. Thus, a better way to phrase the voodoo statement would be this: If reasoning guides me to say C, then, as I am no different from anyone else as far as rational thinking is concerned, it will guide everyone to say C.

Hofstadter's comparison of "choice" and "reasoning" is getting at the idea that people have decision routines rooted in physics, which can themselves be reasoned about, including reasoning that they are similar to one's own. I think this is really the core insight of the TDT idea.

And then the one-sentence:

Likewise, the argument "Whatever I do, so will everyone else do" is simply a statement of faith that reasoning is universal, at least among rational thinkers, not an endorsement of any mystical kind of causality.

Hofstadter presents the problems of cooperation in a context of mainstream risks — risks that public intellectuals and scientists of the time agreed were legitimate topics for discussion, and which members of the news-watching public would have heard of — such as nuclear war and environmental pollution. Yudkowsky presents these problems in a context which features exotic risks — risks that most folks have not heard of outside of science fiction; and particularly ones that deal with agents with nonhuman drives, such as Unfriendly AIs, paperclip maximizers, baby-eating aliens, and so on.

This seems like a matter of literary genre. The math of the Prisoner's Dilemma works the same regardless of whether you're worried about cooperating with the Kremlin or an alien. But it probably has some consequences for how people think of the subject. Someone exposed to superrationality/x-rationality ideas via Less Wrong might get the erroneous impression that they are somehow fundamentally linked to exotic risks.

On the other hand, bringing exotic agents into the discussion takes a bunch of cheap answers off the table — "Oh, humans are fundamentally cooperative; we all share the same values deep down; all we need to do is hug it out, trust each other, and do the right thing." The math works even when you don't share the same values, which is a pretty significant point.

If I'd been one of the participants on Hofstadter's original game, I'd have answered him thusly:

"I know where you're going with this experiment— you want all of us to realize that our reasoning is roughly symmetrical, and that it's better if we all cooperate than if we all defect. And if I were playing against a bunch of copies of myself, then I'd cooperate without hesitation.

However, if I were playing against a bunch of traditional game theorists, then the sensible thing is to defect, since I know that they're not going to reason by this line of thought, and so the symmetry is broken. Even if I were playing against a bunch of people who'd cooperate because they think that's more moral, I ought to defect (if I'm acting according to my own self-interest), because they're not thinking in these terms either.

So what I really need to do is to make my best guess about how many of the participants are thinking in this reflexive sort of way, and how many are basing their decisions on completely different lines of thought. And then my choice would in effect be choosing for that block of people and not for the rest, and so I'd need to make my best guess whether it's better for me if I (and the rest of that block) choose to cooperate or if we choose to defect. That depends on how large that block is, how many of the others I expect to cooperate vs. defect, and on the payoff matrix."

At the time he wrote it, the correct choice would have been to defect, because as Hofstadter noted, none of his friends (as brilliant as they were) took anything like that reflexive line of thought. If it were done now, among a group of Less Wrong veterans, I might be convinced to cooperate.

Huh. I finally understand the "logic" that has been espoused in HPMoR, ch 33 ""Precisely," said Harry Potter, his face now turning serious. "We are faced with a true Prisoner's Dilemma..."

What this reminds me of is the logistic equation: dx/dt = x(1-x).

This simple system has two equilibrium points: x = 0, and x = 1. x=0 is unstable - that is, any perturbation will cause the system to veer away from that equilibrium point. x=1 is stable, and any perturbations return to that equilbrium.

Hofstadter says that superrationalists should decide to pick the x=0 (unstable) equilibrium - i.e., cooperate. But any deviation from superrationality, however slight, will cause the equilibrium to collapse into the all-defect equilibrium.

Seems there’s a demon left a similar set-up in the church steeple down in Dwaynesville

This comment by Vladimir Nesov is relevant.

Those who would caution people that it might be counter-productive to work against the arms race - unless they believe one should work for the arms race - are in effect counseling paralysis. But would they do so in other areas of life? You never know if that car trip to the grocery store won’t be the last thing you do in your life. All life is a gamble.

This problem is very very very difficult.