Re-understanding Robin Hanson’s “Pre-Rationality”

Yes, you seem to understand the idea, at least as far as in what you've written above. Remember that pretty much all of the ways we express what it is to be rational are via constraints, e.g., probs sum to one, update over time via conditionalization, and so on. If you once satisfied the constraints but find you no longer do, well then the obvious plan would be to move your beliefs back to what they would be had you not made whatever errors led you to violate the constraints. In this case, if your pre-prior is consistent and reasonable, but doesn't satisfy the pre-rationality condition relative to your prior, the obvious plan is to update your prior to be whatever the pre-rationality condition says it should be (treating prior "P" as just a label).

Indeed, worked examples are your friend.

Slightly more plausibly, suppose that a set of AI Construction Kits are distributed by lottery, three base-ten digits per ticket. An ACK ends up in the hands of an OB/LW reader, Rational Riana, who constructs the AI to believe that the probability of any lottery ticket winning is 1/1000, and that this probability is independent of the retrospective event of Riana winning.

But Riana believes, and so the AI believes as well, and indeed it happens to be true, that if the lottery had come out differently, the ACK would have ended up in the hands of Superstitious Sally, who believes that lottery tickets in her hand are much more likely than average to win; and Sally's AI would have believed that the chance of Sally's next lottery ticket winning was 1/10. (Furthermore, Sally's AI might believe that Sally winning the previous lottery was additional evidence to this effect, but we can leave out that fillip for now.)

It seems to me that it is quite rational for Riana's AI to believe that the subjunctive Sally's AI it could have been - if, indeed, one's reference class is such as to treat this counterfactual entity as an alternative instance of "me" - is merely irrational.

Does this mean that Riana's AI isn't pre-rational? Or that Riana's AI isn't pre-rational with respect to the lottery ticket? Can Riana's AI and Sally's AI agree on the causal circumstances that led to their existence, while still disagreeing on the probability that Sally's AI's lottery ticket will win?

I similarly suspect that if I had been born into the Dark Ages, then "I" would have made many far less rational probability assignments; but I think this "alternative" me would have been simply mistaken due to being raised in an even crazier environment, rather than coherently updating a coherent pre-prior based on different data. Am I not pre-rational with respect to my birth date?

Wei, I understand the paper probably less well than you do, but I wanted to comment that p~, which you call r, is not what Robin calls a pre-prior. He uses the term pre-prior for what he calls q. p~ is simply a prior over an expanded state space created by taking into consideration all possible prior assignments. Now equation 2, the rationality condition, says that q must equal p~ (at least for some calculations), so maybe it all comes out to the same thing.

Equation 1 defines p~ in terms of the conventional prior p. Suppressing the index i since we have only one agent in this example, it says that p~(E|p) = p(E). The only relevant event E is A=heads, and p represents the prior assignment. So we have the two definitions for p~.

p~(A=heads | p=O) = O(A=heads)

p~(A=heads | p=P) = P(A=heads)

The first equals 0.6 and the second equals 0.4.

Then the rationality condition, equation 2, says

q(E | p) = p~(E | p)

and from this, your equations follow, with r substituted for q:

q (A=heads | p=O) = p~(A=heads | p=O) = O(A=heads)

q (A=heads | p=P) = p~(A=heads | p=P) = P(A=heads)

As you conclude, there is no way to satisfy these equations with the assumptions you have made on q, namely that the A event and the p-assigning events are independent, since the values of q in the two equations will be equal, but the RHS's are 0.6 and 0.4 respectively.

I think you're right that the descriptive (as opposed to prescriptive) result in this case demonstrates that the programmer was irrational. Indeed it doesn't make sense to program his AI that way, not if he wants it to "track truth".

Is it equivalent to state that leaning about the origins of your priors "screens off" the priors themselves?

I'm very busy today, but I'll look this over carefully tomorrow morning and reply then.

I find it puzzling that my post has been upvoted to 10, but nobody has explicitly confirmed that my understanding is correct. Should I take the votes as implicit confirmations, or are the upvoters just indicating that they're interested in posts like this?

For example, if there were such a thing as a gene for optimism versus pessimism, you might believe that you had an equal chance of inheriting your mother’s optimism gene or your father’s pessimism gene.

"I was always going to get the pessimism gene. That's just my luck."

I haven't read Robin's paper. But this setup, and the difficulties you point out, remind me of the "centered" credulity functions that Nick Bostrom discusses in his Sleeping Beauty (SB) paper. (ETA: Here's a link to that paper.)

I haven't thought much about this, so maybe the similarity is superficial. Or I might be misremembering Bostrom's paper in some crucial way. But, as I recall, he gives one credulity function P to SB-before-she-learns-it's-Monday, but he gives another credulity function P+ to SB-after-she-learns-it's-Monday.

The key is that, in general, P+(X) does not equal P(X | MONDAY). That is, you don't get P+ by just using P and conditioning on MONDAY. This is the crucial ingredient that Bostrom uses to evade the usual paradoxes that arise when you naively apply Bayesianism to SB's situation.

Hanson's equation

P(A=heads) = r(A=heads | p=P)

seems analogous to forcing

P+(X) = P(X | MONDAY),

so I wonder if there is some analogy between the difficulties that forcing either one engenders.

ETA: . . . and hence an analogy between their solutions.

I don't know if you understand pre-rationality correctly because I can't parse Robin's paper either, but your setup looks like another of those unstoppable force, unmovable object mindfucks. If an agent's beliefs about the source of their prior don't satisfy the pre-rationality condition together with the prior, you've got an agent with inconsistent beliefs about the world, plain and simple. It can be Dutch booked, etc. Which way to resolve the inconsistency is entirely up to programmer ingenuity: you can privilege the prior above the pre-prior, or the other way round, or use some other neat trick. Of course this doesn't apply to humans at all.