The Pleasures of Rationality

Wei Dai discusses the actual theorem, and in the last section expresses a sentiment similar to mine. I disapprove of the first paragraph of "Aumann agreement" wiki page (but see also the separate Aumann's agreement theorem wiki page).

FWIW, I wrote up a brief explanation and proof of Aumann's agreement theorem.

The wiki entry does not look good to me.

Unless you think I'm so irredeemably irrational that my opinions anticorrelate with truth, then the very fact that I believe something is Bayesian evidence that that something is true

This sentence is problematic. Beliefs are probabilistic, and the import of some rationalist's estimate varies according to one's own knowledge. If I am fairly certain that a rationalist has been getting flawed evidence (that is selected to support a proposition) and thinks the evidence is probably fine, that rationalist's weak belief that that proposition is true is, for me, evidence against the proposition.

Consider: if I'm an honest seeker of truth, and you're an honest seeker of truth, and we believe each other to be honest, then we can update on each other's opinions and quickly reach agreement.

Iterative updating is a method rationalists can use when they can't share information (as humans often can't do well), but that is a process the result of which is agreement, but not Aumann agreement.

Aumann agreement is a result of two rationalists sharing all information and ideally updating. It's a thing to know so that one can assess a situation after two reasoners have reached their conclusions based on identical information, because if those conclusions are not identical, then one or both are not perfect rationalists. But one doesn't get much benefit from knowing the theorem, and wouldn't even if people actually could share all their information; if one updates properly on evidence, one doesn't need to know about Aumann agreement to reach proper conclusions because it has nothing to do with the normal process of reasoning about most things, and likewise if one knew the theorem but not how to update, it would be of little help.

As Vladmir_Nesov said:

The crucial point is that it's not a procedure, it's a property, an indicator and not a method.

It's especially unhelpful for humans as we can't share all our information.

As Wei_Dei said:

Having explained all of that, it seems to me that this theorem is less relevant to a practical rationalist than I thought before I really understood it. After looking at the math, it's apparent that "common knowledge" is a much stricter requirement than it sounds. The most obvious way to achieve it is for the two agents to simply tell each other I(w) and J(w), after which they share a new, common information partition. But in that case, agreement itself is obvious and there is no need to learn or understand Aumann's theorem.

So Wei_Dei's use is fine, as in his post he describe's its limited usefulness.

at no point in a conversation can Bayesians have common knowledge that they will disagree.

As I don't understand this at all, perhaps this sentence is fine and I badly misunderstand the concepts here.

taking into account [as] the Bayesian evidence the other's testimony

This seems like usually an unimportant (and/because unreliable/difficult to use) component, most of the work is done by convincing argument, which helps with inferential difficulties, rather than lack of information.

Wei_Dai used the term back in 2009.

I had enormous dirty fun with this while on holiday. I got talking to a very smart (ex-LLNL nuclear engineer) ninety-year-old who was also very right wing. He proposed to me that the Government should harvest the organs of homeless people in order to give them to combat veterans. What he wanted me to say, of course, was that this proposal was outrageous and wrong, that human life was of greater value than that, and so on, so he could then say that these people were a drain on society and the people who'd made such sacrifices were more important, and so on.

Instead I said "Wouldn't it be cheaper to buy the organs from people in the third world? If you wanted to capture homeless people and take their organs, you'd need some sort of legal procedure to decide who was eligible, and there'd be appeals and so on, and it would all cost about as much as sentencing someone to death in the USA does now, which I'd guess must be hundreds of thousands of dollars at least. There must be plenty of people in poorer countries who would sacrifice their lives for a fraction of that money to feed their families in perpituity. There would be no use of force, no mistaken killings, and their organs would be higher quality. I'm not aware of a problem with getting organs for veterans, but if there is, that seems like a more efficient way to solve it."

His response? He went and got his CV so I'd be impressed at what a smart fellow he was!

Only on LW one can get comments like that :-)

I was actually planning a 'However, ' in that comment but I will leave it as it is now. However, I do think that it is more common not to (openly) change one's opinion -- especially for people in power.

The rule-proving exception in the famous quote, attributed to Keyes:

When the facts change, I change my mind. What do you do, sir?

(and I forgive him the colloquial use of the word 'fact'...)

It's great you can change your mind when so compelled by the facts - and you are right, maybe people do underestimate how good it looks to (not too often :-) admit that you were wrong.

Admitting failures (or not) is discussed at length in the nice pop-psy Mistakes were made, but not by me.