Don't Double-Crux With Suicide Rock

Everything I'm saying is definitely symmetric across persons, even if, as an author, I prefer to phrase it in the second person. (A previous post [LW · GW] included a clarifying parenthetical to this effect at the end, but this one did not.)

That is, if someone who trusted your rationality noticed that you seemed visibly unmoved by their strongest arguments, they might think that the lack of agreement implies that they should update towards your position, but another possibility is that their trust has been misplaced! If they find themselves living a world of painted rocks where you are one of the rocks, then it may come to pass that protecting the sanctity of their map would require them to master the technique of lonely dissent.

You could argue that my author's artistic preference to phrase things in the second person is misleading, but I'm not sure what to do about that while still accomplishing everything else I'm trying to do with my writing: my reply to Wei Dai [LW(p) · GW(p)] and a Reddit user's commentary [LW(p) · GW(p)] on another previous post seem relevant.

Being able to parse philosophical arguments is evidence of being rational. When you make philosophical arguments, you should think of yourself as only conveying content to those who are rationally parsing things, and conveying only appearance/gloss/style to those who aren't rationally parsing things.

I frequently find myself in situations where:

1) I disagree with someone

2) My opinion is based on fairly large body of understanding accumulated over many years

3) I think I understand where the other person is going wrong

4) trying to reach convergence would, in practice, look like a pointless argument that would only piss everyone off.

If there are real consequences at stake, I'll speak up. Often I'll have to take it offline and write a few pages, because some positions too complex for most people to follow orally. But if the agreement isn't worth the argument, I probably won't.

I think this is an important consideration of bounded rational agents, and much more so for embedded agents, which is unfortunately often ignored. The result is that you should not expect to ever meet an agent where Aumann fully applies in all cases because neither of you has the computational resources necessary to always reach agreement.

Yep. "Mutually trusting" would be better than "honest".

The other problem with Aumann's agreement theorem is that it's often applied too broadly. It should say, "Honest rational agents should never agree to disagree on matters of fact." What to do about those facts is definitely up for disagreement, insofar as two honest, rational agents may value wildly different things.

A slightly different frame on this (I think less pessimistic) is something like "honesty hasn't been invented yet". Or, rather, explicit knowledge of how to implement honesty does not exist in a way that can be easily transferred. (Tacit knowledge [LW(p) · GW(p)] of such may exist but it's hard to validate and share)

(I'm referring, I think, to the same sort of honesty Zack is getting at here [LW · GW], although the aspects of it that are relevant to doublecrux that didn't come up in that previous blogpost)

I think, obviously, that there have been massive strides (across human history, and yes on LW in particular) in how to implement "Idealized Honesty" (for lack of a better term for now). So, the problem seems pretty tractable. But it does not feel like a thing within spitting distance.

If an agent is not honest, ey can decide to say only things that provide no evidence regarding the question in hand to the other agent. In this case convergence is not guaranteed. For example, Alice assigns probability 35% to "will it rain tomorrow" but, when asked, says the probability is 21% regardless of what the actual evidence is. Bob assigns probability 89% to "will it rain tomorrow" but, when asked, says the probability is 42% regardless of what the actual evidence is. Alice knows Bob always answers 42%. Bob knows Alice always answers 21%. If they talk to each other, their probabilities will not converge (they won't change at all).

Yes, it can luckily happen that the lies still contain enough information for them to converge, but I'm not sure why do you seem to think it is an important or natural situation?

The usual formalization of "Occam's prior" is the Solomonoff prior, which still depends on the choice of a Universal Turing Machine, so such agents can still disagree because of different priors.

The title of Aumann's paper is just a pithy slogan. What the slogan means as the title of his paper is the actual mathematical result that he proves. This is that if two agents have the same priors, but have made different observations, then if they share only their posteriors, and each properly updates on the other's posterior, and repeat, then they will approach agreement without ever having to share the observations themselves. In other papers there are theorems placing practical bounds on the number of iterations required.

In actual human interaction, there is a large number of ways in which disagreements among us may fall outside the scope of this theorem. Inaccuracy of observation. All the imperfections of rationality that may lead us to process observations incorrectly. Non-common priors. Inability to articulate numerical priors. Inability to articulate our observations in numerical terms. The effort required may exceed our need for a resolution. Lack of good faith. Lack of common knowledge of our good faith.

Notice that these are all imperfections. The mathematical ideal remains. How to act in accordance with the eternal truths of mathematical theorems when we lack the means to satisfy their hypotheses is the theme of a large part of the Sequences [LW · GW].

No, the alternative (and only outcome for honest rational agents) is to converge to one belief. Each takes the other's stated (and mutually known to be honest and rational) beliefs as evidence, on which they update their own.