A Sketch of Good Communication

I like the approach but the problem this may run into on many occasions is - parts of the reasoning include things which are not easily communicable. Suppose you talk to AlphaGo and want it to explain why it prefers some move. It may relatively easily communicate part of tree search (an I can easily integrate the non-overlapping part of the tree into my tree), but we will run into trouble communicating the "policy network" consisting of weights obtained on training by very large number of examples. *

From human perspective, such parts of our cognitive infrastructure are often not really accessible, giving their results in the form of "intuition". What's worse, other part of our cognitive infrastructure is very good in making fake stories explaining the intuitions, so if you are motivated enough your mind may generate plausible but in fact fake models for your intuition. In worst case you discard the good intuition when someone shows you problems in the fake model.

Also... it seems to me it is in practice difficult to communicate such parts of your reasoning, and among rationalists in particular. I would expect communicating parts of the model in the form of "my intuition is ..." to be intuitively frowned upon, down-voted, pattern-matched to argument from authority, or "argument" by someone who cannot think clearly. (An exception are intuitions of high-status people). In theory it would help if you describe why your intuitive black box is producing something else than random noise, but my intuition tells me in it may sound even more awkward ("Hey I studied this for several years" )

*Btw in some cases, it _may_ make sense to throw away part of you model and replace it just by the opinion of the expert. I'm not sure I can describe it clearly without drawing

One of the common issues I see with disagreements in general is people jumping prematurely to the third diagram before spending time getting to the second one.

Jumping to the third diagram is especially bad if you are trying to coach or help someone with their problems. One heuristic that I've found very useful (when I have the patience to remember to follow it) is that if someone asks me for help, the very first thing to do before making any suggestions is to get an understanding of their model of what the problem is and why they haven't solved it yet. Otherwise, I'm just going to waste a lot of time and frustrate both of us by making suggestions which will feel obviously useless to them.

(And in the best case, asking them questions and trying to figure out the solution together leads to the other person figuring out a possible solution themselves, without me even needing to suggest anything.)

This is where you haven't changed your model, but decide to agree with the other person anyway.

I very often notice myself feeling psychological pressure to agree with whoever I'm currently talking to; it feels nicer, more "cooperative." But that's wrong—pretending to agree when you actually don't is really just lying about what you believe! Lying is bad!

In particular, if you make complicated plans with someone without clarifying the differences between your models, and then you go off and do your part of the plan using your private model (which you never shared) and take actions that your partner didn't expect and are harmed by, then they might feel pretty betrayed—as if you were an enemy who was only pretending to be their collaborator all along. Which is kind of what happened. You never got on the same page. As Sarah Constantin explains in another 2018-Review-nominated post [LW · GW], the process of getting on the same page is not a punishment!

This is the second time I've seen this. Now it seems obvious. I remember liking it the first time, but also remember it being obvious. That second part of the memory is probably false. I think it's likely that this explained the idea so well that I now think it's obvious.

In other words: very well done.

This doesn’t make any sense but people try it anyway, especially when they’re talking to high status people and/or experts. “Oh, okay, I’ll try hard to believe what the expert said, so I look like I know what I’m talking about.”

It seems to me, sometimes you just need to ignore your model's output and just go with an expert's opinion. For example, the current expected value of trying to integrate the expert's model might be low (even zero if it's just impossible for you to get it!).

Still, I agree that you shouldn't throw away your model. Looking at your diagrams, it's easy to think we could take a leaf from Pearl's Causality (Ch. 3). Pearl is talking about modelling intervention in causal graphs. That does seem kind of similar to modelling an 'expert override' in belief networks, but that's not necessary to steal the idea. We can either just snip the arrows from our model to the output probability and set the output to the expert's belief, or (perhaps more realistically), add an additional node 'what I view to be expert consensus' that modulates how our credence depends on our model.

Your diagrams even have white circles before the credence that could represent our 'seeming' -- i.e. what the probability we get just from the model is -- and then that can be modulated by a node representing expert consensus and how confidently we think we can diverge from it.

I appreciate that in the example it just so happens that the person assigning a lower probability ends up assigning a higher probability that the other person at the beginning, because it is not intuitive that this can happen but actually very reasonable. Good post!

This is one of the most important reasons why hubris is so undervalued. People mistakenly think the goal is to generate precise probability estimates for frequently-discussed hypotheses (a goal in which deference can make sense). In a common-payoff-game research community, what matters is making new leaps in model space, not converging on probabilities. We (the research community) are bottlenecked by insight-production, not marginally better forecasts or decisions. Feign hubris if you need to, but strive to install it as a defense against model-dissolving deference.

This model of communication describes what LW would ideally be all about. Have mentally referenced this several times.

This is the first post in a series of short thoughts on epistemic rationality, integrity, and curiosity. My thanks to Jacob Lagerros and Alex Zhu for comments on drafts.

I am curious if this sentence turned out to be true, and if so what the followup posts were.

However, 'timelines' is not the interesting question.

Well, they are the decision-relevant question. At some point timelines get short enough that it's pointless to save for retirement. At some point timelines get short enough that it may be morally irresponsible to have children. At some point timelines get short enough that it becomes worthwhile to engage in risky activities like skydiving, motorcycling, or taking drugs (because the less time you have left the fewer expected minutes of life you lose from risking your life). Etc.

But yes, the thing where you discard your model is not great.

(Also, I personally have never generated a timeline using a model; I do something like Focusing on the felt senses of which numbers feel right. This is the final step in Eliezer's "do the math, then burn the math and go with your gut" thing.)

I generally got value of this post's crisp explanation. In particular, the notion that "model driven information-exchange" can result in people forming new beliefs that are more extreme that either person started with, rather than necessarily "averaging their beliefs together", was something I hadn't really considered before.

Some of the images in this post seem to be broken; when I try to load them individually, Firefox complains that they contain errors.

Curated this post for:

• Having a clear explanation of what I feel is a core skill for collective rationality: actually integrating the evidence and models that others have.
• Including the sentence "Don't practice coming to agreement; practice exchanging models" which succinctly summed up the core idea and is easy to remember.
• Having pretty pictures!

How this post could have been improved even further:

• The first time that I looked at this, I semi-automatically started skimming and didn't properly read everything; I had to come back and consciously make myself read everything. I'm not entirely sure of what caused it, but I suspect it had to do with the way that the post contained images and then often only had a brief quote and one or two sentences before the next image; something about this combination made my brain parse it as "there's a bit of stuff between the pictures but it probably isn't very important since there's so little of it", even though that was the actual substantial content.

Not sure how, of even if, this relates but seemed somewhat connected to what you're looking at. https://www.quantamagazine.org/were-stuck-inside-the-universe-lee-smolin-has-an-idea-for-how-to-study-it-anyway-20190627/

For some reason your parenthetical made me think of some of what Smolin was saying about how we view the universe and how we should try to model it.