If we can't lie to others, we will lie to ourselves

"the most powerful tool is adopting epistemic norms which are appropriately conservative; to rely more on the scientific method, on well-formed arguments, on evidence that can be clearly articulated and reproduced, and so on."

A simple summary: Believe Less. Hold higher standards for what is sufficient reason to believe. Of course this is in fact what most people actually do. They don't bother to hold beliefs on the kind of abstract topics on which Paul wants to hold beliefs.

"1. What my decisions are optimized for. .. 2. What I consciously believe I want."

No. 2 might be better thought of as "What my talk is optimized for." Both systems are highly optimized. This way of seeing it emphasizes that if you want to make the two results more consistent, you want to move your talk closer to action. As with bets, or other more concrete actions.

So, on ways of smoothing the incentive gradient for high-quality reasoning:

This is a reason to have a "rationalist community." Humans are satisficers. We won't really care about the opinion of literally all 7 billion people on Earth if we have the approval of our own tribe. If our tribe has some norms about how conversation and thinking work, then we'll be pretty able to follow those norms, so long as we expect that our needs are meetable within the tribe -- that is, that it's a good place to find friends, mates, careers, etc.

It's also a reason to think about how UX affects discourse. I'm by no means an expert in this, but for instance, what does karma reward? what types of expression get attention? How can we offer rewards for behaviors we like?

I think that there are other sources of social incentives for accuracy.

I imagined the "running late for a noon meeting" scenario, with myself in the role of the person who is waiting for you to arrive. At least in my mental simulation, I have a better impression of you if you say "ETA 12:10" and then show up at 12:10 than if you say "ETA 12:05" and then show up at 12:10. I see you as more reliable when your ETA is accurate; the hit in perceived reliability that you take by being later than the originally scheduled time is significantly smaller than the hit you take by being later than your ETA. So my judgments of people seem to be encouraging accurate estimates (at least in my simulation of this scenario). This effect becomes even stronger if I imagine similar interactions with you happening more than once.

Thinking about why this happens, while aiming for reasons that might generalize to other sorts of time estimation and broader sets of biases, I've come up with 4 factors:

1. Repeated interactions. When someone has a track record, patterns become evident. Oh really, traffic was surprisingly bad yet again? Or, in statistical terms, as the number of data points increases the average noise shrinks and systematic errors stand out.

2. The process that you're using to make your estimate is partially transparent. Just as you give off signs when you try to lie, you also give off signs when you try to think things through to give an accurate estimate, or when you try to set things up so that you won't be blamed for them going badly. e.g., If I come to you with a task and you say that it will take you 3 hours to do it, the questions that you asked before making that estimate will give me some clue about whether you're using an accuracy-seeking process.

3. Your stance towards me is partially transparent. When you're running late, is your aim "I value his time and don't want him to have to sit there waiting" or "I want him to think that this mostly wasn't my fault"? If you come to me with a project that needs my approval to move forward, are you thinking "How can I convince him to approve this?" or "Let's think this through together to figure out if it's worth doing"? The cooperative approach, where you see me as an ally and fellow agent in trying to create good outcomes, is well-served by accurate estimates which are communicated clearly. And people give off various signs of whether they have that approach.

4. Accurate estimation is a valued ability. If someone can consistently say things like "I'll be there in 8 minutes" or "that will take about 3 hours" and be very close to correct, that is an impressive ability which will lead me to rely more on their judgment. If they make systematic errors, or if their estimates are very noisy, then I will rely on them less. Trying to hide some bias in the noise is not such a great strategy, since noise also reflects a lack of skill at prediction. Interval estimates (e.g., "I'll be there in 20-25 minutes") help make the ability to give unbiased low-noise estimates more apparent.

These 4 factors don't solve the problem. Just as there are incentives to become a skilled liar, there are incentives to convincingly pretend that you value someone's input or to make your track record look better than your judgment is. But these 4 factors do seem to help, especially in smallish communities where the same people have repeated interactions (and can gather more dating by sharing impressions with others). To the extent that the rationality community (sometimes) succeeds at encouraging accuracy-seeking, I suspect that much of it comes from creating a social context where these factors are more strongly at play.