The "I Already Get It" Slide

There's a point intermediate between "completely new" and "just being difficult".

Fair enough. To me, your previous words pattern matched very strongly to 'being difficult because they think this is dumb but don't want to say why because it seems like too much work' (or something). My mistake.

I didn't mean new to LW, I meant new to the questions you were posing and the answers you got.

Back on the topic at hand,

In order to do that I would have to assume that I know what questions are the right ones and that he does not. Assuming this would amount to assuming that I am right about the subject and he is wrong.

Consider the following: you meet a friend of a friend who seems reasonable enough, and they start telling you about their startup. They go on and on for a long time but try as you might, you can't figure out how on earth they're going to make money. Finally, you delicately ask "how do you intend to make money?". They give some wishy washy answer.

Here they have failed to ask a question that you know to be important. You know this quite definitely. Even if they thought that the question were somehow not relevant, if they knew it was usually relevant, they would probably explain why its not in this particular case. Much more likely that they are just not very good at thinking about startups.

Similarly, if they anticipate all of your objections and questions, you will probably think they are being pretty reasonable and be inclined to take them more seriously. And rightfully so, that's actually decent evidence.

in which case I am again assuming I am right about the subject

There's a middle ground between 'assuming I am right' and 'assuming they are right'. You can instead be unsure how likely they are to be right, and try to figure it out. One way you can figure it out is by trying to assess whether they seem like they are doing good epistemic things (like do they actually pause to think about things, do they try to understand people's points, do they respond to the actual question, do they make arguments that later turn out to be convincing, do they base things on believable numbers, do they present actual evidence for their views, etc. etc.)

Are you familiar with the idea of 'latent variables' from Bayesian statistics? Are you used to thinking about it in the context of people and the real world? The basic idea is that you can infer hidden properties of things by observing many things it affects (even if it only noisily affects them).

For example, you go to a small school and observe many students doing very impressive science experiments, you might then infer some hidden cause that causes the school to have smart students. Thus you might also guess that in several years, different students at the same school will do well on their SATs, even though that's not directly related to your actual observations.

I suspect thinking a bunch about latent variables in the real world might be useful for you. Especially as it relates to inferring where people are reasonable and how much they are. Especially the idea of using data from different topics to improve your estimate for a given topic (say using test scores from different students to improve your quality estimate for a specific student).

This might be a good starting point: http://www.stat.columbia.edu/~gelman/research/published/multi2.pdf (read until sec 2.3).