The Paradox of Expert Opinion

post by Emrik (Emrik North) · 2021-09-26T21:39:45.752Z · LW · GW · 9 comments

Contents

  Sampling bias due to evidential luck is inevitable
  Adversarial Goodhart amplifying deception
None
9 comments

The best-informed opinions tend to be the most selection-biased ones.

If you want to know whether string theory is true and you're not able to evaluate the technical arguments yourself, who do you go to for advice? Well, seems obvious. Ask the experts. They're likely the most informed on the issue. Unfortunately, they've also been heavily selected for [? · GW] belief in the hypothesis. It's unlikely they'd bother becoming string theorists in the first place unless they believed in it.

If you want to know whether God exists, who do you ask? Philosophers of religion agree: 70% accept or lean towards theism compared to 16% of all PhilPaper Survey respondents.

If you want to know whether to take transformative AI seriously, what now?

The people who've spent the most time thinking about this are likely to be the people who take the risk seriously. This means that the most technically eloquent arguments are likely to come from the supporter side, in addition to hosting the greatest volume of persuasion. Note that this will stay true even for insane causes like homeopathy: I'm a disbeliever, but if I were forced to participate in a public debate right now, my opponent would likely sound much more technically literate [LW · GW] on the subject.

To be clear, I'm not saying this is new. Responsible people who run surveys on AI risk are well aware that this is imperfect information, and try to control for it. But it needs to be appreciated for its generality, and it needs a name.

Sampling bias due to evidential luck is inevitable

This paradox stays true even in worlds where all experts are perfectly rational and share the same priors and values.

As long as

1. experts are exposed to different pieces of evidence (aka evidential luck [? · GW]), and
2. decide which field of research to enter based on something akin to Value of Information [? · GW] (even assuming everyone shares the same values), and
3. the field has higher VoI the more you accept its premises,

then the experts in that field will have been selected for how much credence they have in those premises to some extent.

But, as is obvious, experts will neither be perfectly rational nor care about the same things as you do, so the real world has lots more potential for all kinds of filters [? · GW] that make it tricky to evaluate expert testimony.

Adversarial Goodhart amplifying deception

There are well-known problems with the incentives experts face, especially in academia. Thus, Adversarial Goodhart [LW · GW]:

When you optimize for a proxy, you provide an incentive for adversaries to correlate their goal with your proxy, thus destroying the correlation with your goal.

Whatever metric we use to try to determine expertise, researchers are going to have an incentive to optimize for that metric, especially when their livelihoods depend on it. And since we can't observe expertise directly, we're going to have to rely very heavily on proxy measures.

Empirically, it looks like those proxies include: number of citations, behaving conspicuously "professional" in person and in writing, confidence, how difficult their work looks, and a number of other factors. Now, we care about actual expertise $V$, but, due to the proxies above, the metric $U$ will contain some downward or upward error $E$ such that $U=V+E$.

When researchers are rewarded/selected for having a high $U$, we incentivize them to optimise for both $V$ and $E$. They can do this by actually becoming better researchers, or by increasing the Error—how much they seem like an expert in excess of how expert they are. When we pick an individual with a high $U$, that individual is also more likely to have a high $E$. Adversarial Goodhart makes us increasingly overestimate expertise the higher up on the proxy distribution we go.

But, of course, these incentives are all theoretical. I'm sure the real world works fine.

9 comments

Comments sorted by top scores.