Trusting Expert Consensus

Contents

  Evolution
  Economics
  Philosophy
    Philosophy vs. philosophy of religion
  Biblical scholarship
  Climate science
  Other examples?
  Postcript
None
81 comments

I recently read Bryan Caplan's The Myth Of The Rational Voter. It's an excellent book in a lot of ways, and one of those ways is how it got me thinking about the issue of expert consensus. It's pushed me more towards thinking that hard data on what the experts in a given field believe about their area of expertise is incredibly useful. Specifically, based on examples from a variety of fields (listed below the fold), I'll conclude:

• When the data show an overwhelming consensus in favor of one view (say, if the number of dissenters is less than the Lizardman's Constant), this almost always ought to swamp any other evidence a non-expert might think they have regarding the issue.
• When a strong but not overwhelming majority of experts favor one view, non-experts should take this as strong evidence in favor of that view, but there's a greater chance that evidence could be overcome by other evidence (even from a non-expert's point of view). 
• When there is only barely a majority view among experts, or no agreement at all, this is much less informative than the previous two conditions. It may indicate agnosticism is the appropriate attitude, but in many cases non-experts needn't hesitate before having their own opinion.
• Expert opinion should be discounted when their opinions could be predicted solely from information not relevant to the truth of the claims. This may be the only reliable, easy heuristic a non-expert can use to figure out a particular group of experts should not be trusted.

Economics

Expert opinion plays a central role in Caplan's Myth of the Rational Voter. Caplan argues that voters aren't just ill-informed about many topics, they suffer from biases that lead them to have views that systematically differ from those of the experts. And Caplan focuses on economics, both because it's his own area of expertise and because of it's obvious relevance for policy making.

I've heard The Myth of the Rational Voter described as having a "libertarian" perspective, but while Caplan is himself a libertarian, that didn't strike me as coming across particularly strongly in the book. I say this as someone who considers himself a fan of Paul Krugman, and in fact Caplan frequently cites Krugman in support of his points.

Furthermore, Caplan has hard data to back his claims up: one study found that, in 2000, 72.5% of economists mainly agreed with the claim that "tariffs and import quotas usually reduce the general welfare of society." A further 20.1% said they "agreed with provisos," while only 6% "generally disagreed." In another example Caplan cites, nearly three-quarters of economists generally disagreed with the statement "wage-price controls should be used to control inflation."

However, these numbers are not as overwhelming as the numbers on scientific support for evolution. And Caplan does not mind sometimes rejecting the majority view among his colleagues. He writes:

If my premise is that the economic consensus is reliable, how can I reach a conclusion that the economic consensus rejects?...

This complaint would be airtight if my premise were that the economic consensus is infallible. But my actual premise is merely that economists, like other experts, deserve the benefit of the doubt—and that the burden of proof rests on those who would question the expert consensus. Since the rational voter assumption is part of that consensus, my responsibility as a naysayer is to refute it—which is precisely why I needed to wrote this book.

I take it there's a connection between the fact that consensus in economics tends to be weaker, and Caplan's willingness to contradict the consensus on some issues. As a majority gets smaller, the chances that a scholar with the minority position might have good reasons to reject the minority view gets larger. Furthermore, when the majority view only has around 70%-75% of experts in its favor, it's also much more plausible that an informed layman could have good reasons to reject the majority view. Even though I'm not an economist, I don't hesitate to think Caplan is probably right to reject the rational voter assumption. Still, non-experts who have who know little about a debate should generally be willing to accept the conclusions of a 70+% majority.

Also relevant: Krugman himself has written about the issue of consensus in economics. His conclusion is that "most of what economists do is indeed fairly objective and non-ideological," and can produce considerable agreement among economists. However, there are some important issues in business-cycle macroeconomics where there are big, ideologically-driven disagreements. Even so, he notes that in one panel of 42 economists from top schools, 80% agreed with the statement that the 2009 American Reinvestment and Recovery Act "significantly boosted output and employment."

Something I almost forgot to mention, because it seems obvious to me, but probably doesn't seem obvious to other people: the views of economists probably can't be explained away as driven by political ideology. Anecdotally, people like Krugman are a problem for attempts to dismiss economists as libertarian ideologues (though I guess some people would dismiss Krugman as a neo-liberal sellout).

Furthermore, Caplan argues that data from the Survey of Americans and Economists on the Economy suggests the opinions of economists mostly cannot be explained as the result of ideology or socioeconomic class. He notes that, in spite of occasional accusations of "free-market fundamentalism" hurled at economists, the reality is that economists are acutely aware of the weaknesses of markets.

Philosophy

This is my own field, as much as someone who's left academia can claim a field as his own. And in philosophy, we have unusually good data about what professional philosophers believe about the major disputes in their field, thanks to a 2009 survey organized by philosophers David Bourget and David Chalmers through the website PhilPapers.org.

Robin Hanson commented when the survey data was released, and seemed happy to learn he agreed with the modal expert opinion on 25 out of the 30 questions surveyed. But this seemed to me, and still seems to me, the wrong reaction, given that on most questions there was no majority view. On the questions where there was a majority view, on about half of those there was still a large amount of disagreement, with less than 60% of philosophers accepting the majority view.

Under such circumstances, I'm inclined to think the opinion of the experts tells us very little about what the right view is. My reaction was partly colored by experience within philosophy, which has led me to think most philosophers don't have very good reasons for their philosophical opinions and agnosticism towards particular questions is more often the right view than most philosophers would admit.

It's interesting, though, to look at look at the three questions where there was most agreement. 81.6% of philosophers endorse non-skeptical realism about the external world, 75.1% endorse scientific realism, and 72.8% endorse atheism. Note that on all questions, there was a significant number of "other" answers, for example, on God, only 14.6% of philosophers endorsed theism, 5.5% were agnostic, while others endorsed various idiosyncratic views.

My initial reaction to these numbers were that they were the exceptions that proved the rule. "Non-skeptical realism about the external world," is basically just the claim that there's stuff (tables and chairs and trees and rocks and so on, or perhaps subatomic particles) that exists outside our minds and we can know about them. The fact that only about 80% of philosophers are able to agree on this seemingly commonsensical proposition just underlines how incapable of agreeing about anything philosophers are.

Scientific realism, similarly, is just the view that science describes the real world, and the less-than-total agreement of philosophers on this idea is similarly disappointing. The question of God's existence, meanwhile, strikes me as having more in common with the question of whether fairies exist than most other questions on the survey.

Yet as I've thought about the question more, particularly philosopher's opinions on God's existence, I've shifted towards thinking there are some questions where a strong majority of philosophers agreeing is significant. For one thing, it's striking how few theistic philosophers are interested in defending traditional arguments for the existence of God. With a few exceptions (most notably Richard Swinburne and William Lane Craig), they stick to arguing that belief in God is reasonable. That certainly seems worth mentioning if you're an atheist trying to persuade theists that there are no good arguments for the existence of God.

Reading The Myth of the Rational Voter caused my views to shift further, because it made me realize the level of agreement you'll find among philosophers on the external world, science, and God is about as high as you'll find among economists on major questions of economics. This makes me think that probably atheists should not be shy citing strong support for atheism among philosophers as a reason to be an atheist. I wouldn't claim it's conclusive all by itself, but it does create a presumption that atheism is the most rational view, which should place a strong burden of proof on the theist to overcome.

Arguably, there's also a lesson here for philosophy professors teaching undergrads (particularly in 101 classes). The standard approach to teaching philosophy is to make a show of being very even-handed in presenting any given debate, at most subtly prodding students towards the professor's preferred view. But it might be better for professors to be unapologetic about explaining to students that most philosophers are atheists and why.

The value of taking the same approach to knowledge of the external world and science may be less obvious, but I suspect it may be even more important. Too many undergraduates come out of their philosophy 101 classes thinking the lesson of philosophy is that we don't really know anything, or that science is a matter of faith too. Philosophy professors should consider trying to actively fight that tendency.

(What about the other issues that had relatively high levels agreement among philosophers? Professors might make a point of defending them too, though the next four most agreed-upon points from the PhilPapers survey aren't as exciting: the a priori, the analytic/synthetic distinction, the trolley problem, and cognitivism about moral judgement. At least, they're not as exciting until you connect them to other, more controversial, issues.)

The broader lesson I take away from this is that you don't need to worry about looking for the kind of symptoms that Luke has cited in diagnosing philosophy as a "diseased discipline." The example of philosophy suggests that when a group of experts is deeply confused about the subject-matter of their own discipline, they mostly fail to converge on any kind of strong majority agreement. That means all a non-expert needs to do is look at the data on expert opinion and if there's no strong majority agreement, they should feel pretty comfortable ignoring the experts when they feel it's necessary to do so.

And yes, though I'm inclined towards agnosticism on a lot of philosophical questions, that does mean it's often OK for non-philosophers to have strong opinions on philosophical questions that philosophers themselves can't agree on. For example, I think Eliezer should feel pretty comfortable with his advocacy of one-boxing on Newcomb's problem, even though a plurality of philosophers support two-boxing.

Philosophy vs. philosophy of religion

There's an important side-note to make here regarding the opinions of specialists in philosophy of religion. They, unlike professional philosophy as a whole, tend to be theists; 72.3% theists vs. 19.1% atheists. Some theistic philosophy-bloggers took this as vindication of theism, arguing philosophy of religion specialists know the most about the issues so they're who we should trust. On the other hand, it looks suspicious that they haven't managed to convince their colleagues in other specialties. What should we make of this?

The answer becomes obvious once you know a little more about the professional dynamics surrounding philosophy of religion. Philosophy of religion is not taken very seriously in mainstream philosophy. That is not a controversial point; philosophers of religion know it and complain about it. When I was at Notre Dame (the department rated number one for philosophy of religion in the world), one of the department's philosophers of religion specifically had the job of taking the religious grad students aside and telling them they had to specialize in something other than philosophy of religion or else they would not get a job.

This means atheist philosophers have basically no incentive to work in philosophy of religion. If you read the biographies of most philosophers of religion, overwhelmingly they were religious before they started studying philosophy and got into philosophy of religion because they really, really wanted to use philosophy to defend and/or elaborate on their religious beliefs. Even philosophers of religion are beginning to notice that philosophy of religion often resembles religious apologetics more than objective inquiry.

I'll save the technical account of why this matters for the section on Biblical scholarship, but here I'll note that it's hard to explain away mainstream support for atheism in a parallel manner. Academics do tend to be less religious than the general population, true, but philosophers are an outlier even among other academics. A 2009 study that looked at religious belief within the 20 largest academic disciplines found that only about 10 percent of academics are atheists, with psychologists being the most atheistic at 50 percent. 50 percent is extremely high compared to the general US population, but still not up to philosopher levels.

Biblical scholarship

I unfortunately don't have good data on the opinions of Biblical scholars, but I'm going to talk about it because it's an area I know a lot about (particularly historical Jesus research), and there's an important point to be made here about how non-experts can decide whether the opinions of experts are trustworthy.

The point is a rather obvious one: Christian apologists like William Lane Craig often cite the (alleged) consensus of Biblical scholarship in support of their arguments, but even if the basic claims were true, it wouldn't be to find that Biblical scholars tend to have views favorable to Christianity (and Judaism) because Biblical scholars are overwhelmingly Christians and Jews. Biblical scholars tend to get their degrees from seminaries rather than secular academic departments. Atheist Biblical scholar Jacques Berlinerblau once estimated that at a typical meeting of a thousand members of the Society for Biblical Literature, you'd find only a couple dozen atheists (I'm not sure if he meant to include people like Bart Ehrman, who identifies as an agnostic).

Non-believing Biblical scholars tend to be former believers who deconverted in part due to their studies; this is true of Bart Ehrman, Gerd Lüdemann, Michael Goulder, and Robert M. Price. Jacques Berlinerblau is an exception to the "former believers" rule. Hector Avalos is also a partial exception (his interest in the subject stems from his time a teen evangelist, but he deconverted before starting his graduate studies). I e-mailed Berlinerblau about this once, and he could think of one other scholar in the field who he didn't think had ever been a believer, but confirmed that such people are extraordinarily rare.

This doesn't mean that outside the few dozen atheists and agnostics, Biblical scholars are all going to agree with William Lane Craig, because there are also plenty of liberal Christians like John Dominic Crossan and Marcus Borg in Biblical scholarship. It does mean that if it did turn out that most Biblical scholars had views favorable to (traditional) Christianity, it wouldn't mean much because it would be no more surprising that finding out that Muslim scholars tend to have views favorable to Islam. It's the Crossans and the Borgs that are surprising (regardless of the exact liberal Christian / conservative Christian breakdown).

This common-sense argument in terms of what is and isn't surprising is reinforced by a Bayesian analysis: if the opinions of a group of experts can be predicted using information that isn't relevant to whether or not their opinions are true, knowing what their opinions are gives you no information on the claims in question. It's important to distinguish this heuristic from post hoc explaining away of expert opinions you don't like: conservative Christians often propose unflattering explanations for the opinions of the Crossans and the Ehrmans, but it's unlikely that they could have predicted their opinions using information not relevant to whether their opinions are right.

So far, I probably haven't said anything that would surprise the average member of LessWrong: of course Jesus didn't perform any miracles or rise from the dead. So let's tackle a slightly harder question: was there a historical Jesus at all? Or, to put it somewhat more precisely: can Christianity be traced back to the followers of a single Jewish preacher who was executed by the Roman authorities in Palestine in the first half of the first century A.D.?

That Christian scholars think Jesus existed isn't surprising. Even Crossan has his own version of the historical Jesus, as a wise teacher who's palatable to liberal churchgoers, that he wants to sell. Knowing the opinions of Crossan et al. may be slightly informative, since since many liberal Christian and Jewish scholars seem to have reconciled themselves to there being no historical Moses, so the fact that they haven't gone that route with Jesus may give us some information. But looking at the opinions of non-believing scholars specifically will be more informative.

Though we don't have survey data because non-believing Biblical scholars are so rare, and scholars who doubt Jesus' historicity even rarer, we can make an estimate. Richard Carrier (a historical Jesus skeptic with a PhD in history from Columbia University) recently listed Arthur Droge, Kurt Noll, and Thomas Thompson as people who "agree historicity agnosticism is warranted," in addition to listing Thomas Brodie, Robert M. Price, and himself as being "even more certain historicity is doubtful." Because historicity skeptics are so rare, and because Carrier knows the debate so well, this list is probably exhaustive at least as far as publicly expressed views are concerned.

There are a number of difficulties here. First, it's unclear whether Berlinerblau's "two dozen" estimate includes agnostics. Second, categorizing Carrier's list of skeptics gets tricky in a few cases. Robert M. Price is a non-believer who's officially agnostic on whether Jesus existed, but he's written a lot about the issue and is quite confident agnosticism is the right view, justifying his inclusion on the "more certain" list. Brodie has apparently tried to argue he can be a good Catholic while rejecting the idea of a historical Jesus (which, as I've already noted, is a position some people have already taken on Moses). Noll describes himself as "theistic off the job and professionally agnostic."

Finally, Brodie has apparently been convinced there was no historical Jesus since the 1970s, but until recently was keeping quiet about it (for fear of the professional repercussions?) That raises the question of whether there might be other skeptics out there still in the closet. Still, I'd venture that probably at least 75% (Carrier's 6 divided by Berlinerblau's 24) of non-religious scholars with relevant expertise are at least fairly confident Jesus existed. Much of the remainder would be agnostic rather than full-blown mythicists. That would be strong, but not overwhelming, support for historicity.

Having read both defenses of historicity by people like Ehrman, and critiques by Robert M. Price and Earl Doherty (who's an amateur, but has Carrier's endorsement), my impression is that even ignoring head counts the historicists have the better of the argument. But Carrier has said he doesn't expect most scholars to agree with him without having seen his research and arguments. He'll be presenting those in a book which is slated for release early next year. I think the book probably won't persuade many people, but I don't think Carrier's approach is necessarily foolish. It's possible Carrier has some decisive evidence that other people just don't have access to, yet.

Climate science

Reference to global warming on LessWrong sometimes elicit negative reactions (often with references to a misinterpreted version of Politics is the Mind-Killer), yet on the merits global warming seems to be in the same situation as evolution. Wikipedia, again, has a good summary of the data on expert opinion, even better than the one for evolution. Highlights include:

• The 2007 IPCC report concluded that, "Most of the observed increase in global average temperatures since the mid-20th century is very likely due to the observed increase in anthropogenic GHG concentrations," with "very likely" defined as >90% certainty.
• A survey of members of the American Meteorological Society and American Geophysical Union found that 97% agreed that global temperatures had increased during the past 100 years and 84% agreed that human-caused warming was occurring.
• Other studies suggest that, among the climatologists who are most active in publishing on the issue, support for the mainstream consensus on the causes of global warming is even higher: 97-98%.

Other examples?

The above examples largely exhaust the cases I know about where a strong majority of experts agree on an issue, yet the issue is controversial among people outside the field. The age of the earth, the age of the universe, and the Big Bang are also technically examples because young earth creationism, but you already knew that.

However, I can think of some minor examples that seem to support the principle that when there's little to no agreement among the experts, non-experts should focus on other evidence if they're going to have an opinion. For example, there's currently no majority view among physicists on how to interpret quantum mechanics, and nothing strikes me as inherently foolish about Eliezer's advocacy of many-worlds. Similarly, though I don't have data my impression is that evolutionary biologists and psychologists are divided on evolutionary psychology, but I feel no hesitation in thinking that the basic tenets of evolutionary psychology are correct.

Can we find counter-examples to the heuristics suggested above? A case where, for non-obvious reasons, a strong majority of the experts in some field have converged on a crazy conclusion? The more heavily postmodern corners of academia might be a good place to look for that. In "The Myth of the Postmodern University," Brian Leiter argues that the influence of postmodernism on academia has been greatly exaggerated, but it has had significant impact on English, literature, and history. Leiter also quotes a reader e-mail suggesting that communication, media studies, cultural studies, and journalism may also belong on the of fields heavily influenced by postmodernism. So it may be worth looking at those.

But my guess is that there may be less there than meets the eye. I don't doubt you can find many people with crazy pomo ideas in those fields, but I suspect you'd also find lots of people in those same fields who generally reject the crazy pomo ideas. At worst, I'd expect to find 50-odd percent support for the crazy view with a significant number of objectors. But I may be wrong about this, and it would be helpful to find good data.

Another issue: it may seem surprising that I haven't been able to uncover clear evidence of a field corrupted by political ideology, the way philosophy of religion and Biblical scholarship are significantly corrupted by religious agendas. This may be because religions like Christianity have a fairly definite body of doctrine (left-wing theology aside), making bias easy to detect. Political categories like "liberal" or "conservative," on the other hand, are squishier, which plausibly makes bias harder to detect.

Indeed, though the effects may be less obvious, there does seem to be evidence for a general self-selection effect: liberal college students are more likely to pursue careers in academia. That certainly suggests you shouldn't vote for Democrats just because most college professors do, but it's unclear how it affects specific academic debates. The fact that even liberal economists tend to have many views that don't fit the liberal stereotype suggests these effects tend to get swamped by the effects of having relevant expertise, when someone's dealing with their own field. But perhaps there are other, less encouraging examples out there that I'm unaware of.

Finally, left-right political bias may be much less important than nationalistic bias. I once heard it said (by Orwell? Russell? Google isn't helping me find the quote) that if you're from a small country that hasn't played an important role in world affairs, you'll likely be able to find an objective history of your country written by a foreign historian, but otherwise you're probably out of luck for an objective history of your country. Among US historians, there's an obvious counter-point that people like Howard Zinn and James W. Loewen don't seem at all unusual. Which is not to say they're necessary very objective, just that the biases among US historians may run in enough different directions to prevent the entire history profession from converging on nonsense.

On the other hand, even among left-wing historians in the US, how many would go as far as gwern does and say the American Revolution was a mistake? Yet gwern seems to to have a strong case: it's definitely a bad thing when a war kills tens of thousands of people, and the US and Canada didn't turn out all that different. So, contrary to what many would expect, it may be that even left-wing US historians are too positive about America. This suggests that, on issues where nationalistic bias could be a problem, you should be cautious about accepting the majority opinion of historians unless you have international data.

Postcript

For some extra context for this post, see my discussion post on academic cliques.

81 comments

Comments sorted by top scores.