Learning To Love Scientific Consensus

post by Scott Alexander (Yvain) · 2017-09-02T08:44:12.184Z · LW · GW · 1 comments

Contents

  I.
  II.
  III.
None
1 comment

[Related to: Contrarians, Crackpots, and Consensus, How Common Are Science Failures?. Epistemic status is “subtle and likely to be misinterpreted”.]

I.

There’s a list of scientific mavericks who were ridiculed by hidebound reactionaries but later vindicated that’s been going viral. I examined the first ten mavericks on the list to see if its claims held up. Overall I wasn’t too impressed. Let me go over them in more detail.

SVANTE ARRHENIUS:

His idea that electrolytes are full of charged atoms was considered crazy. The atomic theory was new at the time, and everyone “knew” that atoms were indivisible (and hence they could not lose or gain any electric charge.) Because of his heretical idea, he only received his university degree by a very narrow margin.

Sure, the professors who were judging his PhD thesis weren’t too convinced. So Arrhenius sent his proposal to the world’s top chemists at the time, and they were super-interested and started fighting among themselves to work with Arrhenius on it. Top chemist Wilhelm Ostwald received the paper the same day his daughter was born, and suggested that the paper was the more exciting of the two events. He journeyed to Arrhenius’ hometown of Uppsala, Sweden to try to convince Arrhenius to work with him; Arrhenius refused for personal reasons but later got a scholarship and worked with the top physicists in Europe. Arrhenius became a professor in a prestigious university about ten years after presenting his “ridiculed” paper, and won the Nobel Prize ten years after that.

HANS ALFVEN:

Astronomers thought that gravity alone is important in solar systems, in galaxies, etc. Alfven’s idea that plasma physics is of equal or greater importance to gravity was derided for decades.

This isn’t a great description of Alfven’s conflict with the establishment, but the list seems basically right insofar as Alfven’s ideas were ignored for thirty years before being proven mostly correct. I will give them this one.

JOHN BAIRD:

When the first television system was demonstrated to the Royal Society (British scientists,) they scoffed and ridiculed, calling Baird a swindler.

I can’t find any reference to this in various Baird articles and biographies. The closest I can come is this article by someone who was there at the demonstration, who said “They didn’t believe it…the pictures were a bit of a blur but it was amazing, they were all absolutely flabbergasted by it.” It looks like he is using “they didn’t believe it” in the colloquial way of “they thought it was amazing”. A TIME magazine article from the time described the same scientists as “deeply impressed”, though the wording is kind of unclear and they might have been referring to a different demonstration a year later.

In any case, it seems very clear that within a year everyone agreed he was legitimate and overcame their initial shock.

ROBERT BAKKER:

Everyone knows that dinosaurs are like Gila monsters or big tortoises: large, slow, and intolerant of the cold. And they’re all colored olive drab too! 🙂

Bakker did help produce the paradigm shift in paleontology from cold-blooded dinosaurs to warm-blooded dinosaurs. But he was not a lone maverick being ridiculed by everyone else. He learned that dinosaurs were warm-blooded from his professor at Yale, who was also part of the minority-but-totally-existing faction that believed dinosaurs were warm-blooded. He himself got a PhD at Harvard from professors who were apparently sympathetic to the same theory. And within seven years of his first paper being published, Scientific American was calling his ideas “the dinosaur renaissance”, which doesn’t leave a lot of time for him to be ridiculed and ignored in.

BARDEEN & BRATTAIN:

Not ridiculed, but their boss W. Shockley nixed their idea for a non-FET “crystal triode” device. When they started investigating it, he made them stop. They were supposed to be working on FETs instead.

ARG, I GOT THIS WRONG, THIS PART BELOW IS A BELL LABS STORY REGARDING ZONE REFINING OF SILICON, NOT THE BJT TRANSISTOR PROJECT: So, they assembled their ZONE REFINING experiment on a wheeled cart and continued. Whenever the boss was scheduled to check up on them, they could shove it into an adjacent unused lab.

Okay, it looks like the guy compiling the list admits he was wrong on this one. Moving on…

BRETZ:

Endured decades of scorn as the laughingstock of the geology world. His crime was to insist that enormous amounts of evidence showed that, in Eastern Washington state, the “scabland” desert landscape had endured an ancient catastrophy: a flood of staggering proportions. This was outright heresy, since the geology community of the time had dogmatic belief in a “uniformitarian” position, where all changes must take place slowly and incrementally over vast time scales. Bretz’ ideas were entirely vindicated by the 1950s. Quote: “All my enemies are dead, so I have no one to gloat over.”

This one is basically right and I’ll give it to them.

CHANDRASEKHAR:

Chandra originated Black Hole theory and published several papers. He was attacked viciously by his close colleague Sir Arthur Eddington, and his theory was discredited in the eyes of the research community. They were wrong, and Eddington apparently took such strong action based on an incorrect pet theory of his own. In the end Chandra could not even pursue a career in England, and he moved his research to the U. of Chicago in 1937, laboring in relative obscurity for decades.

Sort of true, but he was hardly shunned by the scientific community. He made his discoveries about black holes in the early 1930s, was well-received by many people, and won a Bronze Medal in some physics competition. In 1935, Eddington attacked his theory, possibly because Eddington was racist and didn’t like Indian people. But many other scientists, including Niels Bohr and Wolfgang Pauli, continued to support him (quietly, so as not to offend Eddington, which will be a recurring theme in these kinds of situations). Chandrasekhar was made a Fellow of the Royal Society in 1944, won the Royal Astronomical Society Gold Medal in 1953, and generally led a long and prestigious life. His theories were resurrected once people had better evidence that black holes existed. I’ll give this one half a point.

CHLADNI:

The scientific community regarded Meteorites in the same way that modern scientists regard UFO abductions and psychic phenomenon: quaint superstitions only believed by peasant folk. All the eyewitness reports were disbelieved. At one point the ridicule became so intense that many museums with meteorites in their geology collections decided to trash those valuable samples. (Sometimes hostile skepticism controls reality, and the strongest evidence is edited to conform to concensus disbeliefs.) Finally in the early 1800’s Ernst Chladni actually sat down and inspected the evidence professionally, and found that claimed meteorites were entirely unlike known earth rocks. His study changed some minds. At the same time some large meteor falls were witnessed by scientists, and the majority who insisted that only ignorant peasants ever saw such things were shamed into silence.

As the quote points out, this is a kind of weird one as meteorite work was ridiculed for a long time, but Chladni was taken seriously and helped change minds. Looking at Wikipedia, a lucky meteorite fall two years after Chladni first published his theory helped turn the tide in his favor, and by ten years after publication Chladni’s meteorite theories were pretty well-regarded. Even when people disagreed with him about meteorites, Chladni remained widely respected for some of his other work in acoustics.

There is a story here, but it’s probably not right to center it around Chladni, and his work was only scorned for a few years before everyone agreed it was true. I’ll give this another half a point.

CRICK & WATSON

Not ridiculed. But they were instructed to drop their research. They continued it as “bootleg” research.

The list admits they were “not ridiculed”. They were told to stop their research because there was all sorts of academic politics around who was going to be the first to discover DNA, and the guy in charge of their university was rooting for another team.

DOPPLER

Proposed a theory of the optical Doppler Effect in 1842, but was bitterly opposed for two decades because it did not fit with the accepted physics of the time (it contradicted the Luminiferous Aether theory.) Doppler was finally proven right in 1868 when W. Huggins observed red shifts and blue shifts in stellar spectra. Unfortunately this was fifteen years after Doppler had died.

I haven’t been able to find anything about this in various short online biographies of Doppler (1, 2). Doppler tested the effect himself by having someone play a trumpet on a train (really), someone else successfully tested it in 1845, and it was independently rediscovered in 1848. Doppler himself was made the head of the Institute For Experimental Physics in Vienna and died about as prestigious and beloved as a physicist can get.

So my impression is that only a third of these people really fit the pattern. Most of them were doubted for very short periods, continued to be respected in their fields for their other accomplishments even during those periods, or were part of medium-sized movements rather than being lone geniuses. After a few years – maybe an average of ten, very rarely as long as thirty – their contributions were recognized and they assumed their rightful place in the pantheon. Science isn’t perfect. But it is darned good.

[EDIT: Bill Beatty, author of the original list, responds here. My response to the response here.]

II.

I bring this up in the context of my last post on progress in the rationalist movement. There used to be a stereotype that rationalists were too quick to challenge scientific consensus. I think that was exaggerated, but based on a core of truth. Given that we’re interested in the ways that bias can prevent people from accepting truth, it’s unsurprising that we would focus on cases like these.

But I personally have changed my thinking on this a lot. Not in any way that I can explain explicitly – I’ve always thought something like:

Scientific consensus is the best tool we have for seeking truth. It’s not perfect, and it’s frequently overturned by later scientists, but this is usually – albeit not literally always – the work of well-credentialed insiders, operating pretty quickly after the evidence that should overturn it becomes available. Any individual should be very doubtful of their ability to beat it, while not being so doubtful that nobody ever improves it and science can never progress.

– and I still think that. But I’ve shifted from being the sort of person who shares viral lists of maligned geniuses, to the sort of person who debunks those lists. I’ve started emphasizing the “best tool we have” part of the sentence, and whispering the “isn’t perfect” part, rather than vice versa.

I’ve changed my mind on this because of personal experience. Rather than trying to describe it, it might be more helpful to give the most salient examples.

1. The Replication Crisis: I previously thought the scientific consensus was flawed because it failed to take the replication crisis seriously enough. I later learned that everyone else took the repliaction crisis exactly as seriously as I did. A poll in Nature shows that 90% of scientists believe reproducibility issues constitute a “crisis”, compared to only 3% (!) who don’t. For every person complaining about “methodological terrorists”, there are a dozen who are very concerned and trying to change the way they practice research.

This is especially impressive because as far as I can tell the whole shift happened in about ten years. I would date the beginning of the crisis from Ioannidis’ original 2005 paper, although it was only aimed at medicine. It got into high gear in psychology sometime around 2011 with Simonsohn’s False Positive Psychology. A Google Trends analysis suggests people only started searching the relevant keywords around 2013.

I started thinking about this sort of thing in 2009 after reading this LW post. At the time I thought this was some sort of exciting failure of modern science that I alone had figured out. But this was well after sharp people like Ioannidis were talking about it, and only a few years before everyone was talking about it. Framing this as “I was right and scientific consensus was wrong” seems grandiose. Better might be “I started betting on a winning horse about a quarter of the way between the beginning of the race and when its victory became blatantly obvious to everyone”.

2. Nutrition: The Bad Old Paradigm of nutrition says that obese people just have poor impulse control, that weight is a simple matter of calories in vs. calories out, and that all calories are equally good except fat, which for some inexplicable reason is the Devil. Anybody who’s read a few good books about nutrition science knows that the Bad Old Paradigm is woefully inadequate. I read a few of those books and became convinced that I was right and scientific consensus was wrong.

Unfortunately, this whole issue exploded when Gary Taubes published Good Calories, Bad Calories, which as best I can tell combined the first publicly available good critique of the Bad Old Paradigm with a flawed and basically false attempt at a new paradigm. There were lots of confused attacks against Taubes’ bad information which did collateral damage to his good information, and lots of confused defenses of his good information which inadvertently shielded his bad information from criticism. I previously focused on defend the good parts, but recently shifted more towards criticizing the bad parts.

After reading some more good books here (one of which I hope to review soon), my impression is that most nutrition scientists don’t believe in the Bad Old Paradigm and haven’t for a while. At the very least, most of them seem to believe in the lipostat and think it’s important, which is my proxy for “basically has their heart in the right place”. Insofar as the Bad Old Paradigm continues to be popular wisdom, it’s because of the diet industry, the government, social inertia, and nobody really having a good new paradigm to replace it with. I’m gradually seeing popular wisdom shift, and nutrition scientists themselves seem to be helping this process rather than hurting it.

Maybe somebody in this area has discovered the new paradigm and is a maverick being persecuted by hidebound reactionaries. But it isn’t Gary Taubes. And it certainly isn’t me.

3. Social-Justice-Related Issues: Another narrative I used to believe was that a lot of sketchy ideas were being flattered because they spoke to left-leading academics’ biases in favor of social justice. Implicit association tests, stereotype threat, the idea of zero meaningful psychological differences between men and women, et cetera.

When I started worrying about implicit association tests, I thought I was defying some kind of broad scientific consensus. But the meta-analyses showing the Implicit Association Test didn’t do what people thought had been around since 2009 and have only gotten more numerous since then, with broad media coverage. Problems with stereotype threat research are getting mainstream coverage and even airtime on NPR.

The problem here is that there was no equivalent of the Nature poll on the replication crisis, so I didn’t realize any of this was happening until just recently. For example, in 2016 this Voxsplainer made it sound like there was a monolithic consensus in favor of Implicit Association Tests that no sane person had ever disagreed with, even though by that point there were already several big meta-analyses finding they weren’t practically useful. The correct conclusion isn’t that this is really what scientific consensus thinks. The correct conclusion is that Vox shouldn’t be trusted about any science more complicated than the wedge vs. inclined plane. Once I realized that there was all this intelligent analysis going on that I’d never heard about, my claim to be boldly defying the scientific consensus evaporated.

Yes, Cordelia Fine is still around and is still writing books arguing against gender differences. But she’s starting to sound really defensive, basically the literary equivalent of “I know I’m going to be downvoted to hell for this, but…”. Meanwhile, other scientists are doing a good job pointing out the flaws in her books and conducting studies like this biggest-ever look at male vs. female brain differences, this magisterial look at personality differences, et cetera – not to mention great and widely-accepted work on how intersex people take on more characteristics of their hormonal than their social gender (honestly, we should probably thank transgender people for making this field socially acceptable again). People talk a lot about how Larry Summers was fired from Harvard for talking about male vs. female differences, but Steven Pinker did a whole debate on this and remains a Harvard professor.

Even things about genetic psychological differences between population groups are less bold and maverick-y than their proponents like to think. The relevant surveys I know trying to elicit scientific consensus (1, 2, 3) all find that, when asked anonymously, most scientists think these differences explain about 25% – 50% of variance.

I hate to bring that up, because it’ll probably start a flame war in the comments, but I think it’s important as a sign of exactly how hard it is to politicize science. Global warming skeptics talk about how maybe the scientific consensus on global warming is false because climatologists face political pressure to bias their results in favor of the theory. But scientists studying these areas face much more political pressure, and as long as you give the surveys anonymously they’re happy to express horrendously taboo opinions. This is about the strongest evidence in favor of the consensus on global warming – and scientific consensus in general – that I could imagine.

4. Nuture Assumption and Blank Slatism: The prologue of the first edition of The Nurture Assumption is Judith Rich Harris telling her “maverick genius kept down by hidebound reactionaries” story. But the prologue of the second edition is her being much more hopeful:

To some extent at least, times have changed…there is now more acceptance of the idea that behavior is influenced by genes and that individual differences in behavior are due in part to differnces in genes. People are more willing to admit that children can inherit behavioral quirks and personality characteristics…was it this cultural shift that led to greater acceptance of my theory? Or was it the fact that new findings, consistent with the theory, kept turning up? Over time, the early, angry response to The Nurture Assumption has softened noticeably, both within and outside of academia. Today, the book is widely cited in textbooks and journal articles. It’s assigned and discussed in courses in many colleges and universities; it shows up in exams…in his foreward to the first ediction of The Nurture Assumption, Steven Pinker made a rash prediction about the book: “I predict it will come to be seen as a turning point in the history of psychology”. Perhaps it is too soon to judge whether psychology has rounded a bend; perhaps it will take the perspective of twenty or thirty years. Even at this point, though, there are signs of a slight shift in direction. Within developmental psychology, I’ve noticed that descriptions of procedures and results are beginning to sound a bit defensive. Greater progress has been made in other areas of psychology. And the email I receive from students gives me high hopes for the younger generation coming up.

There were ten years between the first and second editions of The Nurture Assumption. In the almost ten years since the publication of the second edition, my impression is that its ideas have become even more widely-accepted. This month’s edition of the American Journal of Psychiatry, onbe of the top journals in the field, has a great study showing that child abuse does not cause cognitive disability, in contrast to several previous studies in the area. It cites Deary, Plomin, and Ioannidis, hits all of the talking points about genetic confounding of developmental outcomes, and receives glowing endorsement in the journal’s editorial section, which says that “if our causal explanations are wrong, we may be wasting our effort or even doing damage”. Every single psychiatrist in the country is getting exposed to this way of thinking.

And this has real results. I got to present a summary of behavioral genetics to a meeting of psychiatrists, including a lot of psychoanalysts, and I was shocked that most of them were at least a little receptive. I think they misunderstood it. I think they carefully raised caveats in exactly the right places to ensure they didn’t have to change anything they were doing. But the overall response was “Oh, yeah, we’ve heard stuff like that, it seems plausible, good thing that for various hard-to-explain reasons none of it applies to us.” This is what the first stage of progress looks like.

5. Intelligence Explosion And AI Risk: This was another place where I and many of my friends thought we were right and the consensus was wrong. It was another place where a lot of self-appointed defenders of the consensus told us we were crackpots and needed to listen to what real scientists thought. And again, when I looked into it, there was no consensus against the idea and lots of prominent researchers were in favor. Going to the Asilomar Conference and seeing a bunch of people from MIT and Harvard talk about how concerned they were really opened my eyes on this. Google now has an AI Ethics Board, Berkeley, Oxford, and MIT have foundations working on it, and people like Elon Musk and Bill Gates are involved. Bostrom’s survey of AI researchers and some more recent and rigorous not-yet-published surveys I’ve heard about confirm the impression. Nobody would ever say there’s a scientific consensus in favor of Bostrom’s theories. But at this point I think it’s also indefensible to say there’s a consensus against.

Bostrom first started writing about these sorts of things extensively in the early 2000s, so there was really only a ten-year gap between entering the intellectual environment and it becoming a (mostly) accepted part of the established field. Those ten years felt pretty long while we were in them, but the ability of a field to accept an on-the-face-of-it completely-insane-sounding theory within ten years seems to me a very strong argument against the hidebound-reactionaries theory and a very strong argument for considering scientific consenses to be unreasonably effective.

6. IQ: Another case where I worried about apparent failure of scientific consensus due to politically bias. I certainly encountered a lot of falsehoods around this when I was younger. My high school psychology textbook included a section claiming that all IQ tests were biased towards rich white people because they were based entirely on questions like “how many shots below par is a bogey?” Then it presented an “alternate IQ test” which “proved” that poor minorities had higher IQs than rich whites by asking some other questions with the opposite bias (I think they were about slang for drugs – certainly an interesting way to fight stereotypes). This kind of thing naturally made me assume that nobody had any idea what was actually in IQ tests and scientists were idiots.

But more recently I’ve been reading actual surveys, which find that about 97% of expert psychologists and 85% of applied psychologists agree that IQ tests measure cognitive ability “reasonably well”. And 77% of expert psychologists and 63% of applied psychologists agree IQ tests are culture-fair (with slightly different numbers depending on how you ask the question, but always about 50% of both groups).

This seems like less of a problem with expert consensus, and more of a problem of nobody else (including textbook writers!) listening to experts who are continually trying to beat reality into people’s heads. But I have a vague memory of having recently seen a survey (which I can’t find) that even experts in softer fields like sociology are generally in favor of IQ and admit that it has its uses. And even some left/liberal sources like Vox and Freddie deBoer are aware of the consensus and willing to respect it.

At the same time, I’ve encountered some people like Borsboom and Nostalgebraist who have relatively sophisticated (and limited) critiques of IQ, and who have allowed me to round off other people’s less-well-framed critiques to something more like what they are saying and less like the stupid things my high school textbook said.

So it seems to me that generally experts agree with reasonable statements about IQ, and where they seem to disagree they may hold reasonable disagreements rather than unreasonable ones. Again, where this fails is not in the experts but in the ability of people who don’t listen to the experts to get disproportionate social power and hide the existence of the expert consensus.

III.

Last week I wrote about universally-known criticisms of economists, like “they’re silly for assuming everyone behaves perfectly rationally”:

My impression is that economists not only know about these criticisms, but invented them. During the last few paradigm shifts in economics, the new guard levied these complaints against the old guard, mostly won, and their arguments percolated down into the culture as The Correct Arguments To Use Against Economics. Now the new guard is doing their own thing – behavioral economics, experimental economics, economics of effective government intervention. The new paradigm probably has a lot of problems too, but it’s a pretty good bet that random people you stop on the street aren’t going to know about them.

The same pattern explains a lot of my concerns above. I knew some criticisms of a scientific paradigm. They seemed right. I concluded that scientists weren’t very smart and maybe I was smarter. I should have concluded that some cutting-edge scientists were making good criticisms of an old paradigm. I can still flatter myself by saying that it’s no small achievement to recognize a new paradigm early and bet on the winning horse. But the pattern I was seeing was part of the process of science, not a condemnation of it.

Most people understand this intuitively about past paradigm shifts. When a creationist says that we can’t trust science because it used to believe in phlogiston and now it believes in combustion, we correctly respond that this is exactly why we can trust science. But this lesson doesn’t always generalize when you’re in the middle of a paradigm shift right now and having trouble seeing the other side.

I realize I’m (ironically) risking making my narrative of scientific success unfalsifiable. Suppose someone wants to argue that scientific consensus is wrong. If they point to something it used to be wrong about, I can respond “Yes, but it self-corrected and it’s correct now, so that’s fine.” If they point to something where cutting-edge scientists say it’s wrong but nobody else agrees, I can respond “Yes, this is what the beginning of a paradigm shift looks like, so that’s fine”. And if they point to something where nobody in the field thinks it’s wrong, I can say “You’re a crackpot for going against all reputable scientists; the problem is with you.” And if later they turn out to be right, and everyone acknowledges it, I can say “Yes, but it self-corrected and it’s correct now, so that’s fine.”

(and I’m making it even easier for myself in that I say “scientific consensus for” when I probably mean “no scientific consensus against”. I don’t claim that 90%+ of scientists always believe true things, only that there are very few cases where 90%+ of scientists believe things which smarter people know to be false.)

Against this I can only offer a personal narrative: the only light I have by which to judge scientific consensus is my own Inside View assessment of what seems correct. Again and again I have tried to defy scientific consensus. And every time, I either find that I am wrong, find that I am a few years ahead of a trend that most scientists eventually agree with, or find that what I thought was “scientific consensus” was actually a fiction peddled by biased industry or media sources slandering a scientific community which actually had a much more sophisticated picture. My history of trying to fight scientific consensus has been a Man Who Was Thursday-esque series of embarassments as I find again and again that my supposed enemy agrees with me and is even better at what I am trying to do than I am.

Scientific consensus hasn’t just been accurate, it’s been unreasonably accurate. Humans are fallible beings. They are not known for their ability the change their mind, to willingly accept new information, or to put truth-seeking above political squabbles. And our modern society is not exactly known for being an apolitical philosopher-kingdom with strong truth-seeking institutions completely immune from partisan pressure. I feel a deep temptation to sympathize with global warming denialists who worry that the climatological consensus is biased politicized crap, because that is exactly the sort of thing which I would expect to come out of our biased politicized crappy society. Yet again and again I have seen examples of scientific fields that have maintained strong commitments to the truth in the face of pressure that would shatter any lesser institution. I’ve seen fields where people believe incredibly-bizarre sounding things that will get them mocked at cocktail parties just because those things seem to be backed by the majority of the evidence. I’ve even seen people change their minds, in spite of all the incentives to the contrary. I can’t explain this. The idea that scientific consensus is almost always an accurate reflection of the best knowledge we have at the time seems even more flabbergasting than any particular idea that scientists might or might not believe. But it seems to be true.

(note that I’m talking about “scientific consensus” to mean a very high-level pattern, consisting of hundreds of scientists over the space of decades evaluating a broad body of work. Any individual study is still probably total garbage.)

Given how weird all of this is, I realize there’s another possible bias here that should be taken very seriously – which is that I’m wrong about one or both sides of this. Which is more likely: that Science always agrees with Truth? Or that one guy’s perception of Science always agrees with that same guy’s perception of Truth? The latter gives me two degrees of freedom: I can either cherry-pick experts who agree with me and declare them to be Consensus, or I can conform my opinions to consensus so slavishly that I end up discovering only that Consensus agrees with itself. I don’t feel like I’m making this kind of mistake. But then again, nobody ever feels like they’re being biased.

But if I’m making this mistake, I think it’s at least a better mistake than the one where people dream up stories about being mavericks persecuted by hidebound reactionaries. This mistake at least sets the terms of debate as “let’s try to ascertain what the scientific community thinks” and forbids me from believing completely crackpottish things. And it encourages trust in one of our more trustworthy public institutions, always a prosocial sort of thing to do. I would rather have a world of people debating who agrees with scientific consensus or not, than a world of people debating whether scientific consensus is even valuable.

There are two caveats to the above. First, I think it’s dangerous to promote a norm of agreeing with scientific consensus, insofar as that helps encourage exactly the mistakes about the nature of consensus that I discussed above. When poorly-informed diet industry gurus support the Bad Old Paradigm, their rallying cry is usually “You’re a stupid crackpot, bow to the scientific consensus which agrees with me”. I gave three examples above of cases where I would have gotten the scientific consensus 100% wrong if I didn’t have access to a formal survey of scientific experts. In a world where these surveys had never been done – or some existing field without these surveys – or some field where these surveys have been done inaccurately or in a biased manner – people will often believe the consensus to be the opposite of what it really is. In those cases, demands that people respect consensus can be used to shut down people who are actually right – the field-wide equivalent of calling true facts you don’t like debunked and well-refuted. I see this happening all the time and I worry that waxing too poetically about the unreasonable effectiveness of scientific consensus will only serve to empower these people. Goodhart’s Law says that a measure which becomes a target ceases to be a useful measure, so we should be reluctant to target scientific consensus too strongly.

And second, I think that even when the Outside View tells you that the consensus is correct, you should continue pursuing your Inside View hunch that it isn’t. This avoids awkward situations like every individual scientist doubting the consensus, but suppressing their doubts because the “scientific consensus” has to be right.

So maybe the things I’m saying about scientific consensus aren’t very actionable. But respecting scientific consensus in a non-actionable way is a lot less exhausting than believing yourself to be against it, and talking about how you’re against it, and taking flak for being against it. And in the same way it’s helpful to believe that God is good, even if He never really gets around to doing much about it, so it’s reassuring to be able to have faith in our institutions every so often.

1 comments

Comments sorted by top scores.