Does blind review slow down science?

post by Kaj_Sotala · 2009-03-06T12:35:34.304Z · LW · GW · Legacy · 22 comments

Contents

23 comments

Previously, Robin Hanson pointed out that even if implementing anonymous peer review has an effect on the acceptance rate of different papers, this doesn't necessarily tell us the previous practice was biased. Yesterday, I ran across an interesting passage suggesting one way that anonymous review might actually be harmful:

 Second, fame may confer license. If a person has done valuable work in the past, this increases the probability that his current work is also valuable and induces the audience to suspend its disbelief. He can therefore afford to thumb his nose at the crowd. This is merely the obverse of the "shamelessness" of the old, which Aristotle discussed. Peter Messeri argues in this vein that "senior scientists are better situated than younger scientists to withstand adverse consequences of public advocacy of unpopular positions," and that this factor may explain why the tendency for older scientists to resist new theories is, in fact, weak. And remember Kenneth Dover's negative verdict on old age (chapter 5)? He offered one qualification: "There just aren't any [aspects of old age which compensate for its ills] - except, maybe, a complacent indifference to fashion, because people no longer seeking employment or promotion have less to fear."

This point suggests that the use by scholarly journals of blind refereeing is a mistaken policy. It may cause them to turn down unconventional work to which they would rightly have given the benefit of doubt had they known that the author was not a neophyte or eccentric.

 (From Richard A. Posner, "Aging and Old Age")

 If this hypothesis holds (and Posner admits it hasn't been tested, at least at the time of writing), then blind review may actually slow down the acceptance of theories which are radical but true. Looking up the Peter Messeri reference gave me the article "Age Differences in the Reception of New Scientific Theories: The Case of Plate Tectonics Theory". It notes:

Young scientists may adopt a new theory before their elders in part because they are better informed about current research in a broader range of fields. As Zuckerman and Merton observe with respect to life-course differences in opportunities for discovery, young scientists - closer in time to their formal training with an 'aggregate of specialists at work on [many] research front[s]' - are likely to be more up to date 'in a wider variety of fields than their older and more specialized' colleagues. Moreover, defects in existing knowledge may be more apparent to young scientists burdened with fewer preconceptions reinforced by prior practice.  Conversly, older scientists may take longer to be won over to a new theoretical perspective because of greater familiarity with past successes of established theory in overcoming previous theoretical or empirical challenges. Commitment to existing knowledge may also strenghten with age because older scientists have a greater social and cognitive investment in its perpetuation, and have less to gain from adopting new ideas. This may be particularly important when adoption of a new theory requires substantial effort to master new research skills and concepts.

In this light, an older scientist's acceptance of a new, radical hypothesis should tell us to give the new hypothesis extra weight. It might even be appropriate to apply a heavier "reputation weighting" on controversial theories than established ones - we'd expect the established scientist to write papers supporting the established theories, but not controversial ones. However, blind review makes this impossible. The reviewers may even mistake the established scientist as an undiscriminating free thinker, who endorses any controversial theories simply because they're unpopular. This will slow down the acceptance of hypotheses which are, in fact, correct.

A possible objection to this could be that old scientists are unlikely to change their minds, and therefore old scientists not getting enough credit for their achievements won't have much of an effect in the spread of new hypotheses. (After all, if no old scientist endorses controversial hypotheses, then it doesn't matter if those endorsements aren't properly weighted in review.) Not so. Messeri finds that science does actually progress a lot faster than "one funeral at a time" (as Max Planck put it), and old scientists are ready to adopt new theories given sufficient evidence. While the last holdouts for outdated theories do tend to be the old, their increased security also makes them the first who are willing to publicly support new theories:

 ...the episode which promted Planck's 'observation' ... seems like a poor illustration of the 'fact' Planck claims to have learned. ... Wilhelm Ostwald, one of the leaders of the opposition of 'Energetics' school prominently mentioned by Planck, was only five years older than Planck, whereas Ludwig Boltzmann, whose theoretical work on entropy, in no small measure ... helped bring the scientific community around to Planck's view, was fourteen years Planck's senior. ...

In a study of the Chemical Revolution, McCann reports a negative correlation between author's age and the use of the oxygen paradigm in scientific papers written between 1760 and 1795. On closer inspection of the data, he finds the earliest group of converts to the oxygen paradigm (between 1772 and 1777) were middle-aged men with close ties to Lavoisier; the inverse age effect became manifest only after 1785, during the ten-year period of 'major conversion and consolidation'. ...

As for evolutionary theory, Hull and his colleagues find weak support for 'Planck's Principle' among nineteenth-century British scientists. The small minority of scientists who held out against the theory after 1869 were, on average, almost ten years older than earlier adopters. Age in 1859 (the year the Origin of Species was published) was unrelated, however, to speed of acceptance for the great majority of those converting to evolutionary theory by 1869. ...

I examined the publications of ninety-six North American earth scientists actively engaged in pertinent research during the 1960s and 1970s. ... The dependent variable for this study is the year in which a scientist decided to adopt the mobilist programme of research rather than to continue working within a stabilist programme. ... Before 1966, when prevailing scientific opinion still ran strongly against the mobilist perspective, the small number of scientists adopting the programme were considerably older (in terms of career age) than other scientists active during this early period. Thus, scientists adopting the programme through 1963 were on average nineteen years 'older' than non-adopters. ... Adopters in 1964 were twenty-three years older than non-adopters. ... Only with the shift in scientific opinion favourable to mobilist concepts beginning in 1966, do we start to see a progressive narrowing, and then reversal, in the age differentials between adopters and non-adopters.

22 comments

Comments sorted by top scores.