Science advances one funeral at a time

post by Cameron Berg (cameron-berg), Judd Rosenblatt (judd), Diogo de Lucena (diogo-de-lucena), AE Studio (AEStudio) · 2024-11-01T23:06:19.381Z · LW · GW · 1 comments

Contents

1 comment

Major scientific institutions talk a big game about innovation, but the reality is that many of the mechanisms designed to ensure quality—peer review, funding decisions, the academic hierarchy—explicitly incentivize incremental rather than revolutionary progress.[1] 

Thomas Kuhn's now-famous notion of paradigm shifts was pointing at precisely this phenomenon. When scientists work within what Kuhn called "normal science," they're essentially solving low- to medium-stakes puzzles within their field's accepted framework. While it's fairly easy to evaluate the relative quality of work that occurs within any given paradigm, Kuhn argued it's nearly impossible for scientists to reason about the relative power of different paradigms for a given field—especially when they have already drank the paradigmatic kool-aid.

Max Planck captured this idea succinctly in his biting statement that "science advances one funeral at a time."[2]

There is no shortage of examples of this occurring throughout the history of science:

The takeaway here is quite relevant (and not all that unfamiliar [LW · GW]) for alignment research. The still-young field attracts brilliant people who want to help solve the problem—and, by virtue of their technical chops, also care about their career capital. In attempting to check both of these boxes, many naturally gravitate toward "safer," already-somewhat-established research areas. However, when we polled [LW · GW] these very researchers, most acknowledged they don't think these sorts of approaches will actually solve the core underlying problems in time. This seems quite familiar to the old story of incentives driving forward incremental work when what is desperately needed are breakthroughs.

The alignment innovations that will be most-critical-in-hindsight will have come from people who were willing to step outside the bounds, question the premises everyone took for granted, and pursue ideas that initially sounded ridiculous. 

Got a crazy hunch that doesn't fit nicely into the current alignment landscape? Come talk to us at EAG Boston—or apply to work on your idea with us here.

  1. ^

    This is not to say that incremental progress is unimportant or that revolutionary progress is all that matters—only that mainstream science is mostly in the business of operating under established paradigms rather than creating new ones.

  2. ^

    While this is the better-remembered variant, Planck's actual statement was "a new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents die, and a new generation grows up that is familiar with it." But this would have made for too long a title.

1 comments

Comments sorted by top scores.

comment by Dmitry Vaintrob (dmitry-vaintrob) · 2024-11-02T03:24:07.486Z · LW(p) · GW(p)

It's neat to remember stories like this, but I want to note that this shouldn't necessarily update scientists to criticize novel work less. If an immune system doesn't sometimes overreact, it's not doing its job right, and for every story like this there are multiple other stories of genuinely false exciting-sounding ideas that got shut down by experts (for instance I learned about Schekhtman from the Constant podcast, where his story was juxtaposed with that of genuine quacks). Looking back at my experience of excited claims that were generally dismissed by more skeptical experts in fields I was following, the majority of them (for instance the superluminal neutrino, the room-temperature superconductor, various hype about potentially proving the Riemann hypothesis by well-established mathematicians) have been false.

I think there is a separate phenomenon (which was the explanation for the study about funerals), that older high-status scientists in funding-hungry fields will often continue to get funding and set priorities after they have stopped working on genuinely exciting stuff -- whether because of age, because of age-related conservatism bias, or simply because their area of expertise has become too well-developed to generate new ideas. In my experience in math and physics, from inside the field, this phenomenon generally does not look like a consensus that only the established people know what's going on (as in most of the stories here), but either conversely a quiet consensus that so-and-so famous person is starting to go crazy, or alternatively the normal disagreement between more conservative and more innovation-minded people about the value of a new idea. For example the most exciting development in my professional life as a mathematician was Jacob Lurie's development of "higher category theory", a revolution that allowed algebraists to seamlessly use tools from topology. There were many haters of this theory (many very young), but there was enough of a diffuse understanding that this is exciting and potentially revolutionary that his ideas did percolate and end up converting many of the haters (similarly with Grothendieck and schemes). Note that here I think math avoids the worst aspects of these dynamics because it doesn't require funding and is less competitive.

The upshot here is that I think it's valuable to try to resolve the issue of good ideas being shot down by traditionalists, but the solution might not be to "adopt lower standards for criticizing new / surprising ideas" but rather something more like pulling the rope sideways and looking for better standards that do better at separating promising innovation from hype.