The Median Researcher Problem

I'm not sure median researcher is particularly important here, relatively to, say, median lab leader.

Median voter theorem works explicitly because votes of everyone are equal, but if you have lab/research group leader who disincentivizes bad research practices, then you theoretically should get lab with good research practices.

In practice, lab leaders are often people who Goodhart incentives, which results in current situation.

LessWrong has chance to be better exactly because it is outside of current system of perverse incentives. Although, it has its own bad incentives.

Show me a field where replication crises tear through, exposing fraud and rot and an emperor that never had any clothes, a field where replications fail so badly that they result in firings and polemics in the New York Times and destroyed careers- and then I will show you a field that is a little confused but has the spirit and will get there sooner or later.

What you really need to look out for are fields that could never, on a conceptual level, have a devastating replication crisis. Lesswrong sometimes strays a little close to this camp.

I am not sure about the median researcher. Many fields have a few "big names" that everybody knows and who's opinions have disproportionate weight.

community norms which require basically everyone to be familiar with statistics and economics,

I think this is too strong. There are quite a few posts that don't require knowledge of either one to write, read, or comment on. I'm certain that one could easily accumulate lots of karma and become a well-respected poster without knowing either.

Could you please elaborate on what you mean by "highly memetic" and "internal memetic selection pressures"? I'm probably not the right audience for this piece, but that particular word (memetic) is making it difficult for me to get to grips with the post as a whole. I'm confused if you mean there is a high degree of uncritical mimicry, or if you're making some analogy to 'genetic' (and what that analogy is...)

What would this say about subculture gatekeeping? About immigration policy?

I think one thing that's missing here is that you're making a first-order linear approximation of "research" as just continually improving in some direction. I would instead propose a quadratic model where there is some maximal mode of activity in the world, but this mode can face certain obstacles that people can remove. Research progress is what happens when there's an interface for removing obstacles that people are gradually developing familiarity with (for instance because it's a newly developed interface).

Different people have different speeds by which they reach the equillibrium, but generally those who have an advantage would also exhibit an explosion of skills and production as they use their superior understanding of the interface.

A small research community of unusually smart/competent/well-informed people can relatively-easily outperform a whole field, by having better internal memetic selection pressures.

It's not obvious to me that this is true, except insofar as a small research community can be so unusually smart/competent/etc that their median researcher is better than a whole field's median researcher so they get better selection pressure "for free". But if an idea's popularity in a wide field is determined mainly by its appeal to the median researcher, I would naturally expect its popularity in a small community to be determined mainly by its appeal to the median community member.

This claim looks like it's implying that research communities can build better-than-median selection pressures but, can they? And if so why have we hypothesized that scientific fields don't?

People did in fact try to sound the alarm about poor statistical practices well before the replication crisis, and yet practices did not change, 

This rings painfully true. As early as the late 1950s, at least one person was already raising a red flag about the risks that psychology^[1] might veer into publishing a sea of false claims:

There is some evidence that in fields where statistical tests of significance are commonly used, research which yields nonsignificant results is not published. Such research being unknown to other investigators may be repeated independently until eventually by chance a significant result occurs—an 'error of the first kind'—and is published. Significant results published in these fields are seldom verified by independent replication. The possibility thus arises that the literature of such a field consists in substantial part of false conclusions resulting from errors of the first kind in statistical tests of significance.

1. ^{^}

   Sterling isn't explicitly talking about psychology, but rather any field where significance tests are used.

How do you think competent people can solve this problem within their own fields of expertise? 

For example, the EA community is a small & effective community like you've referenced for commonplace charity/altruism practices. 

How could we solve the median researcher problem & improve the efficacy & reputation of altruism as a whole?

Personally, I suggest taking a marketing approach. If we endeavor to understand important similarities between "median researchers", so that we can talk to them in the language they want to hear, we may be able to attract attention from the broader altruism community which can eventually be leveraged to place EA in a position of authority or expertise.

What do you think?

Consistency in research fields and protection against elementary methodological malpractices such as p-hacking and the like should be enforced through automation. IQ over median does not correlate with creativity over median, as indicated by recent research, so i wouldn't worry too much about this side of your argument. I think future research in general has to contemplate what is the best way to harvest human creativity while ensuring that consistency, novelty and methodological robustness are enforced through automation.