Statistical checks on some social science

Really good article.

There may be several problems here. These are my guesses, without special/private data to back them up. I'll use psychology as the name of the field, but this isn't specific to psychology.

1. Lots of people know statistics, but to criticize the statistics of psychology articles, one would be well served to also be familiar with the subject matter of the field. A pure statistician criticizing psychologists, looking in from the outside as it were, would probably come across as arrogant or offensive, and wouldn't be listened to by psychologists much. Whereas if the statistician is also a psychologist, perhaps primarily a psychologist but also a good statistician, they don't really want to make a career out of criticizing and disproving other psychologists.

2. When you point out a problem with someone else's article, your criticism is judged by third parties in the field. If the problems relate to psychology itself, then presumably other psychologists will correctly evaluate your criticism. But if the problems relate to something external like statistics - and we stipulate here that most psychologists don't understand these problems themselves, because we're talking about common problems - then they won't be able to judge your criticism on its own terms.

   Because people must still reply to officially published criticism, they will be forced to say explicitly that they disagree. And because they will be unable to justify their disagreement on technical statistical grounds, they will rationalize it as politics or a status game. So the likely result would be that people dislike you personally, and by extension your theories, for producing criticism that they don't understand, or that they can't disprove but also can't admit as valid.

   Ideal scientists accept valid criticism, change their minds, and say they were wrong. But even ideal scientists can't do so if they don't really understand the criticism. They are using statistics incorrectly, because they were taught to do so by authority figures or by imitating authoritative works, and only another authoritative figure can effectively tell them they are wrong.

3. Once this situation exists it is self perpetuating. People who understand and care about statistics will observe the state of the field, and will tend not to become psychologists. People who want to reference widely accepted articles and theories, in order to advance their careers, will thereby absorb some wrong statistics and replicate them in their own work.