How to Navigate Evaluating Politicized Research?

post by Davis_Kingsley · 2022-07-01T05:59:24.239Z · LW · GW · No comments

This is a question post.

Contents

  Answers
    6 localdeity
None
No comments

I want to look into the research literature on an issue that has unfortunately become extremely politicized and controversial. (It isn't gun ownership, but let's say for the sake of argument that it is.) If I do a search for, say, "effects of gun ownership on crime", it is quite predictable that I will find a bunch of studies that say gun ownership decreases crime and a bunch of studies that say gun ownership increases crime. Some of those studies may be legitimate efforts at truthseeking, while others may be cherrypicked or manipulated, studies where the authors' bottom line has already been written before they do any work (to one degree or another), etc.

This seems like a potentially difficult environment to evaluate things in compared to one with less of a charged atmosphere, and I'm curious how best to navigate it -- any recommendations for how to best do that?

Answers

answer by localdeity · 2022-07-01T07:03:11.780Z · LW(p) · GW(p)

Off the top of my head:

  • search for facts that all sides agree on
  • look for cases of someone on side X who admits a fact that side X doesn't like, and weight that more highly (though don't take any individual to be infallible)
    • note that you may find cases of a nominally pro-X person who works for an institution that's nominally neutral but in practice anti-X, and in that case the person might be selected for being only barely pro-X, or for being willing to keep their mouth shut / parrot some things they don't believe
  • given some facts that you're convinced are true, evaluate sources according to their stances on those facts
  • generally evaluate how much the author seems to be motivated by e.g. "natural fascination with the subject", "hatred for the bad people", "taking pride in meticulously and unemotionally describing reality", "starting a political movement", etc., and how these would affect likelihood of truth-seeking vs nurturing political biases

The above is most suited for "nonfiction" books and newspaper-type articles.  I'm not very familiar with likely-biased research papers, except in the sense that most researchers express a bias towards thinking their research is important and valuable...

  • Look for overstatements, and cases where they interpret data the way they like and don't mention that it could have been interpreted less favorably.
    • If they volunteer counterarguments to their position, counterarguments that they're not able to immediately knock down, this seems like a good sign for truth-seeking.
  • As an advanced exercise, tag papers and authors as belonging to one side or another, and pay attention to how much an author approvingly cites papers written by the other side.

(Source: rank amateur.)

No comments

Comments sorted by top scores.