How to Find Sources in an Unreliable World

I like to start by trying to find one author who has excellent thinking and see what they cite — this works for both papers and books with bibliographies, but increasingly other forms of media. 

For instance, Dan Carlin of the (exceptional and highly recommended) Hardcore History podcast cites all the sources he uses when he does a deep investigation of a historical era, which is a good jumping-off point if you want to go deep.

The hard part is finding that first excellent thinker, especially in a domain where you can't differentiate quality in a field yet. But there's some general conventions of how smart thinkers tend to write and reason that you can learn to spot. There's a certain amount of empathy, clarity, and — for lack of a better word — "good aesthetics" that, if they're present, the author tends to be smart and trustworthy. 

The opposite isn't necessarily the case — there are good thinkers who don't follow those practices and are hard to follow (say, Laozi or Wittgenstein maybe) — but when those factors are present, I tend to weight the thinking well.

Even if you have no technical background at all, this piece by Paul Graham looks credible (emphasis added) —

https://sep.yimg.com/ty/cdn/paulgraham/acl1.txt?t=1593689476&

"What does addn look like in C?  You just can't write it.

You might be wondering, when does one ever want to do things like this?  Programming languages teach you not to want what they cannot provide.  You have to think in a language to write programs in it, and it's hard to want something you can't describe.  When I first started writing programs-- in Basic-- I didn't miss recursion, because I didn't know there was such a thing.  I thought in Basic. I could only conceive of iterative algorithms, so why should I miss recursion?

If you don't miss lexical closures (which is what's being made in the preceding example), take it on faith, for the time being, that Lisp programmers use them all the time.  It would be hard to find a Common Lisp program of any length that did not take advantage of closures.  By page 112 you will be using them yourself."

When I spot that level of empathy/clarity/aesthetics, I think, "Ok, this person likely knows what they're talking about."

So, me, I start by looking for someone like Paul Graham or Ray Dalio or Dan Carlin, and then I look at who they cite and reference when I want to go deeper.

I wonder if a good pre-reading strategy is to search for, or ask experts about, the major controversies and challenges/issues related to the topic in question.

Your first step would be to try and understand what those controversies are, and the differences in philosophy or empirical evaluation that generate them. After you've understood what's controversial and why, you'll probably be in a better position to interpret anything you read on the subject.

One way you could potentially further your work on epistemic evaluation is to find or create a taxonomy of sources of epistemic uncertainty. Examples might include:

• Controversy (some questions have voluminous evidence, but it's either conflicting, or else various factions disagree on how to interpret or synthesize it).
• Lack of scholarship (some questions may have little evidence or only a handful of experts, so you have limited eyes on the problem)
• Lack of academic freedom (some questions may be so politicized that it's difficult or impossible for scholars to follow the evidence to its natural conclusion)
• Lack of reliable methods (some questions may be very difficult to answer via empirical or logical methods, so that the quality of the evidence is inevitably weak).

You can find papers addressing many of these issues with the right Google Scholar search. For example, searching for "controversies economic inequality" turns up a paper titled "Controversies about the Rise of American Inequality: A Survey." And searching for "methodological issues creativity" turns up "Methodological Issues in Measuring Creativity: A Systematic Literature Review."

My guess is that even just a few hours spent working on these meta-issues might pay big dividends in interpreting object-level answers to the research question.

I assume your two "N best books" examples are intended as bad examples. Since your other links are to good examples and the whole bullet-list block is introduced by offering "heuristics for finding good starting places", I think it would be worth making it even more explicit that they are intended as examples of what not to do (rather than e.g. a couple of rare counterexamples to the general pattern you've just mentioned).