Scholarship: how to tell good advice from bad advice?

When you receive advice, it must fall into one of these categories:

1. The advice is worthless or even outright bad.

2. The advice is valuable, but it's common knowledge, or at least can be obtained from public information in a straightforward way.

3. The advice is valuable, and it's not common knowledge, nor can it be obtained from public information in any straightforward way.

Now, if you believe you're receiving advice that is in category (3), you must ask yourself what makes you so special that you are privy to this information. This leads to the following heuristics:

• If the advice is in category (2), it is likely to be good if the source is reputable. For example, a book or website about programming written by a reputable author is likely to give you good advice on how to program. (But note that while it certainly adds value in terms of a convenient and attractive presentation, such a source doesn't give any significant information that wouldn't be available from other public sources.)

• If the advice purports to be in category (3), and yet the source of information is public (e.g. a book or website, even a non-free one), it's almost certainly bunk. The only exception is if the message is highly unpopular or counter-intuitive, and somehow you know that you have overcome biases that prevent most people from evaluating it correctly, which is very difficult and rare. For example, nearly any book that claims to bring special wisdom about investment, career, relationships, etc. is bunk.

• If the advice comes from a person on a private occasion, then there are several steps that you need to do. First, does this person show clear indications of the relevant knowledge and competence? If not, it's likely bunk for obvious reasons. Second, is the advice in category (2) or (3)? If it's (2), it's probably good, though it still pays off to check against other sources of information. If it purports to be (3), then you need to do the third, and most difficult evaluation: does this person have the motivation for an extraordinary degree of altruism towards you? If not, it's likely bunk, or otherwise they wouldn't grant you this privilege. If yes, for example if you're getting advice from your parents, then it is probably highly valuable.

One common failure mode is when people believe they're giving you advice of type (3), but in reality, their motivation for altruism towards you is weaker than their motivation for saying things that have high signaling value (and omitting things that have negative such value). This is one danger of socializing with people who are higher-status and more accomplished than you -- you'll be tempted to take their advice seriously, but in reality, even if they are giving it with good intentions, it's likely to be heavily censored and distorted so as to maximize its signaling value.

(This is exacerbated by the fact that good no-nonsense advice on topics that involve any aspects of human social behavior, both personal and professional, tends to sound crass, disreputable, cynical, or worse.)

The best single rule I have found is useful for academic disciplines as well: "Everybody overstates the accuracy, applicability, universality, precision, or other some other aspects of claims they have made." Anybody who actually cares enough about something to go through the trouble of writing it up, unless they are being paid by the word (which as you note has its own problems), is going to be emotionally invested in their work and writings. So my best advice is to assume from the beginning that it is overstated, and consider a weaker version, or a charitable interpretation, of their claims.

when you're researching a "how to" topic, how do you tell the good advice from the bad advice?

Before advice is tested, it may not be possible to know if the advice is good or bad. So I look for qualities that make advice testable.

Bad advice will cost more when it fails than pay when it succeeds. Bad advice will explain too much and thus be hard to measure if it's working or not ('prayer works, so if I just pray a little harder... '). Bad advice will involve people who aren't consenting or informed about what I'm doing. Bad advice is often a package deal ('if you're a feminist, you have to be a vegan'). Bad advice is unaccountable ('no, it's different when I do it' / 'I was only following orders').

Some of what is left over from that winnowing process will be good advice.

"I conceive of nothing, in religion, science, or philosophy, that is more than the proper thing to wear, for a while." - Charles Fort, 'Lo!' (1932)

So: when you're researching a "how to" topic, how do you tell the good advice from the bad advice?

Pretty much the same as in the sound academic fields, differing only in the amount of woo, bullshit, and self-interest you'll have to assess. Compare sources, look at results, look at reviews, look at the evidence, ask whether it even makes sense, try things out yourself, etc.

In the area of self-help (as some comedian has remarked, the idea of getting self-help from a book is already something of a contradiction) you also have to contend with the basic therapeutic conflict: anyone selling a cure for a disease has given themselves an interest in never curing it, but providing only palliatives. This is especially true when the disease is an imaginary one and their first task is to persuade you that you have it, which is why so many self-help books begin by telling the reader what a schmuck he or she is.

Here is an extreme example of how not to do it. Conversation I once overheard between a middle-aged man and woman:

The woman: talks about how convenient she is finding her new microwave oven.

The man: "Oh, I've heard you shouldn't use those, they destroy the vibrations in the food."

The woman: "Oh dear, I'd better stop using it then."

Ok, the man's talking woo, but that's not the point here. Substituting a topic more easily able to pass the rationalist filters:

The woman: talks about how wonderful she finds her new bread-maker.

The man: "Oh, I've heard you shouldn't eat bread, it wasn't present in the ancestral evolutionary environment."

The woman: "Oh dear, I'd better stop using it then."

Whatever the facts about eating grains, the epistemic process of transmission here is completely corrupt. So, don't just read a book and nod along to the author's argument, thinking "How true this is, I must do this."

For starters, I'd say it would be best to take advice from people whose careers and accomplishments are to some extent a matter of public record. Then you can evaluate (a) whether they seem to have actually accomplished the things they're trying to teach you to accomplish, and (b) whether they seem to have accomplished those things via the procedure they're encouraging you to follow. If yes to both, then you might proceed further.

In that case, the problem of making good advice seem too easy might come down to a couple of things. First, you want to see a good step-by-step procedure where you can really understand each step and imagine exactly what you'd have to do to achieve it. Second, it would be a red flag if any of those steps seem to be "magic" steps such as "Have a brilliant, lucrative idea for a business."

So: when you're researching a "how to" topic, how do you tell the good advice from the bad advice?

Find many credible sources of advice - preferably a combination of raw scientific studies and the applied practical advice of domain experts. Compare and contrast. Then try what remains and discard what doesn't work for you after a sufficient implementation period.

1. Start with minimal knowledge. Everything you read seems plausible.
2. Try stuff, keep track of your results. Form hypotheses, and seek new information being slightly more skeptical and discerning than before.
3. Repeat 2 until 99% of what you read is crap.
4. Follow the 1% advice, but you already know what you're doing by that point, anyway. Diminishing returns.

It's really hard to tell good from bad when you don't have the domain knowledge. You have to acquire a little bit by luck/accident, and then you start iterating.

This worked for me re: diet, meditation protocol, exercise protocol, time management, goal management, existential suffering, and much more. (Some are pretty steady state, others are still changing, and the steady state stuff may change if it stops working--which means my knowledge is incomplete, and more iteration is required.)

While it isn't always possible to obtain, advice that makes specific predictions is typically good. For instance, for fitness I've tried both Starting Strength (for exercise) and Lean Gains (for diet). They both made strong predictions about what would happen to you after doing their program for one to two months (Starting Strength: get substantially stronger; Lean Gains: lose a substantial amount of weight). For me personally, the Starting Strength predictions were accurate, but the Lean Gains one were not. So, I continue to do Starting Strength, but not Lean Gains.

In general, even if strong predictions are not made for you, you can translate advice into implicitly making such strong predictions. (This is, unfortunately, harder for things like planning for the future, when waiting to see if the predictions pan out is very expensive.)

I would go seriously mainstream (as in mainstream scientists) for such advice. Accomplished mainstream scientists. The advice will be boring: enroll in a course, maybe online course, do not skip homework, do not skip problems, do not skip test. The kind of stuff you probably already know.

If you are looking at some much easier way to become awesome competent, I am afraid you're asking for bad advice.

The Luke sure thinks that what he did was scholarship and that it benefitted him, but you should look for people who did objectively benefit from their studies, in a way not involving some sort of selling of ideas to the uninformed public. You need to somehow exclude as source of the advice the people who just babble of some advanced subjects they have incredibly poor understanding of, unaware of own incompetence.