What are the optimal biases to overcome?

Basically, the problem is that K&T-style insights about cognitive biases -- and, by extension, the whole OB/LW folklore that has arisen around them -- are useless for pretty much any question of practical importance. This is true both with regards to personal success and accomplishment (a.k.a. "instrumental rationality") and pure intellectual curiosity (a.k.a. "epistemic rationality").

From the point of view of a human being, the really important questions are worlds apart from anything touched by these neat academic categorizations of biases. Whom should I trust? What rules are safe to break? What rules am I in fact expected to break? When do social institutions work as advertised, and when is there in fact conniving and off-the-record tacit understanding that I'm unaware of? What do other people really think about me? For pretty much anything that really matters, the important biases are those that you have about questions of this sort -- and knowing about the artificial lab scenarios where anchoring, conjunction fallacies, etc. are observable won't give you any advantage there.

Note that this applies to your biases about abstract intellectual topics just as much as to your practical life. Whatever you know about any such topic, you know largely ad verecundiam from the intellectual authorities you trust, so that chances are you have inherited their biases wholesale. (An exception here is material that stands purely on rigorous internal logical evidence, like mathematical proofs, but there isn't much you can do with that beyond pure math.) And to answer the question of what biases might be distorting the output of the official intellectual authorities in the system you live under, you need to ask hard questions about human nature and behavior akin to the above listed ones, and accurately detect biases far more complex and difficult than anything within the reach of the simplistic behavioral economics.

Of course, the problem you ultimately run into is that such analysis, if done consistently and accurately, will produce results that clash with the social norms you live under. Which leads to the observation that some well-calibrated instinctive bias towards conformity is usually good for you.

Here are some tentative guesses about this whole rationality and success business.

Let's set aside "rationality" for a minute and talk about mental habits. Everyone seems to agree that having the right habits is key to success, perhaps most famously the author of 7 Habits of Highly Effective People. But if you look at the 7 habits the Covey identifies ("Be Proactive", "Begin with the End in Mind", "Put First Things First", "Think Win/Win", "Seek First to Understand, Then Be Understood", "Synergize", and "Sharpen the Saw") they don't look too much like what gets discussed on Less Wrong. So what gives?

I think part of the problem is the standard pattern-matching trap. Perhaps books like Covey's genuinely do address the factors that the vast majority of people need to work on in order to be more successful. But analytical folks tend not to read these books because

• they're part of a genre that's sullied its reputation by overpromising
• even when they don't overpromise, analytical people are rarely part of the target audience, and the books do things like give incorrect folk explanations for stuff that actually happens to work (but the analytical people don't try the stuff because they can tell the stated explanation for why it works is bogus)
• they tend to distrust their emotions, and a good part of how the books work, when they do, is by manipulating your emotions to elevate your mood and make it easier for you to get to work or implement changes
• analytical people typically manage to figure out stuff up to the level discussed in popular books for themselves, and don't do further careful study because they've written off the genre

So in the same way that pure math grad students are smarter than psychology grad students, even though good psychology research is probably higher-value than good pure math research, Less Wrong has focused on a particular set of mental habits that have the right set of superficial characteristics: mental habits related to figuring out what's true. But figuring out what's true isn't always that important for success. See Goals for which Less Wrong does (and doesn't) help. (Although the focus has gradually drifted towards more generally useful mental habits since the site's creation, I think.)

A big problem with addressing these more generally useful habits through the internet is that people who get good enough at applying them are liable to decide that surfing the internet is a waste of time and leave the conversation. I'm quite interested if anyone has any suggestions for dealing with this problem.

So when Holden Karnofsky says something like "rationality is a strong (though not perfect) predictor of success", maybe he is claiming that mental habits that make you better at figuring out what's true are actually quite useful in practice. (Or maybe by "rationality" he meains "instrumental rationality", in which case his statement would be true by definition.) Perhaps the reason Stephen Covey doesn't write about that stuff is because it's too advanced or controversial for him or his audience?

(Disclaimer: I haven't read The Seven Habits of Highly Effective People, although I did read the version for teenagers when I was a teenager.)

Eliezer noted (in a comment on a blog post I made about Nonprofit Kit For Dummies) that he did in fact buy the book and try to apply it. This suggests the difference was in fact Luke, and that we need How To Be Lukeprog For Dummies, which he is of course posting piecemeal ;-)

But if you look at the average person and ask why they aren't getting what they want, very rarely do you conclude the issue is that they're suffering from anchoring, framing effects, the planning fallacy, commitment bias, or any of the other stuff in the sequences.

I very often conclude that people are suffering from the planning fallacy.

I really don't like the word 'bias' especially in combination with 'overcoming'. It implies that there's the ideal answer being computed by your brain, but it has a bias added to it, which you can overcome to have a correct answer. Much more plausible is that you do not have the answer, and you substitute some more or less flawed heuristic. And if you just overcome this heuristic you will get dumber..

The difference in optimisation targets between LW and H&B researchers is an important thing to point out, and probably the main thing I'll take away from this post.

Biases can:-

• Be interesting to learn about
• Serve an academic/political purpose to research
• Give insight into the workings of human cognition
• Be fun to talk about
• Actually help to achieve your goals by understanding them

And the correlations between any 2 of these things need not be strong or positive.

Is it the halo effect if we assume that a more interesting bias will better help us achieve our goals?

No, it's because lukeprog did what seems like common sense: he bought a copy of Nonprofits for Dummies and did what it recommends.

There's a similar principle that I use sometimes when solving physics problems, and when building anything electronic. It's called "Do it the Right Way."

Most of the time, I take shortcuts. I try things that seem interesting. I want to rely on myself rather than on a manual. I don't want to make a list of things to do, but instead want to do things as I think of them.

This is usually fine - it's certainly fast when it works, and it's usually easy to check my answers. But as I was practicing physics problems with a friend, I realized that he was terrible at doing things my way. Instead, he did things the right way. He "used the manual." Made a mental list. Followed the list. Every time he made a suggestion, it was always the Right Way to do things.

With physics, these two approaches aren't all that far apart in terms of usefulness - though it's good to be able to do both. But if you want to do carpentry or build electronics, you have to be able to do things the Right Way.

Gerald Weinberg is a celebrated author of computer and management books. And for many years he was a management consultant. Often he would get a consulting gig at an organization he had consulted for in the past. The unhealthy organizations, he observed, had the same (crushing) worst problem during his second gig that they had during his first gig, whereas the better organizations tended to have lots of little problems, which he took as a sign that the organization was able to recognize their worst problems and slowly or quickly shrink them.

I am not sure because I do not have access to the book, but that is probably from the chapter or section "Rudy’s Law of Rutabagas" from Weinberg's book The Secrets of Consulting: A Guide to Giving and Getting Advice Successfully.

What Weinberg did after he stopping doing management consulting, by the way, is to run workshops on improving what we would call individual and team rationality, and he maintained that people learned the skills he taught a lot better in the right kind of interpersonal situations (e.g., workshops) than they do from written materials.

Hope that helps someone.

Use direct replies to this comment for suggesting things about tackling practical biases.

I think that you are using the word 'bias' somewhat idiosyncratically here, and that this might be causing some people to have a hard time understanding the main point of this post, which (if I may) I would summarize as follows:

Many people in this community seem to believe that, when we do not get what we want, this is primarily because we are afflicted by one or more cognitive biases, such as anchoring or scope insensitivity. But this is not so. The main source of practical irrationality is lack of certain practical skills or habits, like "figuring out what your goals really are" or "looking at your situation objectively and listing the biggest problems". What are the best ways to develop these skills?

As much as I think lukeprog - EY comparisons are informative, I wonder if the difference is just different amounts of energy. I hear that lukeprog is working 60 hour weeks and that EY had trouble being productive for more than 4 hours a day, and looking for citations I noticed this down the page.

That can't explain everything- there's another comment that comes to mind that I'm having difficulty finding, in which one of lukeprog's hacks dramatically increased EY's writing output- but it seems like it's part of a complete explanation.

My guess would be that risk analysis and mitigation would be one of the more useful positive techniques in practical rationality. I wish every organization with executive officers had a CRO (chief risk officer) position. Of course, a person like that would be highly unpopular, as they would be constantly asking some very hard questions. Imagine that it is you against Murphy. What can go wrong? What are the odds of its going wrong? What are the odds of you mis-estimating that it will go wrong? What has gone wrong in the past? What are the potential mitigation steps? What are the odds of the mitigation steps themselves going wrong? Basically, a CRO would ensure that an organization is (almost) never blindsided, except maybe for true black swans. Otherwise the most that can happen is "a failure mode described in has occurred, we should now review, possibly update and implement the risk mitigation steps outlined". The standard business plan is certainly not a substitute for something like that.

Most companies do not do nearly enough risk analysis and management, possibly because the CEOs are required to be optimistic, and neither the CEO nor the board are personally responsible for failures. The worst that can happen is that they are booted out and get a golden parachute.

My top 2....

Looking at unlikely happenings more sensibly. Remembering that whenever something really unlikely happens to you, it's not a sign from the heavens. I must remember to take into account the number of other unlikely things that might have happened instead that I would also have noticed, and the number of things that happen in a typical time. In a city of a million people, meeting a particular person might seem like a one in a million chance. But if I know a thousand people in the city, and walk past a thousand people in an hour, the chance of bumping into one of my friends is pretty good.

The other one? We're all too optimistic about our own abilities. Most of the time that's pretty benign, but it's a good thing to remember when considering employing yourself as a stock picker, gambling advisor, or automobile driver. We're actually much more average than we think, most of the time.

http://lesswrong.com/lw/ahz/cashing_out_cognitive_biases_as_behavior/ may be of relevance. The single strongest correlation with various unhappy behaviors or outcomes (the DOI) in Bruine de Bruine 2007 (weaker than the overall correlation with succumbing to various fallacies, though!) was 'Applying Decision Rules':

Applying Decision Rules asks participants to indicate, for hypothetical individual consumers using different decision rules, which of five DVD players they would buy (e.g., “Lisa wants the DVD player with the highest average rating across features,” describing an equal weights rule). Each consumer chooses from a different set of five equally priced DVD players with varying ratings of picture quality, sound quality, programming options, and brand reliability (from 1 [very low] to 5 [very high]). The decision rules are taken from Payne, Bettman, and Johnson (1993) and include elimination by aspects, satisficing, lexicographic, and equal weights rules. The present task uses more complex rules than the Y-DMC, which, in pretests, proved too easy for adults. Performance is measured by the percentage of items for which the correct DVD players are chosen, given the decision rule to be applied.

Seems somewhat reasonable to me - if you can't even shop well, you're probably going to overspend or buy unsatisfactory stuff or just junk, behavior which will cost you over your entire life.

As I recall, experiments show that people who learn about anchoring are still susceptible to anchoring, but people who learn about the sunk cost fallacy are less likely to throw good money after bad.

This obviously doesn't help right here and now, but I would like to point out that CfAR is in a good position to investigate this question experimentally. We'll have to wait awhile to be sure, but it looks like they have developed decent debiasing procedures and life outcome measures. I'm also guessing that they can't train people against every bias in a single retreat. Thus the can include different biases in different curriculums and compare their practical effects.

It's a good point that academics are likely to focus on those biases that are likely to be easy to prove, not those that are likely to be important to fix. But I'd expect the most important biases to also manifest themselves in a big way and in lots of different scenarios, and therefore be relatively easy to prove.

''"A large body of evidence[1][2][3][4][5][6][7][7][8][9][10] has established that a defining characteristic of cognitive biases is that they manifest automatically and unconsciously over a wide range of human reasoning, so even those aware of the existence of the phenomenon are unable to detect, let alone mitigate, their manifestation via awareness only."''

AFAIK, currently, none of them. The entire effort is futile and the introductory paragraph to Lesswrong appears self-defeating in light of this. I think there is far more too this place that cognitive bias mitigation.