A Taxonomy of Bias: The Cognitive Miser

post by Kaj_Sotala · 2010-07-02T18:38:08.995Z · LW · GW · Legacy · 38 comments

Contents

  The Cognitive Miser
  Default to the Autonomous Mind
  Serial Associative Cognition with a Focal Bias
  Override Failure
None
38 comments

This is the second part in a mini-sequence [? · GW] presenting content from Keith E. Stanovich's excellent book What Intelligence Tests Miss: The psychology of rational thought. It will culminate in a review of the book itself.

Noting that there are many different kinds of bias, Keith Stanovich proposes a classification scheme for bias that has two primary categories: the Cognitive Miser, and Mindware Problems. Today, I will discuss the Cognitive Miser category, which has the subcategories of Default to the Autonomous Mind, Serial Associative Cognition with a Focal Bias, and Override Failure.

The Cognitive Miser

Cognitive science suggests that our brains use two different kinds of systems for reasoning: Type 1 and Type 2. Type 1 is quick, dirty and parallel, and requires little energy. Type 2 is energy-consuming, slow and serial. Because Type 2 processing is expensive and can only work on one or at most a couple of things at a time, humans have evolved to default to Type 1 processing whenever possible. We are "cognitive misers" - we avoid unnecessarily spending Type 2 cognitive resources and prefer to use Type 1 heuristics, even though this might be harmful in a modern-day environment.

Stanovich further subdivides Type 2 processing into what he calls the algorithmic mind and the reflective mind. He argues that the reason why high-IQ people can fall prey to bias almost as easily as low-IQ people is that intelligence tests measure the effectiveness of the algorithmic mind, whereas many reasons for bias can be found in the reflective mind. An important function of the algorithmic mind is to carry out cognitive decoupling - to create copies of our mental representations about things, so that the copies can be used in simulations without affecting the original representations. For instance, a person wondering how to get a fruit down from a high tree will imagine various ways of getting to the fruit, and by doing so he operates on a mental concept that has been copied and decoupled from the concept of the actual fruit. Even when he imagines the things he might do to the fruit, he never confuses the fruit he has imagined in his mind with the fruit that's still hanging in the tree (the two concepts are decoupled). If he did, he might end up believing that he could get the fruit down by simply imagining himself taking it down. High performance on IQ tests indicates an advanced ability for cognitive decoupling.

In contrast, the reflective mind embodies various higher-level goals as well as thinking dispositions. Various psychological tests of thinking dispositions measure things such as the tendency to collect information before making up one's mind, the tendency to seek various points of view before coming to a conclusion, the disposition to think extensively about a problem before responding, the tendency to calibrate the degree of strength of one's opinion to the degree of evidence available, the tendency to think about future consequences before taking action, the tendency to explicitly weigh pluses and minuses of situations before making a decision, and the tendency to seek nuance and avoid absolutism. All things being equal, a high-IQ person would have a better chance of avoiding bias if they stopped to think things through, but a higher algorithmic efficiency doesn't help them if it's not in their nature to ever bother doing so. In tests of rational thinking where the subjects are explicitly instructed to consider the issue in a detached and objective manner, there's a correlation of .3 - .4 between IQ and test performance. But if such instructions are not given, and people are free to reason in a biased or unbiased way as they wish (like in real life), the correlation between IQ and rationality falls to nearly zero!

Modeling the mind purely in terms of Type 1 and Type 2 systems would do a poor job of explaining the question of why intelligent people only do better at good thinking if you tell them in advance what "good thinking" is. It is much better explained by a three-level model where the reflective mind may choose to "make a function call" to the algorithmic mind, which in turn will attempt to override the autonomous Type 1 processes. A failure of rationality may happen either if the reflective mind fails to activate the algorithmic mind, or if the algorithmic mind fails to override the autonomous mind. This gives us a three-level classification of this kind of bias.

Default to the Autonomous Mind

Defaulting to the autonomous mind is the most shallow kind of thought, where no Type 2 processing is done at all. The reflective mind fails to react and activate the algorithmic mind. Stanovich considers biases such as impulsively associative thinking and affect substitution (evaluating something primarily based on its affect) to be caused by this one.

Serial Associative Cognition with a Focal Bias

In this mode of thinking, Type 2 processes are engaged, but they are too conservative in their use of resources. For instance, consider the following problem (answer in rot13 below):

Jack is looking at Anne but Anne is looking at George. Jack is married but George is not. Is a married person looking at an unmarried person? A) Yes B) No C) Cannot be determined.

Gur pbeerpg nafjre, juvpu yrff guna 20 creprag bs crbcyr trg, vf N. Vs Naar vf zneevrq, gura gur nafjre vf "Lrf", orpnhfr fur'f ybbxvat ng Trbetr jub'f hazneevrq. Vs Naar vf hazneevrq, gura gur nafjre vf "Lrf" orpnhfr fur'f orvat ybbxrq ng ol Wnpx, jub'f zneevrq.

In this example, people frequently concentrate too much on a single detail and get the answer wrong. There are numerous biases of similar kind. For instance, when asked to guess the amount of murders in Detroit (which is located in Michigan) they give a higher number than when asked to guess the number of murders in Michigan. This is because people are using crude affect-laden images of the locations in question to generate their guess. Vividness, salience and accessibility of various pieces of information have an overly strong effect to our thinking, becoming the main focal point of our evaluation. Focal bias is also involved biases such as framing effects (the presented frame is taken as focal), the Wason selection task, motivated cognition, and confirmation bias.

Override Failure

In an override failure, Type 2 processes notice that Type 1 systems are attempting to apply rules or heuristics that are not applicable to the situation at hand. As a result, the Type 2 processes attempt to initiate an override and take the Type 1 systems offline, but for whatever reason they fail to do so. Override failures can be divided into two categories: "cold" and "hot" ones.

• Premise: All living things need water
• Premise: Roses need water
• Conclusion: Roses are living things

The above reasoning is invalid ("Living things" implies "need water", but "need water" does not imply "living thing"), but many people will instinctively accept it, because the conclusion is a true one. It's an example of a cold override, where you need to override a natural response with a rule-based one. In another example, test subjects were presented with two cans of jelly beans. One of the cans had nine white jelly beans and one red jelly bean. The other had eight red jelly beans and ninety-two white jelly beans. The subjects were told to pick one of the cans and then draw a jelly bean at random from their chosen can: if they got a red one, they'd win a dollar. Most picked the can with one red jelly bean (a 10% chance) but 30 to 40 percent of the subjects picked the one with the worse (8%) odds. Many of them knew that they were doing a mistake, but having a higher absolute amount of beans was too enticing to them. One commented afterwards: "I picked the one with more red jelly beans because it looked like there were more ways to get a winner, even though I knew there were also more whites, and that the percents were against me."

A "hot" override, on the other hand, is one where strong emotions are involved. In what's likely to be a somewhat controversial example around here, Stanovich discusses the trolley problem. He notes that most people would choose to flip the switch sending the trolley to the track where it kills one person instead of five, but that most people would also say "no" to pushing the fat man on the tracks. He notes that this kind of a scenario feels more "yucky". Brain scans of people being presented various variations of this dilemma show more emotional activity in the more personal variations. The people answering "yes" to the "fat man"-type dilemmas took a longer time to answer, and scans of their brain indicated activity in the regions associated with overriding the emotional brain. They were using Type 2 processing to override the effects of Type 1 emotions.

Stanovich identifies denominator neglect (the jelly bean problem), belief bias effects ("roses are living things"), self-control problems such as the inability to delay gratification, as well as moral judgement failures as being caused by an override failure.

38 comments

Comments sorted by top scores.