The Substitution Principle

Good post, and upvoted, but I would phrase this part differently:

If a satisfactory answer to a hard question is not found quickly, System 1 will find a related question that is easier and will answer it.

The problem with a question like "What jobs make the most money" isn't so much that it's hard as that it's vague (or if you want to be harsh, "meaningless"). The question "How much would you contribute to save an endangered species" is even worse - if I were to actually answer it (by, for example, saying "Exactly two hundred seven dollars!"), you would be terribly confused and have no idea what I meant.

Seems to be a social norm that anyone asking an interlocutor to clarify a question is nitpicking and annoying, even though the overwhelming majority of questions people debate are meaningless as asked. People get rejected as poor conversational partners if they ask "To save all individuals in the species, or just to ensure at least one breeding pair...and are we talking per year, or pretending we have a 100% chance of saving them forever?", whereas if they say "We should pay whatever it takes!" they will be considered interesting even though that answer is clearly insane. It's no wonder that most people avoid becoming rationalists in such a situation.

This post gives what could be called an "epistemic Hansonian explanation". A normal ("instrumental") Hansonian explanation treats humans as agents that possess hidden goals, whose actions follow closely from those goals, and explains their actual actions in terms of these hypothetical goals. People don't respond to easily available information about quality of healthcare, but (hypothetically) do respond to information about how prestigious a hospital is. Which goal does this behavior optimize for? Affiliation with prestigious institutions, apparently. Therefore, humans don't really care about health, they care about prestige instead. As Anna's recent post discusses, the problem with this explanation is that human behavior doesn't closely follow any coherent goals at all, so even if we posit that humans have goals, these goals can't be found by asking "What goals does the behavior optimize?"

Similarly in this instance, when you ask humans a question, you get an answer. Answers to the question "How happy are you with your life these days?" are (hypothetically) best explained by respondents' current mood. Which question are the responses good answers for? The question about the current mood. Therefore, the respondents don't really answer the question about their average happiness, they answer the question about their current mood instead.

The problem with these explanations seems to be the same: we try to fit the behavior (actions and responses to questions both) to the idea of humans as agents, whose behavior closely optimizes the goals they really pursue, and whose answers closely answer the questions they really consider. But there seems to be no reality to the (coherent) goals and beliefs (or questions one actually considers) that fall out of a descriptive model of humans as agents, even if there are coherent goals and beliefs somewhere, too loosely connected to actions and anticipations to be apparent in them.

This seems to be something of a fake explanation. The statement is: Sometimes the brain provides false information, that actually answers a different question to the one asked.

That would be true no matter what answer was being given (unless it was completely random), because so long as the answer thrown up correlates with something, you can say: "Aha! The brain has substituted something for the question you thought was being asked!"

And since this explanation could be given for any answer the brain throws up, it doesn't actually give us any new information about the cognitive algorithm being used.

Ok brain, I'm going to ask you a question. I'm not going to say what it is yet. But before you answer, I want you to promise not to use System 1! Got it? Ok here goes: what career should I choose?

"Academia"

No you idiot, that was too fast!

You are making many unanalyzed assumptions here.

1) You are assuming that your mind did or did not do certain things in those moments when it was quietly answering the question. In particular, you make assumptions like "....your answer probably didn't take any of the above complications into account. It's as if your brain, while generating an answer, never even considered them .....". Why are you so sure that it probably didn't take any of the above considerations into account?

To illustrate how wrong this might be, consider that when a cognitive psychologist gives someone a visual priming task, with (for example) masked cues, the subject reports that she did not take ANY account of the masked cues. And yet, she shows clear evidence that she did very much take the cue into account! The proof is right there in the reaction times (which depend on what was in the cue).

So if someone can be that wrong in their self-report assessment of what factors they are taking into account in a situation as siimple as masked priming, what is the chance that a person in one of the scenarios you describe above is also doing all kinds of assessments that actually happen below the reporting threshold? At the very least this seems likely. But even if you don't accept that it is likely, you still have to give reasons why we should believe that it is not happening.

So, when Kahneman cites substitutions, his evidence clearly distinguishes substitutions from complex assessments that may be interpreted as substitutions, or which are correlated with substitutions? I don't buy that.

My second objection has to do with the oversimplification of the analysis:

2) You seem to be framing a lot of scenarios as if they were all instances of the same type of problem. As if the same mechanism was operating in most or all of these circumstances. Your mechanism involves a "target question", a "substituted question" (assumed to be of dubious validity) and a resulting answer that is assumed to be of sub-optimal quality. While there may be some situations where this frame neatly applies to a situation, I do not believe that it applies to all, and nor do I believe that it helps to try to oversimplify all instances of "bias" so they can be squeezed into this narrow frame.

At the very least, there appear to be situations that do not fit the pattern. Chess skill, for example. The question asked by the chess player is "How do I take the opponent's King?". But rather than address this question directly (as I did, in my very first chess game, when I imagined a sequence of moves that culminated in me taking that King, then started executing my planned sequence of moves .....), the expert chess player knows that a different set of questions have to be asked: to wit, "How do I make pleasing, coherent patterns of support and strength on the board?" and "Do I recognize anything about the current pattern as similar or identical to a situation I have seen in the past?"

That particular "substitution" happens to be extremely optimal. It also happens to be not how chess computers work (by and large: let's not get sidetracked by the finer points of chess programming ... the fact is that machines rely on depth to a very large extent, whereas humans rely on pattern). So it makes no sense to talk about substitution as a "problem" in this case. Far from it, substitution seems to be why a tiny little lookahead device (human mind) can give the massive supercomputer a run for its money.

3) Finally, this analysis overall has the feel (like almost all "human bias is a problem" arguments) of fitting the data to the theory. So many people (here on LW, and in the biasses community generally) want to see "Biasses" as a big deal, that they see them everywhere. All the evidence that you see supports the idea that the concept of "biasses and heuristics" is a meaningful one, and that there are many instances of that concept, and that thos instances have certain ramifications.

But, like people who see images of Jesus in jars of Marmite, or evidence of divine intervention whenever someone recovers from an illness, I think that you see evidence of bias (and, in this case, substitution) primarily because you have trained yourself to theorize about the world that way. Not because it is all real.

It seems to me like the substitution principle might be an explanation for the power of framing effects. You get a different answer for a question depending on what question you end up substituting for System 1 to answer.

That might explain the framing effect Eliezer noticed in Circular Altruism. The first time the question was asked it triggered System 1 to ask "How much would it suck if that gamble in option 2 didn't pay off?" When the question was reframed it triggered System 1 to ask "How much does it suck that 100 people die with certainty?"

The probably explains why political activists like to fight over framing effects so much. When people are asked the question "Would it be good for the government to do this?" System 1 spits out "Would it be good for my favorite metaphor for the government to do something vaguely analogous." What specific metaphor System 1 gets fed would obviously have huge power in framing the question.

This involves three skills: first recognizing a problem as a difficult one, then figuring out what heuristic you might have used, and finally coming up with a better solution.

Figuring out the particular heuristic seems more interesting than useful - "don't trust immediate answers" is a good rule, but it seems easier to start again than to de-bias an immediate guess (i.e. look up average salaries instead of trying to figure out how much you need to discount your guess of doctors' average salaries.)

What are the best careers for making a lot of money?

Here is a fantastic chart showing the professions of the top 1%.

A classic example of this, which I learned in a middle school class, is that when you go grocery shopping, you ask your brain, "How much food do I need in the next two weeks?" and it returns the value for how hungry you are right then. So to prevent overspending, don't go grocery shopping while hungry.

That's one of the problems that disappear pretty much completely if you consistently apply the Outside View to everything.

Another post on attribute substitution is How You Make Judgments: The Elephant and its Rider.

The Planning Fallacy: ”How much time will this take” becomes something like ”How much time did it take for me to get this far, and many times should that be multiplied to get to completion.”

More like “how much stuff I have left to do divided by how much did I do today”; even ”How much time did it take for me to get this far, and many times should that be multiplied to get to completion” wouldn't be that bad, because that would mean I take in account the time I have procrastinated (or have been hindered) thus far and expect to procrastinate (or be hindered) for roughly the same fraction of the time.

That's an interesting concept, with the particular advantage that it avoids the "evolved for the ancestral environment" just-so story.

I especially like the unpacking of various specific biases. How do you think the in-group bias would be unpacked?

All the examples Kahneman gives, I do not seem to substitute for the questions he lists (or worse ones), for even a moment.

This is the first time in a while I've felt immune to a cognitive bias.

What am I doing wrong? Does my introspection not go deep enough?

Maybe I really have read enough trick questions on the internet that I (what part of me?) immediately tries to tackle the hard problem (at least when reading a Less Wrong article which just told me exactly which mistake I'm expected to make).

I have an impression I've fallen for problems of this type before, when they were given for instrumental rather than epistemic reasons. But I can't remember any examples, and I don't know frequently it happens.

Forgive me if I'm not really getting it - which is entirely possible.

But it seems to me that "the substitution principle" is not really an explanation. All it seems to be saying is that it's possible to express biases, rewrite their descriptions, in that particular form. It doesn't describe a mechanism, but is more of a semantic trick, like you've bounced off a problem. I can't possibly imagine making useful predictions based on it.

EDIT: Ezekiel possibly said it better.

My immediate thought was "Do you mean the best for reliably becoming affluent, or the ones that have the potential to make you the richest?"

I think I have a pretty general tendency to stay on system 2 reasoning when asked hard to answer questions by reflexively looking for ways to clarify rather than reframe the question. I don't know if this is something I ever went through a learning process for.

I often find myself getting stuck on the "come up with a better solution" step though. Since I didn't generate another answer in the first place, "How should financial advisers who prey on the elderly be punished?" becomes "How much do I care about working out a fair, effectively preventative punishment for financial advisers who prey on the elderly? I could be at this for a while."

At least coming up with an answer of "I don't care enough about this/have a good reason to figure this out properly" gives you useful information on your own motives and the gaps in your knowledge.

The Conjunction Fallacy: ”What's the probability that Linda is a feminist” becomes ”how representative is Linda of my conception of feminists”.

I think this is more precisely an example of the Represenativeness Heuristic, though the point about substitution still stands.

How popular will the president be six months from now? becomes How popular is the president right now?

I wish people would quit using this as an example of a fallacy. With the gross uncertainties involved in predicting presidential popularity, "current popularity" is probably the best predictor available.

It seems that I don't do that. Like for example with the making money thing, my immediate thought was "banking is the fastest way to silly money, but you need deep math skills, and the burnout rate is harsh". Or the share of chores question: I immediately try to divide total chores by my effort. I seem to be caching the methodology.

I disagree with the notion that this 'quick and dirty answer' is even handled by a substantially different part of the brain in a substantially different fashion than a better answer. The better answer is perhaps made by several refinement steps each conducted in this exact same fashion as the quick and dirty answer.

Consider the job question with complications - the question of who we are talking about. Absent a perfect brain scan and repeated re-simulation of that person, the personal qualities are themselves heuristic answer to some kind of substitute question.

Likewise for the properties of the society and the like. Each single step in the chain of reasoning about that kind of items, even a very long and detailed chain, is some sort of quick conclusion based on some substitutions of this kind.

It is just the case of reasoning with the unknowns. If the personal qualities of "who" are an unknown, then what is the point of reasoning as if they were known? If the preference for a lot of money and low probability versus less money and higher probability is unknown as well? Absent statistics of the most common preference, it IS best answered by picking the most common profession that made someone rich, in the way that you (your only data point) prefer. It's hardly a case of substituting anything for anything else. It's a case of giving best possible answer when the information is incomplete.