SotW: Avoid Motivated Cognition

This often works for me: think of some smart, critical and rational friend that you have, and then imagine/visualize presenting your argument to them. Or suppose that you just put up your argument on the Internet to be critiqued. Is any part of your reasoning such that you'd actually prefer not to present it in public, knowing that it won't hold up to scrutiny?

For me at least, if I imagine presenting my reasoning to someone else, I suddenly become a lot more conscious about its weak spots. And then I try to remind myself that if I can't think of a reason why those weak spots should hold up to scrutiny, my instinct should be to abandon the argument instead of just hoping that those problems won't come up.

"Well," says Lucy, "I suppose it could have played a role in suggesting that I eat a whole chocolate cake, but the reason why I decided to do it was to support the sugar industry. Lots of people have jobs in the sugar industry, and they've been having some trouble lately."

The obvious response to Lucy is "Is buying and eating that whole choclate cake really the best way to help the sugar industry? How does it compare to other strategies, like directly giving them money?"

The generalization is that when you say you are taking action A in pursuit of goal G, ask what other actions might be more effective at achieving G.

Set your baseline to always assume you are rationalizing, and then have to prove that you aren't, rather than vice versa.

Something I do, that I'm surprised I don't see mentioned here, is to just assume that any point I am trying to make, or anything I think, is a rationalization (which is pretty likely).

So instead of having to think "Am I rationalizing?" (which can be hard to figure out), I change my baseline, to ALWAYS assume "I am probably rationalizing. How am I rationalizing?" and go from there. Sort of a quick run-through of what biases and semi-hidden desires could be influencing my decisions or statements at any given time. From there I can either accept or reject these "rationalizations".

This also ends up leading to many disclaimers in conversations, as mentioned in the OP. (i.e. "Well I can't know for sure what I HAD thought, because by now hindsight bias has taken hold...." or "Well, I'm completely anchored off your estimate now, but...") I see "Conversation Disclaimers" themselves to be a major skill. Maybe an exercise could be made out of that?

Quick idea: Have people in pairs have a conversation about a debatable subject. Every sentence they say has to be pre-empted with a disclaimer.

(Note: This is my immediate reaction to this post. I'll give it more thought later.)

Our conceptual understanding of 'motivated cognition', and why it's defective as a cognitive algorithm - the "Bottom Line" insight.

"Defective" isn't quite enough; you want a prescription to replace it with. Saying "this is a bad habit" seems less useful than saying "here is a good habit."

There are two obvious prescriptions I see: provide correct rationales for decisions, or do not provide rationales for decisions. Which prescription you shoot for has a radically different impact on what exercises you do, and so should be talked about quite a bit. It may be desirable to try and wipe out rationalization, first, and then progress to correct rationales.

One exercise might be asking "who will this convince?" and "whose desires do I want to maximize?". Lucy probably doesn't actually expect Marvin to be swayed by the plight of Big Sugar, and probably doesn't actually suspect that Marvin will believe she's motivated by the plight of Big Sugar, and so that deflection may be the wrong play because it's incredible.

It seems to me that social incentives will swallow most internal incentives here. If I can get more out of others by rationalizing, then it may be a losing move for me to not rationalize- and so it may be more profitable to focus specifically on internal desire-desire conflicts. If Marvin will buy the cake for Lucy if she gives Marvin-optimized reasons, then Lucy should seek to determine whether or not she wants the cake for Lucy-optimized reasons and then present the case to Marvin in terms of Marvin-optimized reasons.

When Lucy senses one desire to eat a whole chocolate cake, and comes up with the sugar industry reason, perhaps Lucy should ask which Lucy that represents (Altruist Lucy) and which other Lucys want to have a say on the issue. Thin Lucy and Cheap Lucy might both think that Lucy shouldn't buy the cake, and Sweet Tooth Lucy wants that cake.

And when Lucy simulates their internal discussion, she quickly realizes that Altruist Lucy doesn't actually care much about the cake issue, compared to the other three. If Altruist Lucy was fully modeled here, she'd probably side with Cheap Lucy (as those dollars can do more good elsewhere). And so the question is what tradeoff Lucy wants to make between the preferences of Thin Lucy, Cheap Lucy, and Sweet Tooth Lucy.

Notice that the rationalization is an explicit call for alliances or disguise in this model. Only three Lucys are really interested- and the weight is against the cake- but Sweet Tooth Lucy can call in other Lucys by constructing arguments that tangentially involve them. That should be a costly move- at the beginning of a decision, Lucy should determine which Lucys are most relevant to the decision, and then be skeptical of attempts to bring in other Lucys.

The first exercise would be labeling the desires involved in a decision. I suspect there will generally be at least three, but in some decisions one or two will dominate. It might be useful to start with decisions where one desire dominates, and then move to where two desires agree, and then three desires agree, and then start introducing conflicting desires.

Jack tripped, and is falling. He notices a desire to stop his fall.

Healthy Jack wants to not get hurt.

Jack tripped, and is falling, within sight of his girlfriend. He notices a desire to stop his fall.

Healthy Jack wants to not get hurt and Impressive Jack wants to not make a fool of himself. They agree on the recommended action.

On a lazy Saturday afternoon, Jack notices a desire to do a mildly dangerous trick in front of his girlfriend.

Impressive Jack wants to show off, and Healthy Jack wants to not get hurt. They disagree on the recommended action.

The second exercise would be declaring other desires as invalid (or possibly valid). This one seems like it could be done either as a worksheet - "does Cheap Jack have anything important to say about Jack tripping, conditioned on Healthy Jack and Impressive Jack already being in the discussion?" - or better yet, socially, in which someone describes a recent decision they faced, which three desires they thought were the most important, and then their partner / other members of the group seek to argue for the inclusion of other desires. It's not yet clear how to get a good balance of suggestions that should be shot down and suggestions that should be considered more deeply, and assigning any sort of points to performance in this exercise could cause motivated cognition, which is bad.

The third exercise would be finding a quick way to resolve this competition between desires. This seems the area where it's hardest to be prescriptive- different methods will fit different minds. Here are a few I can think of:

1. Summarize each desire's case in a single sentence, put all the sentences next to each other, and choose one side or the other.

2. Summarize each desire's case in a single sentence, then go with the one that's most compelling.

3. Summarize each desire's case in a single sentence, assign each a weight, and then randomly determine which desire to go with (using the weights).

4. Take the proposed courses of action, and then find compromises along the axis of each desire. Cheap Lucy could be satisfied more and Sweet Tooth Lucy only a little less if Lucy just bought a bag of sugar and ate some of it. Thin Lucy could be satisfied more and Sweet Tooth Lucy only a little less if Lucy bought a cake made with Splenda instead of sugar. Imagine the expanded alternative set and choose from one of the options in it.

I'm sure there are more.

My girlfriend says that a common case of motivated cognition is witnesses picking someone out of a lineup. They want to recognize the criminal, so given five faces they're very likely to pick one even if the real criminal's not there, whereas if people are leafing through a big book of mugshots they're less likely to make a false positive identification.

She suggests a prank-type exercise where there are two plants in the class. Plant A, who wears a hoodie and sunglasses, leaves to go to the bathroom, whereupon Plant B announces that they're pretty sure Plant A is actually $FAMOUS_ACTOR here incognito. Plant A pokes his head in, says he needs to go take a call, and leaves. See who manages to talk themselves into thinking that really is the celebrity.

OK, here's an exercise that could at least help noticing how motivated cognition gives us incorrect beliefs; it's like a mix between calibration and paranoid debating.

Requirements

This exercise requires a set of "interesting questions" with a numerical answer, for example "How many people were killed by guns in New York in 1990-1999?" or "What is the unemployment rate in China" (the questions should be at least related to political/social issues, no "What's the population of Mozambique").

It is also best done in a classroom-type place with a big video projector, and a bunch of computers with internet connections. Somebody who knows Excel should also need to prepare a special spreadsheet.

Step one: Crafting Arguments

Students are together in a room, with one computer each (or they can take turns using a computer, timing isn't critical); an organizer goes to each student and gives him a paper with the question, then flips a coin and announces "high!" if it's heads, and "low!" if it's tails.

Each student then has 30 minutes to prepare a set of arguments for why that particular value is high, or low. The result should be one powerpoint slide (or maybe better, the Google docs equivalent), containing his best arguments; he is allowed to look anything up on the internet (including the true value), but his slide can only contain true information ("New York was rated the most violent city by XXX magazine", things like that).

Step two: Everybody guesses

Once everybody is ready, the organizers collect all the argument powerpoint slides; for each one of them: the question is read aloud, and then the list of arguments are displayed, as well as whether those are arguments for a high or a low value. Each student (except the arguer) writes down his best guess for the answer to the question, as well as a 90% confidence interval; they should be given about thirty seconds.

Once the time is up, everybody reads his answer out loud to an organizer who enters them into Excel and immediately gets a nice chart (projected on the wall), with everybody's answers and confidence intervals compared to the true answer, and then a scatterplot showing how narrow people's intervals were compared and how close they were to the answer.

The clever arguer gets points for how many estimates were too high (/too low), and how wide the confidence intervals were on the side he argued; others get points for the probability they assigned to the correct answer (log scoring rule etc.)

Comments

If this works (if people do indeed bias their estimates despite knowing that the arguments are the result of the flip of a coin, and if they learn to correct this as the exercise progresses), it should give participants a "gut level" understanding of why motivated cognition gives wrong belief (Our conceptual understanding of 'motivated cognition', and why it's defective as a cognitive algorithm - the "Bottom Line" insight.).

This exercised is designed to make the feedback as close as possible to the estimation, to make learning stronger. Having it in a slightly competitive setting also discourages people from just giving huge confidence intervals.

It may be interesting to first collect a bunch of estimations from people who won't participate, just to compare them to the student's estimates on the same questions, so that the students can then be shown the difference between "how people guess with the bottom line argument" and "how people guess without the bottom line argument" (normally, they should guess worse).

A simple variant is having yes-or-no questions, and students giving probability estimates.

Another variant is to have multiple choice questions (ideally with six possible answers, so the organizers can roll a die); this simplifies the guessing (no more confidence intervals!), and questions about numerical values can be transformed into multiple choice questions with a list of intervals.

We might also need an exercise just for getting people to understand the concept of motivated cognition at all.

"Motivated cognition" in the first place seems like a poor label because most thinking is motivated. It's redundant and arguing against "motivated cognition" at first glance sounds like arguing against any kind of motivated thinking. That's problematic because good thinking is also motivated, i.e. "I'll invent FAI because it would help the world."

One interesting thing I've heard repeated and found to be true about rationalizations is that you can usually get the truth out of someone by asking them a canned line: Is there any other reason? I'm sure this is from some Dale Carnegie book or something, but my guess is that most people don't feel the need to come up with multiple rationalizations. They usually come up with one "good" one and try to stick with it. "Is there any other reason why you want the cake?" "Well, I also really love chocolate cake." "Is there any other reason you don't want to go to [Event] besides being busy?" "Well, my ex will be there, too." By no means an airtight method but still useful for getting other people to tell you what they're really thinking.

"If the sky is blue, I want to believe the sky is blue[.]" The question is why people don't want to believe what is true, and the areas of focus here should be the same as with self-deception and procrastination. Irrational people believe that if they can't see/hear/feel/recognize/know it, it can't hurt them -- self deception -- and even in those moments where some thought tells them this is irrational or that they must eventually face the truth, they put it off -- procrastination, preferring the short term over the long.

As kids, I think the experience of "unseeing" or "trying not to see" is common, as we thought of the monster in the dark shadows. On nights when we just shut our eyes and go to sleep there is no monster. On nights where we keep our eyes open and look around, there suddenly is one. So the most important factor in this pre-rational understanding of the world is association -- "I looked, there seemed to be a monster; I didn't look, there didn't seem to be a monster. Therefore looking determines whether there is a monster." At some level we recognize that "the monster" is not a monster, but a trick of shadows, a trick of perception. If we don't look, we also undo the monster itself. This is a detrimental lesson, at this level of understanding. But basically everyone, even kids, recognize this as self-deception. If there is a monster in your closet, crawling under the covers is like making a tasty kid-burrito. If you had actually seen, rather than suspected, a monster, you would run/fight/scream.

Even though someone might logically reason that there isn't a monster or that there isn't a god or that their moral views are contradictory, there will still be a compulsion to lie to themselves because it's easier in the short term. People procrastinate against facing the truth like they procrastinate against writing essays. The kid doesn't turn on the light because what if the monster is actually real even though he knows it isn't? He'll be more scared than he is inside the shelter of covers, even if the monster turns out to be fake, so he prefers to sit right where he is.

The way to deal with this situation is to apply the methods against self-deception and then procrastination. The unfortunate part of that is, it seems some people never get over the hurdle of procrastination.

how to verify that a skill has been acquired

I suppose that one way to test #2 and #3 is to ask the participants (informally, so it does not appear like a test) how useful or successful they think some other skill learning activity was. This is probably a part of the procedure already. There is a significant pressure to say that "they learned a lot and are now better at it", due to several biases, including motivated cognition. When asked to elaborate, they are likely to put forth a number of arguments. To a trained eye, it should be easy to spot whether a particular argument is in fact a rationalization. For example, any immediate supporting argument that is not a specific example of successfully using the relevant skill is very suspect. Some of the better replies would be along the lines of "Wait, how do I know whether this activity was helpful?" and "OMG, I caught myself wanting to rationalize that it was! I guess I did learn something, after all!"

One of the motivations of motivated cognition is consistency. People want to be predictable and they want to be seen as stable. So I suggest demotivating it. Have people read chapter 3 of Cialdini's Influence. I particularly like the Emerson quote:

A foolish consistency is the hobgoblin of small minds.

Yes! Teach that preferring consistency over accuracy is low-status!

I predict this will have no effect unless coupled with the following technique:

Detachment. You are not your beliefs; you are not your actions: When the opportunity to think arises, within five seconds, think those words.

One of the key markers of rationalization I've seen is rationalizations ignore tradeoffs and other option. This is obviously true only about the rationalizations about actions and policies. For instance "I want to eat the whole cake to help the sugar industry..." never finishes "...and this help to the sugar industry is worth any ill health effects." or "...and this is more efficient than other ways to help the sugar industry,"

One activity that might help is to give people a plausible proposition to argue for in their own life that they do not follow. So "Veganism is the optimal dietary option." could be given to someone who eats meat. Have them argue for it, telling them that they will only be evaluated on the persuasiveness of the argument. After this is done, ask them what costs and tradeoffs are for that position. Also ask them what other alternatives there are to achieve the goals and values they hoped to achieve.

This can provide an example to the participants of what rationalization and the results of rationalization looks like. It also provides a demonstration of the efficacy of a couple of questions to catch rationalization.

I notice that you don't mention any existing studies. Is there a reason for this? A fairly cursory search (no backtracking of citations, or significant effort to use synonyms), brings up several relevant articles (mostly paywalled) and a book. It doesn't seem like a a super well studied field, but I don't see why its being completely ignored. The (semi) relevant stuff I found.

The Human Brain as an Evolved Rationalization Machine http://www.epjournal.net/wp-content/uploads/EP102934.pdf

Rational processing or rationalization? The effect of disconfirming information on a stated religious belief.

Self-interest masquerading as ingroup beneficence: Altruistic rationalization and interindividual–intergroup discontinuity.

Reactance versus rationalization: Divergent responses to policies that constrain freedom.

Moral credentialing and the rationalization of misconduct

Rationalization of indigenous male circumcision as a sacred religious custom: Health beliefs of Xhosa men in South Africa.

Regret and rationalization among smokers in Thailand and Malaysia: Findings from the International Tobacco Control Southeast Asia Survey.

:edit spacing

:edit to add I'm currently trying to write a Main level post about current research on rationalization. In doing so I've found that many of the article in the above list aren't actually that relevant, on the other hand I am finding a good bit of useful information so my core point still stands

On the other hand, LW!Hermione failed to reproduce this experiment

I'm not sure how wide of an audience this post is targeting, but the !-notation feels gratuitous here. How about:

On the other hand, Hermione (LW user) failed to reproduce this experiment

The title strikes me as slightly problematic: I think of skills as (positive) ways to achieve outcomes, and framing "avoidance" of some mistake as a skill doesn't feel quite right.

Rationalizing anything: ask participants questions like "why did you have that cake?" or even "yesterday, you hit your kid. Why?". These questions should not be based on reality, and the instructor(s) should probably do the first two or three rounds themselves to get people used to these kinds of questions. Despite the fact that the facts are blatantly false, the participant should come up with a reason that sounds as impressive as possible. ("Yes, I hit my child yesterday. Fourteen generations of Smiths have been brought up that way and gone on to be successful businessman, politicians and lawyers; I'll be damned if I let some wishy-washy state nanny tell me what to do.")

This may teach people to recognize rationalizations in themselves and others.

I am somewhat surprised that I cannot find the honest Devil's advocacy anywhere on the list of techniques of spotting rationalizations. I understand that EY dislikes it, because it's easy to "invent arguments for anything", but how easy is it to invent a good argument against something you deeply believe in? And by "good" I mean an argument that does not appear silly at the first or even second glance (so, no "chocolate cake in the asteroid belt" nonsense). Or maybe this is covered by the "The world is not like X and I believe X" quadrant, though not explicitly.

To me, one of the best examples of the technique is popularized in the ST:TNG episode The Measure of a Man).

Val also used an upside-down W-diagram with the two worlds at the top and the four beliefs at the bottom, to emphasize the idea that the world is there first, and is fixed, and we have only a choice of what to believe within a fixed world, not a choice of which background world to live in.

I don't know what a "W-diagram" is (and a simple Google search didn't help), so I don't see how this works. Perhaps a picture could help explain.

A mantra I use: "I want things to be a certain way. But I don't want to believe they are. I want to believe what is true."

A little more explanation: There was a time when I would say to myself things like, "I want to think that people are basically good." or "I want to believe that my life has not been a mistake." But now I realize that what I want is actually for people to be basically good; I don't want to think that, not if it isn't true. I don't want to believe that my life has not been a mistake; I want my life to not be a mistake. I used to think of the two as almost synonymous; but in fact they are radically different.

In terms of an exercise, it would be a start to get people to list things that would be nice if they were true but probably aren't---e.g. "I will win the lottery tomorrow", "the US federal budget is balanced", "I have everything I want in my life". Then go through each one, and ask: "Do you want to believe that, knowing it's not true? Or do you just want it it to be true? Is your desire about your beliefs, or is it about reality?" Some of the things on the list may be things that you can make true, in which case the next step is, "Go do it." For the others, maybe we could work on techniques for helping people accept unfortunate realities.

The Retrench or Retreat Game

You're making a decision and want to check if you're rationalizing. Imagine a friend or someone else whose opinion you respect raises a criticism of/question about your stated reason. Which feels easier?

1. Falling back to a different justification
2. Arguing with the friend to defend yourself

Maybe this is better as a "True Rejection" exercise, but if you find it a lot easier/more comfortable to shift arguments, your surface answer may have been a rationalization.

The original post mentions some techniques for getting people to avoid rationalizing once they've realized they're doing it, but an earlier step is to get them to realize that they're doing it.

The key to this may be that a person who is rationalizing without realizing it is arguing with him/herself without realizing it, since it's easier to recognize (and to accept) that you're arguing than that you're rationalizing. Accordingly, getting people to realize that they're rationalizing would involve getting them to realize that they're the one that they're arguing with.

The 5-second-level goal would be simply to get them to realize that they're arguing, if that itself is an issue. The next step, getting them to recognize that they're arguing with themselves, would take longer for some people.

(Rationalizing can be distinguished from people who are arguing honestly with themselves in that a rationalizer cares which "one of them" wins, and the non-rationalizer doesn't.)

Step one: recognizing that you're arguing:

If this is an issue, the question to ask is whether the potential rationalizer is thinking about reasons. If you're thinking of reasons, you're mentally arguing with someone. When you want to get a glass of water, you generally don't think about any reasons why you should or shouldn't. When you want to get a soda or a cup of coffee, though, you might about reasons relating to the cost and/or health of the beverage. If so, you're arguing, whether with yourself, or with a mental representation of a friend who suggested drinking less soda/coffee, or whoever.

Step two: determining who you're arguing with:

This step would work best as a series of questions. First, who do you think you're arguing with? Is it a specific person? Is it a hypothetical person?

Why does this person disagree with you? What alternative position do they take, and why that one? What kind of person is it that you're arguing with?

What exactly are this person's arguments, that you're arguing against?

How much do you want to win the argument with this person? Why?

Any suggestions on other questions it would be good to ask in Step two? Personally, I tend to notice it if I'm rationalizing, so I'm not entirely sure how someone who doesn't notice would respond to these questions.

EXERCISE

Ask the person to do an unpleasant task (dishwashing standing on slippery floor), which will cause the unconscious desire to finish the task.

While the task is executed, the person answers questions which can be TRUE or FALSE, the task only ends if the person complete a certain number of TRUE responses, but if the answers are wrong, it decreases the score for the task.

I don't always have a problem with motivated cognition, but when I do, my brain usually makes it some or all of the way through the following steps:

• Notice that I'm becoming more comfortable with (feeling safer about) a decision or action I'm about to make or just made.
• Notice that the physical cause of my comfort is that I recently had a thought consisting of a reason the decision could have a good outcome.
• Some brain process that I haven't pinned down yet that feels like and possibly is a mix of noticing optimization by proxy, feeling disdain for non-generalizable reasoning algorithms, and wondering about the true component strength of the reason.
• Apply my bullshit detector to my comforting thought.
• If appropriate, begin to legitimately think about the decision or action.

If this is a procedure that will work more generally, then these exercises may help:

• Mindfulness meditation. Attention control is critical, mindfulness and a thread on mindfulness
• Noting emotions. Name your emotions, simply and without judgment, as you see them arise.
• Trace emotion causes. When you do manage to notice an emotion and have a moment, practice asking yourself what the proximate cause of that emotion was until it becomes a not-uncommon practice upon noticing a surprising emotion.
• Avoid Optimizing Proxy Measures Train yourself to intuit that optimizing the heuristic measure correlating with your purpose does not optimize your purpose.
• Desire Generalizable Decision Processes Feel uncomfortable with the hint of uncertainty that maybe the algorithm that generated the reason you feel safe would have statistically bad results if applied more generally.
• Check Yourself For BS Practice applying your bullshit detector on your own thoughts.
• Notice Rationalization Directly Actively reward noticing rationalization.

Shinzen Young's "Do Nothing" Meditation:

http://www.youtube.com/watch?v=cZ6cdIaUZCA (Note that he is "neuroscience aware.")

http://www.shinzen.org/Retreat%20Reading/FiveWays.pdf

http://www.basicmindfulness.org/

(Also the discussion of "Willingness" in http://www.amazon.com/Acceptance-Commitment-Therapy-Second-Edition/dp/1609189620/)

The goal is to raise the baseline of equanimity and mindfulness. You naturally start allowing emotion and inner talk to thunder through you without letting it drive behavior. This is a prerequisite for rational deliberation and choice under increasingly emotional situations. Maybe.

Whenever you find yourself about to express a belief, ask yourself if you really believe it it first. This works for me, but may not work for people not feeling curiosity or for people not already in the habit of being honest with themselves.

...often that thought causes me to not know what I believe.

Of course the hard part is figuring out a way to reliably put people in a situation where they will find themselves rationalizing. Social pressure would work on a lot of people, but probably not so well on LW types.

This seems like it'll be easiest to teach and test if you can artificially create a preference for an objective fact. Can you offer actual prizes? Candy? Have you ever tried a point system and have people reacted well?

Assume you have a set of good prizes (maybe chocolate bars, or tickets good for 10 points) and a set of less-good prizes (Hershey's kisses, or tickets good for 1 point).

Choose a box: Have two actual boxes, labeled "TRUE" and "FALSE". Before the class comes in, the instructor writes a proposition on the blackboard, such as "The idea that carrots are good for your eyesight is a myth promoted as part of a government conspiracy to cover up secret military technology" or "A duck's quack never echoes, and nobody knows why." If the instructor believes that the proposition is true, the instructor puts a bunch of good prizes in the TRUE box and nothing in the FALSE box. Otherwise, the instructor fills the FALSE box with less-good prizes. The class comes in, and the instructor explains the rules. Then she spends 5 minutes trying to persuade the class that she believes the proposition. After that, people who think she actually believes it line up at the TRUE box, and everyone else lines up at the FALSE box. Everyone who guessed right gets a prize from their box. If you guess TRUE and you're right, your prize is better than if you guess FALSE and are right. Repeat this for a few propositions, and it's at least a useful test for whether you can separate what you want from what seems plausible.

I've reading about the CFAE exercise prizes. Now that there are some great tools I can learn, ay tips for mitigating the initial pain of going from any given irrational state to a more rational one? Any stories of how learning these rationality exercises have made an impact on some part of the quality of your lives?

Exercise: Trick people into rationalizing

Engineer a situation where subjects will perform an action because you tricked them into doing it, and then ask them for reasons why they performed that action. Specifics here that actually work are beyond my pay-grade. But I'm reminded of an experiment where a subject was hypnotized and given an umbrella. After the hypnosis, the researcher asked "Why do you have an umbrella?" and the subject, initially looking a bit shocked at the umbrella, answered "I thought it was going to rain".

It would help to have a list of reliable triggers to make people perform actions without conscious decision, which most people wouldn't be aware of.

This is conceptually motivated by the finding that giving a child specific warnings is less effective than giving nonspecific warnings. Supposedly, if given a vague warning the child will make up their own reasons for avoidance of bad behavior, like "I didn't really want to do that anyway."

And the point of this, of course, is to have a definitive case of rationalization to point to, in order to train the ability to spot rationalization in oneself.

Think about a time when you were badly disappointed. Then think of any opportunities you had previously, to notice that things weren't going to turn out the way you'd expected. In particular, cases where you came across an argument or evidence that in hindsight should have changed your opinion, but you dismissed by a clever argument, or by not thinking about it.

Think about what you could have done in the time you had between your missed opportunity and the disappointment. Plans you could have changed to make the best of a bad situation. Mentally contrast the actual outcome with the outcome had you been forewarned, and allow yourself to be very annoyed at your past self for throwing away time.

Instructor writes a decision on the board and asks all participants to give factors to consider before making that decision. Participants are then asked as a group if each factor (for typical possible values of the factor) is 1. necessary and 2. sufficient to determine whether the decision should be taken.

I would ask Lucy if she would still eat the cake if the sugar industry were doing fine. More generally, I would ask if there is any action Lucy is taking that they would not take if the sugar industry were doing fine.

Ask the participants to come up with a belief that they feel is important to them but hasn't been necessary (though they may mistakenly think it sufficient) for making any meaningful decisions.

One needs to avoid "sloppy cognition". If you are making a mathematical proof to show off and really want to be correct, you still won't make a mistake just as long as you are at least somewhat competent at writing proofs. If you are a programmer and you want your software to work, and inevitably want to believe it'll work, if you are competent and not sloppy in your thought - which you can acquire by training - you won't make extra mistakes just because you want the outcome.

On the other side, if your thinking is sloppy, somehow eliminating motivated cognition won't help you a whole lot. It might marginally decrease the error, in some circumstances when the inference chain is very short and some sense goes through despite the sloppiness.

And of course, on the worst side of sloppy, is the thinking where there is no correct logical pathway from something true to what you're thinking, or all pathways are too computationally expensive to traverse, in which case, you're doomed to nonsense, motivated or not (the best you can do is refrain from making an answer in such circumstances).

edit: TL;DR; : if your thinking style can be affected by 'motivated cognition', it is most likely sloppy enough so that the result is most likely entirely untrustworthy even if you remove the 'motivated cognition' from the equation. Same goes for other biases. No, removal of biases does not give you magical computing superpowers allowing you to answer dramatically harder questions. And the unmotivated sloppiness is still sloppiness.