SotW: Check Consequentialism

So... how would you design an exercise to teach Checking Consequentialism?

I would check to see if such a thing already exists or if there are people who have experience designing such things. I know of a Belgian non-profit 'Center for Informative Games' that not only rents games designed to teach certain skills but will also help you create your own.

From their site: On request C.I.S. develops games for others. The applicant provides the content of the game, while C.I.S. develops the conceptual and game technical part to perfection. The applicant has the opportunity to attend some game tests and to redirect when necessary.

They also offer coaching if you want to work on your own: Do you want to create a game concept on your own, but you don't know where to start? No worries C.I.S. can give you a hand. During a number of concrete working sessions C.I.S. facilitates your development. In between sessions the applicant continues the work to, finally, end up with a solid game.

I have enjoyed their games in the past and can attest to their quality. The obvious problem is that it's a purely Belgian organization and the 'search' function only works with Dutch words. However if you want to check them out I'd be happy to act as a go-between. Since a couple of months there is even a Brussels LW meetup, I'm certain I could get a couple of members to help in the production process (again, if this seems interesting)

In a group, with a leader who knows the exercise:

Get a volunteer to act as a judge (or a few to act as a jury, in a large group). Have her leave the room. The leader presents the rest with a short set of Contrived Hypothetical Situations, each with finite options and either clearly-defined outcomes for each option, or a probabilistic distribution of outcomes for each option. The leader says, "Please write down your choice for each problem, sign your paper, and turn it in to me. Then I'll call in the judge, and have her decide on each problem. You get a point wherever her decision agrees with yours. The winner is the one with the most points." When the judge is called in, however, the leader doesn't tell them the actual problems. Rather, the leader just reports the outcomes (or distributions), and asks them to choose which outcome or distribution is best. The winners are announced based on that.

Example: One of the situations given is some variant of the trolley problem. When the judge comes in, she is just asked whether she'd prefer one person to get hit by a trolley, or five. Everybody laughs as she replies "...one?"

Example: The problem given to the group is "You drive 45 minutes away from home to go to a new restaurant for dinner. When you get there, you discover that you dislike the ambience and the selection is poor. You remember that you have decent leftovers at home. You're mildly hungry. Do you try the restaurant anyway (25-minute wait, 10% very enjoyable meal, 10% decent meal, 80% unenjoyable meal) or just head back home (5-minute-prep once you get home, 100% chance decent meal)?" The problem given to the judge is "You're mildly hungry. In 25 minutes, you can have a meal that is (10% very enjoyable, 10% decent, 80% unenjoyable). Or, in 50 minutes, you can have a guaranteed decent meal."

The novel part of Checking Consequentialism is the ability to discriminate "consequentialist reasons" from "non-consequentialist reasons" - being able to distinguish that "Because a PhD gets me a 50% higher salary" talks about future positive consequences, while "Because I don't want my years of work to have been wasted" doesn't.

It's possible that asking "At what time does the consequence occur and how long does it last?" would be useful for distinguishing future-consequences from non-future-consequences - if you take a bad-thing like "I don't want my work to have been wasted" and ask "When does it occur, where does it occur, and how long does it last?", you will with luck notice the error.

"Intense, long, certain, speedy, fruitful, pure—
Such marks in pleasures and in pains endure.

— Jeremy Bentham's mnemonic for the signs of "consequentialist reasons"

EDIT: It occurs to me that I should explain this more. Bentham was trying to popularize consequentialism and remind his readers of what sorts of things count as consequentialist reasons to prefer a particular outcome. Eliezer suggests that we should ask about the closeness in time and the duration of a consequence. Bentham mentions these ("speedy" and "long") but also includes some others:

• Intensity — How great of a pleasure or benefit does this consequence bring?
• Certainty — How sure are we that it will actually happen? Is it a relatively sure thing, or just a chance at one?
• Fruitfulness — Will this pleasure be followed by others of the same kind, or will it exhaust itself?
• Purity — Is this a "pure pleasure" or a "mixed blessing"? What are the downsides; what's the catch?

Cleverness-related failure mode (that actually came up in the trial unit):

One shouldn't try too hard to rescue non-consequentialist reasons. This probably has to be emphasized especially with new audiences who associate "rationality" to Spock and university professors, or audiences who've studied pre-behavioral economics, and who think they score extra points if they come up with amazingly clever ways to rescue bad ideas.

Any decision-making algorithm, no matter how stupid, can be made to look like expected utility maximization through the transform "Assign infinite negative utility to departing from decision algorithm X". This in essence is what somebody is doing when they say, "Aha! But if I stop my PhD program now, I'll have the negative consequence of having abandoned a sunk cost!" (Sometimes I feel like hitting people with a wooden stick when they do this, but that act just expresses an emotion rather than having any discernible positive consequences.) This is Cleverly Failing to Get the Point if "not wanting to abandon a sunk cost", i.e., the counterintuitive feel of departing from the brain's previous decision algorithm, is treated as an overriding consideration, i.e., an infinite negative utility.

It's a legitimate future consequence only if the person says, "The sense of having abandoned a sunk cost will make me feel sick to my stomach for around three days, after which I would start to adjust and adapt a la the hedonic treadmill". In this case they have weighed the intensity and the duration of the future hedonic consequence, rather than treating it as an instantaneous infinite negative penalty, and are now ready to trade that off against other and probably larger considerations like the total amount of work required to get a PhD.

Not attempting to answer the question, but I've been nursing a thought about the rationality org that might be useful: The nearby Stanford University has a world-renown program in "decision sciences" http://decision.stanford.edu/ which is basically "how to make decisions scientifically"; they virtually invented influence diagrams, they teach biases as a part of the program, etc. The head of the program, Ronald Howard, also co-founded http://www.decisioneducation.org/ , his teen-oriented "rationality org".

• there is probably things to learn from both

• if "rationality org" has a value proposition to these organizations they can be useful in teaching opportunities and for credibility building

I am reminded of a game we played in elementary school:

There are 100 pieces of candy in a jar, and 20 students. Each student gets to vote "share" or "selfish". If EVERYONE votes to share, the candy is split evenly. If ONE person votes "selfish", they get all the candy. If MORE than one person votes "selfish", no one gets candy, and the experiment is repeated until either the candy is distributed, or X iterations (3-5 seems normal) have passed.

Before each iteration, the students are allowed to discuss strategy. The solution, of course, is for a single trustworthy person to make a binding commitment to vote "selfish" and then evenly distribute the candy. By pre-commiting to vote "selfish", everyone else knows that voting "selfish" themselves will result in no candy - unlike a commitment to have everybody share.

I've always considered it a decent test of group rationality and social skills whether or not this consensus actually gets reached before the final iteration. I've seen groups that hit on this, had a single iteration with a few outliers testing it just to be sure that the person would really vote "selfish" like they said, and then implementing the strategy. I've seen others where 10-20% of the audience simply would not believe the person who made the pre-commitment, and so there was never a successful iteration

An important question to ask that you are leaving out is "What are my alternatives to this course of action?"

The comparison of consequences requires an alternative set of consequences to compare to. Considering the question "Should I be in graduate school?" The answer may well be different if the alternative is getting a decent job than if the alternative is starving unemployment.

The listing of alternatives also helps catch cheating. If the alternative is implausible and disastrous (Stay in grad school or be a hobo) then it is likely that Checking Consequentialism isn't being done seriously.. The alternative compared needs to be a serious answer to the question "What would I do if I couldn't/wouldn't do this?"

This seems like a comparatively reliable procedure: imagine a collection of possible worlds generated by possible actions; explain what future events distinguish between these worlds in a way that makes one of them preferable to the others; then choose the action that leads there.

Paying attention to events that distinguish these possible futures from each other guards against errors such as comparing to status quo or should-world (neither of which is among the possible futures), or worse comparing an arbitrarily picked-out event (in one of the possible futures) to an arbitrary anchor (i.e. a cost that feels excessive in some absolute way, and not by comparison to alternatives). Focusing on future events and not past events or actions themselves guards against deontological and identity-based arguments, such as "this is a proper action", "he shouted first" or "because I'm a human being".

Saying "positive consequence" sounds like a bad habit to me: positive compared to what? The comparison should be among the alternatives, without anchoring on some baseline of neutrality such that some consequences are more "positive" than that.

Concreteness Game The object of this game is to train players to generate examples for explaining concepts quickly. The game requires at least two people, but may work better with groups of three or four.

To play, one of the players (Asker) names a concept to be explained, such as "How do you traverse a linked nodal network", "Explain the law of Conservation of Energy", or "What constitutes good financial advice?"

The other player (Generator) then tries to explain the concept/skill/other by using a nearby object to assist. The Generator should relate the example back to the original query and explain how the example demonstrates the experimental predictions that the concept makes (see Extended Example below). The Asker listens to the explanation, and once the Asker feels as though the generator has been able to explain the concept fully, they indicate to the Generator that the Generator should pick another object and start again. Another option the Asker should exercise is to ask follow-up questions about the Generator's example, asking for clarification or elaboration by interacting with their object in some way (examples in the Extended Example section below)

Extended Example:

Asker: "Okay, let's try this. Explain Newton's Third Law." Generator: "All right, hmm... Okay, take that big old oak tree as an example! Now, imagine that I want to push this tree over. If I push on it, it doesn't really move anywhere, and neither do I. That's because the tree has a really strong root system to prevent it from being tipped over, while I'm braced against the ground. That means the force I put on the tree doesn't go anywhere. Now, what happens if I don't brace against the tree? I'm going to try and push with the same strength that I did just now. If the tree didn't exert any force on me, then why did it force my arms away strongly enough to tip me over?" Asker: "What happens if you were standing on ice instead of dirt?" Generator: "Hmm... If I wanted to avoid spinning away from the tree, then I couldn't push against it too hard. If I did want to, though, then I could turn this whole area into a pinball machine by pushing off of all the trees!" Asker: "Okay, another example this time!" Generator: "All right, look at that guy in his kayak over there. Every time he uses his paddle to push himself forward, you see a whirlpool where his paddle was. That's because he's pushing against a lot of water with the flat of the paddle. The paddle pushes on the water, and the water pushes back on the paddle. You can see this because he moves forward while the water is disturbed. If the law didn't apply then the water shouldn't-" Asker: "Okay, you've got that one, give me another one!" Generator: "Hmm..."

Through playtesting, we've noticed that there are two broad categories of useful questions in this game. The first type are "You have a rule, now apply it with the objects in front of us". For example, one might describe good financial advice by pointing at a nearly-broken-down refrigerator: "Some advice you have to use right away like take-out food before it spoils. Other advice will keep basically forever, like ketchup maybe, while other advice will let you repair the refigerator to let it keep everything fresh for longer. What you want is to avoid poisoning yourself and keep healthy, so good advice is anything that keeps from poisoning you or your friends. Stock tips are like take-out food, while better mental models are like fixing the fridge." The other type of useful question seems to be descriptive in nature. "Explain the life-cycle of a caterpillar": "Well, imagine riding around on a cheap kid's bicycle and picking up tubing, gears, and other supplies as you ride around. Then you take the bike and everything you collected into a room, work for a while, and you come out with an awesome racing bike that lets you do things you never could before".

Example questions: Explain Conservation of Energy. What constitutes good scientific practice? Explain the sunk cost fallacy. How do you lift heavy things safely? What constitutes an efficient algorithm? Explain (Hansonian) signaling. Explain priming and how expectations about the quality of a thing can affect your assessment of its quality. Describe how Omega might optimize for happiness. Explain what a set (in set theory) is and some of its basic properties. Explain the fallacy of gray.

Theory (SPOILER ALERT! This section contains material likely to prime your reactions) This game is designed to help people provide concrete examples on demand. The expectation is that: Forcing the players to compare their mental models against physical objects makes their explanations more concrete because physical objects can be interacted with. If a Generator relates a tree to AI theory, the question "What is a branch and what happens when I push on it?" seems to yield concrete answers more often in practice than "What are the features of AI theory and why do they matter?". The concept of follow-up questions seems to greatly increase the fun of the game. Many more grins were observed when the Asker occasionally quit saying "Okay, got it, another example now!" and instead interacted with the physical model in some way. Playtesting seemed to also show that people really enjoyed coming up with examples up to their third, and the fourth became difficult to generate while the fifth was simply not all that fun to force ourselves to come up with. Priming may have been an issue, but initial results suggest asking for roughly three examples is the most fun before moving to a different question.

EDIT: Formatting

You could use mazes where your score is -(total distance traveled) . First, give a simple maze with two obvious paths, A and B but Path A is much shorter. Then give a second maze identical to the first but you are taking over from another player who has already gone down Path B but the shortest way to the exit is to double back and then go down Path A. Then give the same maze but now there is an obstacle on Path A that you must go around if you take this path and so it's now optimal to go on Path B. The obstacle was placed there for some unfair reason (Perhaps only women will face the obstacle.) Next, the same situation as before but you have the ability to erase the obstacle at no cost. Finally the same as before but now you are told that the maze is only a map and you can't erase the obstacle on the map. You should have the people play for real money to make the game more emotionally "meaningful." Also, you could plant someone who makes arguments against the consequentialist outcome using analogies to real life situations (i.e. women should never give in to discrimination.)

Kahneman suggests such an exercise for groups after pointing out that organizations generally act more rationally than individuals. The devil's advocate role and thinking at the worst possible outcome. We don't always have the luxury of having others near us for checking our thoughts. But we often have imaginary conversations with friends or parents. So it shouldn't be very difficult to assign a devil's advocate position to a imaginary voice. That should put in perspective the way we feel about the subject. It is a basic mean of delaying the strong coherence of the first good narrative.

Maybe it would be great to have an imaginary Bayesian friend...

This wouldn't work as the only exercise, but could be useful if paired with another.

Presumably all students have other things they could be doing with their time, some of it possibly fun. Near the end of the lesson, perhaps just before the last exercise, with maybe 15-20 minutes left in the session, present this option: You can leave right now and get on with your day.

Obviously this is always an option, no one is required to stay against their will, but it's usually considered bad form and it's never given as an explicit option. Tell everyone to think about it for at least a full by-the-clock minute. What they could learn in the next 20 minutes vs what else they could do, and the consequences of both options. Then let them make their decision.

Afterwards both those who stayed and those who left may rethink their choice at least once and wonder if their choice was the best one. If nothing else, it could be memorable.

I think of this as "looking effectward", one of the basic directions in concept space (opposite causeward, making it the inverse operation of asking "why").

Another inferential path: it may be valuable to differentiate them as attitudes and events. If my motivation for getting a PhD is "I will feel terrible if I do not get a PhD", that's an attitude motivation, which in theory I have internal control over. If my motivation for getting a PhD is that I will get hired for a different sort of job, that's an event motivation, for which control is primarily external. I don't have control over whether or not a variety of job requires a PhD, but I do have control over whether or not my attitude will be negative if I don't have a PhD.

The obvious social mirror of the Really Getting Bayes game is to have people pair up and dissect motivations for something, trying to identify what's event-based and what's attitude-based (or however you're presenting it). It will help for them to be motivations they're feeling, rather than descriptions they're reading.

Asking them to provide examples from their life is potentially useful and will promote bonding between participants, but requires heavy investment by participants. Another approach is to offer them a choice between gambles, where all potential consequences are as apples and oranges as you can make them. Some more trickery could be used to provide non-consequential reasons to pick one gamble over another- perhaps give participant's names to the gambits, but don't require them to pick the gambit with their name.

I think an important pre-skill is to decide what your goals are BEFORE checking the consequences. Otherwise, you can easily retroactively change your goals to fit whatever action you want.

For example, in the sunk costs PhD scenario, it would be easy for someone to say something like "If I pursue my PhD, it will make my family proud. This is very important to me, so I will pursue my PhD." However if you asked them what their goals are BEFORE they listed the consequences of each action they probably would not list "Make family proud", but would list things like "Job I enjoy" and "Make more money in my career".

The trap is that it can be very difficult to admit to your ACTUAL goals. For example, it took a while for me to realize that I don't ACTUALLY care about making money once I have enough to survive (unless phrased in a way such as "If I do Job B, I only have to work 12 hours/week" or somesuch).

So I think figuring out what your actual goals are (noit the cached thoughts of your goals) is a set of exercises itself, but that they should probably be done before the Consequentialism exercises.

an audience of non-Sequence-reading experimental subjects, who were mostly either programmers or in other technical subjects, so I could go through the math fairly fast

I don't have suggestions on the main question, but I strongly recommend that you design the curricula to be consumable by corporate executives. If you can convince companies that they need to send their execs to week-long rationality seminars, to help them improve their decision-making skills, you are going to make megabucks.

We did the Really Getting Bayes game at the Mountain View meetup this week. My impression of it was that the explanation was at first a little unclear, but that once we had gotten the sense of it, it was worthwhile.

One thing that I realized during the game was the degree to which I was using the availability heuristic to provide my likelihood ratios. For instance, one object I was given was either an electrical extension cord or an audio cable. In coming up with RGB likelihoods, I thought, "Electrical extension cords are usually black or white, hence not RGB; whereas audio cables are often RGB" — by using instances I could bring readily to mind to come up with this thought. My partner then told me the item was RGB, and I correctly predicted it as an audio cable. However, the electrical cord in his other hand was in fact orange, not black or white. While my availability heuristic worked for the purpose at hand, it clearly would have been wrong for other purposes.

Why is teaching people to think like consequentialists a good idea again? Serious question.

If they're (relatively successful) mathematicians and programmers I don't see how it could go wrong but I'm awfully worried about some of the rest of the population. Specifically people not being able to sustain it without letting other things slip.

second edit: I should clarify. It's teaching the habit that I'm thinking about. Everyone should be able to think like a consequentialist but is instilling these reflexes gonna be a net positive?

"You shouldn't kill because it's the wrong thing to do" can be rescued as "Because then a person will transition from 'alive' to 'dead' in the future, and this is a bad event" or "Because the interval between Outcome A and Outcome B includes the interval from Fred alive to Fred dead."

Why the fancy words? This just seems like a complicated way of saying: "Because the person would then be dead. And that is bad".

To me, this comes down to what I am trying to learn as my anti-akrasia front kick: I cache the question "Why am I doing what I am doing?". While I lose some amount of focus to the question itself, I have gained key insights into many of my worst habits. For instance, my employer provides free soft drinks- I found that I would end up with multiple, open drinks at my desk. The cached question revealed I was using the action of getting a drink whenever I felt the need to stretch and leave my desk. Browsing reddit too much at work- cached question can catch it. Eventually, when I have affirmative answers for the question, it no longer even draws focus away from the task at hand- it is simply an itch that is easily scratched, as I know I am doing something that accomplishes a larger goal.

Many of the pain points listed have a common trait: the decision would seem easier with less information. For example, the PhD decision is easier if you didn't know about the costs which have been sunk, the identity decisions are easier if you're not sure of your own identity, cached thought problems are easier without having that thought cached, etc...

But we know that information should never have negative value. So why not highlight that dissonance? Imagine the following exercise:

Handout: "You spent the last 3 years working toward a PhD. You passed up a $90k job to stay in the program. Now you have 2 years left, and another $90k job offer has come your way. Do you take it?" (I don't know much about PhD programs, so feel free to imagine more plausible numbers here and add narrative).

Exercise 1: Is there any information you would prefer to not know? Exercise 2: How much would you pay to not know it?

If you really want to have fun, give people monopoly money and let them bid to remove information from a range of scenarios. Note that we're not offering to change the facts, just to not know them.

Personally, I think this would be a lot easier if I could just forget about all that time spent in the PhD program.

At least in this case, the exercise highlights the difference between consequentialist and non-consequentialist reasons/excuses for doing things. The "how much would you pay to not know it" is especially handy, since it puts a number on that mental pain. Then we can ask whether the mental pain is worth all the money you'd lose by, in this example, staying in the PhD program.

If the world were going to end right after I took an action, which action would I choose? (Alt: If everybody saw what choice I was about to make, but then circumstances changed and my decision turned out not to matter, what choice would I want to have made?)

Did answering that question feel the same as answering the actual question? If so, I'm not really thinking about consequences.

So... how would I design an exercise to teach Checking Consequentialism?

Divide the group into pairs. One is the decider, the other is the environment. Let them play some game repeatedly, prisoners dilemma might be appropriate, but maybe it should be a little bit more complex. The algorithm of the environment is predetermined by the teacher and known to both of the players.

The decider tries to maximize utilitiy over the repeated rounds, the environment tries to minimise the winnigs of the decider, by using social interaction between the evaluated game rounds, e.g. by trying to invoke all the fancy fallacies you outlined in the post or convincing the decider that the environment algorithm actually results in a different decision. By incorporating randomness into the environment algorithm, this might even be used to train consequentialism under uncertainty.

Maybe the easiest way to teach it is to teach how it applies to others. That is, train people to catch nonconsequential reasoning in arguments that others make, and then hope that they apply that to themselves. The easiest way to do that is by reflexively asking, "so what?"

Here's a long-form exercise:

1. Break up into small groups (2-5 people)
2. Someone in each group, picked at random, presents an upcoming problem or decision.
   The other participant(s) ask questions to clarify the problem/decision.
   The problem/decision can be either real or imaginary, but if imaginary the presenter must come up with appropriately detailed answers to questions.
3. Everyone collaborates in generating a list of possible solutions. Five is plenty.
4. Everyone privately notes their preferred solution.
5. Everyone collaborates on a list of expected consequences of each solution (with confidence intervals, if desired) and publicly announce which consequence-list they consider preferable.
6. Everyone reveals their earlier preference from step 4, and if it's different from their preference from step 5 explains what changed.

This will probably take a fair amount of time.
The shorter-form version starts with steps 1-3 already done as example problems.
It probably helps if step 6 is a surprise.

Incidentally, I prefer "Compare Likely Consequences" to "Check Consequentialism" as a label for the skill.

Your check consequentialism sounds a lot like risk management. Risk is the effect of uncertainty on objectives (ISO 31 000). The risk management process involves indentifying risks, analysing how significant they are, and then treating the big ones so that they don't prevent you from attaining your objective. This is fairly straightforward to do. The difficult part is building a risk management culture where the risks are considered before making a decison, embarking on a project, etc. Just identifying the risks is often the big deal. Once you are aware that a risk exists you will probably deal with it. Sorry that I have not given you an activity, but perhaps I have given you a useful keyword to help your search.

Obligatory link: Urges vs. Goals: The analogy to anticipation and belief.

By the way, if you don't mind my asking, once you've come up with your rationality curriculum what do you plan to do with it? Are you making inroads with whoever you would need to talk to to get this in a school curriculum, for instance?

What, we're not even allowed to have identities now?

It occurs to me that games with some significant strategic component might be useful for priming the "but what consequences does it have?" response. I'm thinking of games like Magic: the Gathering, Settlers of Catan, Risk, etc. (I'm sure the board game aficionados will have better examples than I). I say this because of personal experience with Magic players - as they get better at magic, they tend to get better at life. Well, some of them do. The others perhaps compartmentalize too much, so maybe this won't help with everyone.

In any case, my model for what would work is a relatively easy social game that allows a non-trivial number of actions with unclear consequences... unless you stop to think about them. Magic would be perfect... if it wasn't so complicated and if fantasy tropes didn't turn off a large segment of the population. Ideally the game would be something you create instead of something your subjects/clients may have played before.

I have no ideas for the actual game, but maybe this sparks someone else's imagination.

As far as I can tell, most people find it fairly easy to think about others' decisions in consequential terms, but have a lot more trouble thinking about their own that way. So, a good technique to switch to consequential thinking is to imagine that instead of thinking about your own decision, you're thinking about a decision that someone in your exact situation is making. Consider what advice you'd want to give this person, and what choice he or she should make. Disassociating yourself from the decision like this should remove the influence of most things that would normally lead to you making bad decisions.

I'm not entirely sure what would be a good exercise to teach this. One thing that comes to mind is that it might be useful to ask participants to think of to give advice to their selves from five years ago (for example), and then to imagine themselves five years from now giving advice to their current selves.

Possible exercise: Assume that you have no source of income except what you can beg, steal, or find ownerless/abandoned. Assume that you have a friend in similar straits (we'll call this person Paul Poor), and that both you and Paul know of a very wealthy person (whom we'll call Richard Rich). One day, you find — apparently abandoned in the street — a loaded gun. Think of various reasons for you to use the loaded gun to force Richard to give money to Paul. Which of these reasons are non-consequentialist, and why? Now think of various reasons for you to NOT use the loaded gun to force Richard to give money to Paul. Which of these reasons are non-consequentialist, and why? Next step: think of various reasons to use the gun to force Richard to give money to you. Which of these reasons are non-consequentialist, and why? Finally: think of various reasons NOT to use the gun to force Richard to give money to you. Which of these reasons are non-consequentialist, and why? In case you use this as an exercise, and/or wish to contact me for any reason, my name is Kate Gladstone and my e-mail address is handwritingrepair@gmail.com

Exercise: Notice results.

Example: (participants A and B)

A: What would be the consequences if you told everyone you were not wearing any underwear?
B: writing it down: I would be really embarrassed for the rest of the day, and everyone would laugh at me so I wouldn't want to show my face.
A: Go do that.
B: gives paper to A, does it
(some time later) A: What were the consequences of telling people...?
B: not reading the earlier paper I was a little embarrassed at the time, but people laughed so it was okay. Also, one person hit on me.
A: shows paper to B
They talk about the difference between the predictions and what actually happened.
Instructor (optional): See, this and this were not physical consequences of the action, so that's why your predictions were off-base.

Description:

2 participants. A asks B to do some action, ideally somewhere in the range that is emotionally-loaded but not completely inappropriate. B has to write down the consequences of that action (perhaps out to the end of the day / session), give the paper to A for safekeeping, and perform the action. Later, A asks B to describe what the consequences actually were and checks them against the written prediction. Ideally, an instructor points out any listed consequences that aren't 'legitimate', and explains how they could have been improved.

Rationale:

We feel differently about consequences after the fact, so independently checking our predictions about consequences against our memories of consequences should highlight any 'illegitimate' consequences listed.

Caveats:

This might just be an exercise to combat Impact Bias, or other biases related to affective forecasting.

On a high level, practice asking: If I do X, what does the world look like 5 minutes from now? An hour from now? A day from now? etc.

If I don't do X, what does the world look like 5 minutes from now? An hour from now? A day from now? etc.

So, let's take the PhD example. Try talking about it without using the word "because".

If I decide to finish my PhD, 5 minutes from now I feel OK. An hour from now I'm eating dinner. A day from now I'm grinding away at my dissertation. A month from now, I'm grinding away at my dissertation. A year from now, maybe I'll have finished my PhD, and I'll be on the job market. A decade from now, I'll likely have a research job in my field, or I'll be teaching, or I'll have abandoned my field and done something else.

If I don't decide to finish my PhD, 5 minutes from now I feel like a total failure, I wasted so much time. An hour from now I'm stuffing my face with junk food to deal with the stress of figuring out what to do with my life. A month from now, I've done something totally different with my life. (got a job? started a business? backpacked through Europe?) A year from now, something similar. A decade from now, who knows?

To make it vivid, ask people to talk about a choice they themselves face, and weigh the pros and cons in terms of likely future world-states. Each time you talk about a consequence, you have to mention what is going on if they chose differently, at the same distance in the future. Everyone else's job is to call them out if they go off track.

It seems to me that since it's easier to notice it in other people, starting with illustrative examples would be good. This allows you to establish the basic idea, and establish a model in the audience's head. I'd suggest fictional examples would be easier to come up with, but using real examples might add veracity and help the audience engage. You could possibly even invite a few audience members up to discuss things where they might be stuck, but that runs in to the usual risks of audience examples and would probably take a fair amount of time.

Once you've established the basic idea, I think you then need to transform the idea by getting them to apply it to themselves. Relying on the audience to all have a situation where this skill applies, and one which is a good learning example, seems foolish. Instead, I'd suggest a game where you lead someone to get stuck in a commitment or trend, and then throw them at a situation where they have to break that trend to succeed.

A simple example would be to pick an audience member, and tell them to answer each question you ask "yes or no". You ask a bunch of questions that all produce an immediate, unthinking 'yes' response, and then ask them one where a 'yes' would be humorously inappropriate.

"Do you understand the idea?" "Of course" "It needs to be a yes or no. Understand?" "Yes" "Good. Is the sky blue?" "Yes" "Are dogs a mammal?" "Yes" [...] "Have you mastered Consequentialism?" "Yes" pause, cue audience laughter "Oh. No."

I'm entirely certain one could come up with both better and more complex examples, but I think that serves to illustrate the basic idea. Consequentialism suggests that putting more work in to examples probably isn't wise unless someone suggests this is a good idea and would like to see me flesh it out more :)

This is the sort of in-person, hands-on, real-life, and social exercise that didn't occur to me, or Anna, or anyone else helping, while we were trying to design the Bayes's Theorem unit. Our brains just didn't go in that direction, though we recognized it as embarrassingly obvious in retrospect.

There's something important here. Problem solving. That's the use of intelligence that got us to the Moon. That's the use of intelligence which gave us Bayes theorem. And the best way you can spend your time is focussing on this. It does not help you a whole lot if you can very rationally pick between two ways of teaching the students, if you cant generate those ways. For success, one needs, first and foremost, problem solving abilities. There may be general intelligence factor, but there is also a lot of very high IQ people who are comparatively bad at free form problem solving, and even at fairly basic combinatorial things like fitting most items into a box carefully so they won't break when transported. The rationality may help you de-bias yourself wrt which items you consider more and less 'important' - you may be able to rid yourself of bias of how dear an item is to you - but it won't so much help you process the immense number of combinations and generate the best one, and your packing will still be much less effective than the packing of someone irrational who puts a nearly indestructible object that is dear to them on the top, if they are just a bit better at processing the huge solution space.

Beware of motivated stopping. If someone wants to do A, because B will happen, that is only the beginning. There are several directions it's worth exploring further, with one person exploring and another prompting them with questions such as these:

Will B actually happen (or be more likely to), given A?

What makes B a desired consequence? Some further consequence that it leads to? Some larger purpose for which B is a means? Or is B terminally desirable?

At some point one has to stop, but the very first consequences one thinks of may not be that point.

Split the class into groups and get each group working on something they all will easily become invested in. I'm thinking have them spend 10 minutes creating/building something as a group, and make it a competition (bragging-rights only) to solidify the investment.

Before anyone has enough time to finish, offer $100 to the first person to destroy their group's creation. (Obviously, it would be best if doing so could be done in a quick motion: like if they were building a large tower with jenga blocks or something.)

After 5 seconds, pause and have each person self-evaluate: did they Check Consequentialism? Then, group up again to rescue consequentialist reasons to act/not act from non-consequentialist reasons for doing so. i.e. "I don't want to do selfish things" -> "I don't want to look selfish in front of my peers".

What kinds of exercises you use to teach a skill like "checking consequentialism" should probably be placed in the greater context of a rationality curriculum. You have to know where the students are coming from at each step.

That said-- making the assumption that the students are already familiar with the theory of heuristics and biases, and just need to learn how to apply them-- I think most of these can be taught with similar kinds of hypotheticals and problems.

For checking consequentialism, you might want to focus on problems involving sunk costs. To illustrate how sunk costs can affect how someone automatically approaches a problem, split students up into groups (they will probably produce worse results as a group, which is good for this part) and give them all similar problems, with the initial conditions modified slightly. Example: "John has been working on his PhD for X years, and expects to finish in Y. He knows W and Z facts about how his degree will benefit him." Modify parts of the problem, X and Y especially, to try and prime their System 1 for a different result. Have them make a decision quickly. Reconvene, discuss the problem, point out the issues with sunk costs and why the groups did or didn't reach a different result.

This is just a starting activity; it could be followed by having students do more hypotheticals individually. The instructor needs to give a lot of feedback on the problems as they go, asking students key questions that they might not have thought of, so they'd ideally be well trained in a) rationality and b) drawing out student thinking.

Ideally the exercises wouldn't require too much instructor skill, though, so I'll think about this some more.

Practicing this could be fun in pairs, dissecting an acted out scenario. Two instructors act out previously conceived scenarios, with a Influencer and a Reactor. At some point, 'twill be implied the Reactor wishes to act on the scenario itself or the knowledge presented therein; the scenario will then halt, and the students put in pairs to brainstorm the beneficience and maleficience of possible actions. Each student will take turns (which can be timed) being the brainstormer and the consequentialist (utilitarian?); of course the pairs can have different functions, like as suggested. These just serve to outline the general idea.

For example:
INFLUENCER: Good day, sir! On your way to the place we are going?
REACTOR: Why yes, I am! However odd you too shall be going there; I wish we shall fall upon their fancy!
INFLUENCER: Oh dear! The Gods are weeping once more!
REACTOR: Dear me! I prefer not to be wet, and so I always carry an umbrella upon my person!
INFLUENCER: Indeed, I see it now grasped in your hand! Whatever shall you do? Poor me, if only I were so prepared....
Halt

BRAINSTORMER: He opens his umbrella, and uses it himself.
CONSEQUENTIALIST: The umbrella protects him, but not his companion to any significant degree. (The companion must dodge the edges so as not to be poked in the eye, and may be offended.) The umbrella may wetten things once inside(, and earn him the ire of some people by whom he'd rather not be thought ill.)
Both would then consider the merits and, as outlined by the parentheticals, disadvantages of these outcomes, moving on to the brainstormer's next suggestion afterward.

After each has taken a turn, the instructors would go around the room asking each pair their brainstormed actions, their potential consequences, and the positive and negative aspects of each; as Vladimir suggests, these aspects can ("should") be relativized against each other - if they do relativize, they would state the pair's preferred action and its predicted consequence. The instructors could reinforce correct applications, and constructively criticize incorrect applications, with care taken to not put any pairs down too much (using softeners, etc.: "That's quite creative! We're glad you thought of that, this is an excellent example of how even the best consequentialists can go wrong...").

Identity - "I'm the sort of person who belongs in academia."

This could also be caused be confusing correlation with causation.

sd

The best idea I have for teaching rationality (in the general sense) is to:
1) explain the concepts to people (ie. explain the idea of consequentialist thinking, and the rationale behind it).
2) have people write essays about thoughts/ideas they have (they should be excited to write these essays), and then peer review the essays, pointing out errors in rationality. Like not supporting claims with evidence. Then have an instructor go over the essays and the evaluations to make sure they did a good job.

Also, I think what you're doing right now - crowd sourcing - is probably the best thing for idea generation.

A simple 4 Step Process:

Step 1) write down a list of the consequences

Step 2) => take this list and eliminate all descriptions of actions

Step 3) Eliminate indirect descriptions of what someone has done: honor, merit, guilt, virtue (and all their derivations); this includes personality descriptions like: beeing a good friend, winner, loser, hero, asshole, slut, murderer

Step 4) Eliminate all "consequences" that are defined as the fulfillment or not-fulfillment of plans/goals

"What positive future events does this action cause?"

When reading this, Thomas Sowell's 3 questions came to mind.

• Compared to what?
• At what cost?
• What data do we have?

Without identifying the other options, even identifying something as "the cause" becomes problematic. And it is always problematic because "the cause" is generally used as "that event which I assign the credit or blame to".

If you're only looking for the positive events, you're obviously biasing your search. You should be looking for all consequences first, good, bad, and neutral. Further, by encouraging people to start looking for either the good or bad, they're starting off with a position and attitude first, then the reality of events second.

Finally, you have to consider the data you have to back up your claims of the likely consequences. What's your confidence in the model?

Checking consequentialism should answer Sowell's three basic questions -

• Compared to what? ** What are my options?
• What data do you have? ** What probabilities do I assign to possible states of the universe for each state?
• At what cost? ** What value do I assign to these different states?

(What am I wrong with the asterisks? I wanted my versions of each of his questions to have an extra level of nesting.)

All these things come out automatically if you follow the mathematical formalism for decision theory. That would be my suggestion - have people work out an everyday problem in the formalism of probabilistic decision theory. Show them that the problem has already been solved, and they just have to turn the crank.

One possible exercise:

1. In pairs or in groups one person is asked by instructor, what he or she wants to buy in near future. For example, the person wants new digital camera.

2. Then group should calculate full cost of this camera, including all accessories and expendables.

3. After that people in the group suggest alternative activities and expenses, based on this full cost of digital camera, what the person can buy instead of this camera. For example, the person can buy a bike and ride around, instead buy a camera and take pictures around.

4. Then the person, who wants this camera, can check all alternatives and find the best, Maybe the group can also discuss, why one alternative is better, and another one is worse. For example, will digital camera gives more pleasure, comparing with skateboard? Or usage of some of the proposed items will give more physical exercises and more health to the owner, instead of using another one.

5. When one person from the group will find some explanations or answers to the questions group discuss now, other people should check the rationality of this explanation: is it rational thinking, or is it bias? And one person can also check the reasoning of another person in the group in any time.

Another exercise: Group watch or read some news, proposed by instructor, then analyse the news. Group should discuss the following questions:

• Is it possible, that the event from the news was really happened?
• What details of this event prevent it from happening? What details show, that it looks like the genuine event?
• If we trace this event to the future and to the past, can we find the confirmations or refutations of the discussed event?

Possible exercise: Take one decision, two groups. First group works out all the details of what would happen in either case, best case scenario. Second group works out all the details what would happen in either case, worst case scenario. Don't be afraid to get creative or exaggerate, have fun with it. Then write down the key points, and both groups make their decision.

Then discuss both options between groups, being more realistic.

Is there a difference in approach? Reflect as a group, what have you learned? Will you use this in future decisions? If you have time, try this again but reverse group roles, with a different decision.

Bigger decisions work better, as they have larger consequenses. Try investing as a multinational, or use a current political topic. Controvercy works well if you wish to teach critical thinking without judgement.

So, first of all

The easiest way to help people learn this skill, I think, would be to teach people:

• Good posture

• How to relax and open their muscles and joints

• How to breath properly

And, the easiest way to teach people this skill, I think, is to instead teach them about this skill. This means that exercises should be somewhat indirect. Exercises should definitely get people to experience the problem instead of getting people to learn the solution, and only make available this solution as an option. Partly because the proposed solution is not the only solution - it is not an absolute solution - it does not work in all instances.

Lastly, teach people how to make the connection between their awareness of themselves (real situations) and the method of Checking Consequentialism. Get them to realize that it is applicable and usable almost all of the time, in good or bad situations, not just when something seems wrong or off, not just to improve their situations or make something better. If this isn't done, they won't use it when it's important.

-

Now on to reasons why most attempts at teaching this will fail, and most exercises will not reach their audience in the desired way.

This one is REALLY HARD to teach

Seriously.

The reason this is really hard is because of the concept of habits and defense mechanisms and internal realities (such as the should reality).

As you already stated, when trying to check consequentialism, most people will come up with excuses or other defense mechanisms to maintain their reality, instead of actually checking. You mentioned many ways to overcome that, but they all require recognizing that the person is trying to maintain his/her reality.

When people have a habit, they will often go to extreme lengths to maintain that habit. This is especially true for addictions.

when people have a habit of retreating to a reality (such as the should reality), the teacher has to be very careful not to give them an opportunity to retreat further (don't say "this method should work," or "you should try this". Just plain don't say "should" unless you know what you're doing).

So, we have a problem of habits and defense mechanisms and realities. This is on the level of karma. This means that there are infinite reasons why people will continue doing this, and zero reasons why they will stop. This means that logic will not work. Trying to logic people into learning this skill will not work. There are infinite barriers in the way of teaching someone this skill directly, especially of when to use it.

-

Now, with that said, it should still be possible to teach this as a skill. It's obvious that people won't use it all the time, but if they learn how to use it, they might just use it little by little.

The thing that I do instead of checking consequentialism is that whenever I notice that I am in a cycle, I exit the cycle; I stop participating in the cycle. This requires willpower that most people don't have, and an awareness of the present that most people don't have (and sometimes requires a safety net that many people don't have); however it does not require analytical skills.

The two concepts could be combined, though. It is much easier to discern a cycle than it is to determine whether an idea has gone unchecked or ducked under the radar. If someone finds themselves thinking the same thing multiple times, that person is in a cycle. If someone finds themselves compelled to do something that person has a habit of, that person is in a cycle (a minor addiction). If someone finds themselves emotionally reacting to a trigger word, that person is in a deep cycle related to that word. The thing about unchecked consequentialism is that it's really hard to catch the first time it happens (for each subject), but easy to catch the second time (and if it happens once without being fixed, it WILL happen again in a similar way).

If you want people to learn how to catch it the first time it happens (for each subject before it impacts their lives), you have to teach them how to meditate (the Taoist way, which essentially just means teach people to become aware of themselves and their surroundings). Otherwise, instead teach them how to recognize when it has happened in the past, and how to recognize when it happens again. If you do not teach them meditation, then forget about trying to get them to recognize it the first time it happens.

Man, even after writing all that, I still don't have any good ideas for exercises.

I remember reading about an experiment performed by behavioral economists where person A divides some cash and person B either accepts their division and gets their allocated share of the money or rejects it and neither party gets their allocated share. You could say the consequentialist solution is to always accept the division of money, which most folks don't do, so this could make a good trial exercise. On the other hand, if person A is someone person B is going to have repeated interactions with, one could argue that the social capital of training person A to divide things fairly might be worth forgoing cash... So maybe it wouldn't work in a scenario where the class meets again and again? (Unless things were anonymized somehow...)

There is also the Newcomb's Problem aspect to this, where having taught the class about consequentialism will make it appear as though you have made everyone who is Person B worse off.

Reading up on experiments behavioral economists have done in general seems like it could be a good source of ideas.

I haven't had a chance to test it much, but I think I have an idea. Frame the question as what will happen if I do x that won't happen if I do y. At this point it seems like it should be possible to reuse the mental processes for making beliefs pay rent. Basically typecast "I do x" and "I do y" to beliefs. Then see what experiences I anticipate as a result of the beliefs "I do x" and "I do y". Then determine which set of anticipated experiences has higher utility.

The day-to-day cognitive skills I've mastered most completely (I will not say "rationalist skills," because this is true of my countless irrational skills too) are the ones which I learned during a moment of strong emotion — any emotion, excitement or curiosity or joy or surprise or depression or fear or betrayal.

In the case of this particular skill, it was betrayal. I brought it on myself — the details aren't important; suffice it that I spent two weeks living in the "should-universe" (I like this term) before a rude reminder of reality — but the emotion, the physical neurendocrine experience of betrayal, was quite real. And I've been able to return to it ever since, and if I'm ever in a situation where I might be working from a cached plan, I can relive a hint of it and ask myself, "Now, you don't want to feel that again, do you?"

Unfortunately, this experience strongly ties the five-second skill of "check consequentialism" to the emotion of betrayal in my mind. It is very easy for me to construct social experiments in which the teacher radically betrays her students, and then turns around and says, "Don't let anyone do that to you again!" But that is horrible teaching. It's a lot more difficult for me to imagine what "check consequentialism" would feel like if it carried a strictly positive emotional association, and then extrapolate outward to what kind of social situation would provide that emotional/cognitive link.

Students must abandon a cached plan, and evaluate the real-world consequences of their actions instead, at precisely the moment they get a strong positive emotional charge. Preferably "fun." Preferably in the sense of a party game, not a strategy game: both because people who have learned to win without disrupting social bonds (or who care more about winning than about socialization) have often already learned this skill, and because the moment I construct "winning" as a state which disrupts social bonds, I've set up a false dilemma which misleads my students about what rational thought actually is.

But what's the chain of causation? A dispassionate experimenter times the payoff to correlate with the decision? That seems awfully Pavlovian. Leaving the plan causes a reward which provides an emotional payoff? Maybe, but if a student only leaves the plan in expectation of reward, they haven't actually learned anything beyond the latest professorial password. The excitement of getting the right answer to a puzzle inspires leaving the plan? I suspect this is the way to go. But then what sort of puzzle?

I'm going to press the "Comment" button now, even though I don't think I've contributed much beyond a restatement of your original dilemma. Perhaps having done so, I'll think of some specific scenarios overnight.

Well, this might be a completely trivial suggestion but, if the point is to get people using consequentialist thinking in their lives, why not have them each pick some big important decision in their lives (either an upcoming one or an ongoing one), preferably one they aren't entirely set in already or are uneasy with, so they would be more open to changing their mind, then get into groups and each take turns to discuss their decisions and options (hopefully without applying any judgement to them at this stage), then the other members trying to come up with as many new options as possible. Then they discuss the outcomes and benefits of each option and try to find the best one, not moving on until the entire group feels that the last person's problem was solved adequately. That way not only do they practice consequentialism in a practical way, they might get immediate help with a current problem which could end up being useful on its own and make consequentialist thinking seem more useful.

A fun and social exercise could involve Facebook timeline!