Retrospective on Teaching Rationality Workshops

post by Neel Nanda (neel-nanda-1) · 2021-01-03T17:15:00.479Z · LW · GW · 2 comments

Contents

  Introduction
  Motivation
  Workshops
    Content
    Impact
  Teaching Philosophy
  Doing These Yourself
  Conclusions
  Acknowledgements
  Appendix A: Lesson Content Continued
    Effective Planning
    Building Good Habits
    Building Useful Systems
  Appendix B: Highlighted Testimonials Continued
None
2 comments

TL;DR: I organised a series of afternoon applied rationality workshops for the Cambridge Effective Altruism group based on some core CFAR classes. These went much better than I expected them to, and seem to have added long-term value to participants. My goal in this post is to share my resources and lesson plans, my thoughts on teaching applied rationality well, my attempts to distill the concepts into key ideas & mental habits, and to convince other people to organise similar workshops!

(Disclaimer: This was inspired by CFAR material, but is very much all my interpretation and framings. Any problems with the material are likely from me, not CFAR)

Introduction

I went to a Centre For Applied Rationality (CFAR) workshop last year and found this a really valuable experience. I especially found that a few techniques stuck well with me and became valuable parts of how I thought. So, to help consolidate these and to share them with others, I organised some afternoon workshops for the Cambridge EA group based on my favourite classes. I think these were extremely successful and added more value than I expected to the participants. So this post is my attempt to write up a retrospective on the workshops, what I think was valuable about them, and what I learned from them.

I taught four workshops:

• Having productive disagreements (based on Double Crux)
• Effective planning (based on Murphyjitsu)
• Building good habits (based on Trigger-Action Plans, or TAPs)
• Building useful systems (based on Systemisation)

The format was fairly different from the standard CFAR workshop (90 minute afternoon workshops every 2 weeks, rather than an intense 4 day workshop), which I expected to make it much harder to have an impact. But, based on a followup 2-3 months later with participants, I found that some insights had stuck, and were being used regularly. A significant component of this was that I focused on distilling the techniques down to the key ideas, and useful mental habits. These seemed to stick well with some people, without requiring much followup effort.

Overall, I feel very happy with the impact, and I think afternoon classes are low cost to run (and seem to transfer acceptably to remote). I don’t think a CFAR-style framing of rationality resonates with everybody, but there are enough people who found it high value that I think this was super valuable on average. So I would be very excited to see more people try to run classes like these!

I’d be especially excited to see more EA student groups running them I expect improving the long-term effectiveness of young EAs to be very high leverage, and a significant way for a local group to add value. I’d also expect it to be valuable to the organisers, I found writing and running these workshops extremely helpful for understanding the ideas more deeply myself, and for improving at teaching! And I’ve found myself applying these ideas much more widely in my life.

Motivation

I have a few different goals with this post. As a result this post ended being pretty long, so I’ve tried to write each section to be self-contained, and to indicate which sections different people might find interesting:

• Convincing other people to run workshops like this: Explaining how I did things, and sharing my attempts to assess the longer-term impact on participants
• I’d be especially excited to see other local Effective Altruism or Less Wrong groups running workshops like these.
• See the Doing These Yourself [LW · GW] section for more thoughts on who should do this
• See the Impact [LW(p) · GW(p)]section for my case for why these were worth running (continued in Appendix B [LW · GW])
• Making it easier for other people to run workshops like these: All existing sources for this content I found are focused on explaining the content (eg the CFAR Handbook). I think applied rationality is very much something you learn by doing, and that it works best when taught in a structured environment that makes the default action to practice things, rather than requiring agency. And that comes with clear exercises to guide you through the ideas.
• It took me a fair amount of effort to restructure this content as a coherent lesson plan with exercises. To my knowledge, my lesson plans are the best publicly available sources of this, so I hope this is useful!
• See the Content [LW(p) · GW(p)]section for summaries of the content, and my thoughts on what to emphasise, and common misunderstandings (continued in Appendix A [LW · GW])
• See the Teaching Philosophy [LW · GW] section for my meta-level thoughts behind structuring the lessons the way I did
• Sharing my general teaching philosophy for applied rationality, and specific insights I’ve learned about teaching this stuff well. I mainly focus on making things intuitive and actionable: having clear and concise insights, motivating examples and useful exercises. See the Teaching Philosophy [LW · GW] section for this.
• Sharing my distillations of these four techniques and ideas with anyone interested in rationality
• If you’re interested in the ideas but not in teaching, I recommend the section on Content [LW(p) · GW(p)]and Appendix A [LW · GW]
• If you like how I think about them, I recommend going through the relevant lesson plan and actually doing the exercises and questions!

Workshops

Content

Format: These were 90 minute afternoon classes on weekends, mostly aimed at student EAs (late teens/early twenties). The target audience were people with a prior interest in EA, rationality and optimising their life, but without much specific experience of CFAR techniques.

The following is a rough summary of the key takeaways and structure of my productive disagreements workshop (based on CFAR’s Double Crux class). I’ve tried to go into detail on Pedagogical-Content Knowledge (PCK): knowledge about the topic, how students tend to engage with it, and how to teach it well. Eg examples that worked well, common misconceptions, subtle nuances to emphasise, etc. I think PCK is really useful to teach the classes well, but also very useful to understand the ideas more deeply yourself, even if you don’t intend to teach them. In the interests of space, I’ve put similar sections for my workshops on planning, habits and systems in Appendix A [LW · GW].

• Lesson Plan
• Motivation: I think disagreements should be about truth-seeking, but it is difficult to do this productively. Three major failure modes I wanted to address:
• Focusing on minor disagreements rather than important points
• Not taking each other seriously
• Not talking about the same thing
• Intended takeaways:
• Discussions should be about seeking truth, and understanding the other person’s model that leads them to their beliefs
• You get the most value from a discussion if you’re open to changing your mind and take the other person seriously. A good way to achieve this is to paraphrase the other person’s position to them, and iterate until they say it’s a fair representation
• Often discussions derail because they focus on misunderstandings or minor points. This can be addressed by explicitly identifying and stating your core and most important beliefs. We call these cruxes, and define them as ‘an underlying belief, where if you changed your mind on this belief, you’d change your mind about the point of disagreement’
• Our emotional reactions are an important component in what we believe, and engaging with them is a key part of truth-seeking discussion
• This was inspired by the CFAR Double Crux class (a paired framework for having productive discussions), but I reframed it around techniques you could apply in a discussion with anyone, rather than people explicitly using the same techniques. See the CFAR Handbook for more details
• Techniques:
• Replace the symbol with the substance (tabooing words) - notice load-bearing words, words with a very nuanced and unclear meaning, and replace that word with a definition
• Paraphrasing: Repeat the other person’s position back in your own words. Ask whether it’s correct, get feedback, and iterate until they’re happy.
• Emphasis on ask, you’re probably wrong first time!
• This ensures you’re on the same page, highlights misunderstandings, and helps to take them seriously
• Seeking cruxes: A crux is an underlying belief, where if you changed your mind on that, you’d change your mind on the conclusion. To find them, first freely generate supporting arguments for the conclusion. Then, for each argument, imagine a hypothetical world where that argument is false, and introspect on whether you’d change your mind on the conclusion
• Emphasis on introspect - it’s easy to trick yourself into thinking something should be a crux when it is not your true justification, you just can’t imagine it being wrong.
• Resonate: When hearing a position, first say the point you most agree with in it, before giving any counter arguments
• Structure:
• Introduction: Outlining philosophy & goals of discussion (the goal is to seek truth, and you should assume good-faith until proven otherwise)
• Operationalising: Failure mode 1 is not talking about the same thing
• Resolve this by paraphrasing
• Resolve this by noticing load bearing words and tabooing them
• Resolve this by giving examples and making things concrete
• Exercise: Practice operationalising vague statements
• Seek cruxes: Failure mode 2 is talking about minor points, rather than the root cause of disagreement
• Defining the idea of a crux, giving the algorithm for finding cruxes. This is useful because cruxes expose the models beneath your beliefs, and so are more valuable to focus on.
• Encourage the social norm of saying “my crux is”
• Exercise: Give a list of prompts, have them practice seeking cruxes on a favourite
• Arguing in good-faith: Failure mode 3 is not taking your opponent seriously or trying to learn from them
• Resolve this by paraphrasing
• Exercise: Pair up and practice explaining your cruxes to a partner, and having them paraphrase it back
• Resolve this by steelmanning
• Resolve this by resonating
• PCK:
• It’s easy to get mired in a “conflict vs mistake theory” argument here, I try to address this by clearly outlining a mistake theory framework of assuming good faith at the start
• Feedback says I overdid this, and my specific audience thought that was all obvious. Some friends objected to the mistake theory framing
• The operationalising exercise didn’t work super well, the prompts I generated felt too vague and the task was unclear
• In future, I would scrap the operationalising section, it seemed the weakest
• The cruxes exercise worked really well. Some people were initially skeptical and found the notion of cruxes dull and obvious. But seeing an explicit algorithm and trying it gave a visceral sense of “huh, I understand my position better”
• The paraphrasing exercise worked really well. People had a visceral sense of “huh, that was way harder than I expected”
• Note: Be clear before people seek cruxes that they’ll pair up and share them later, so they shouldn’t pick anything too private
• It was valuable to identify all the techniques as mental reflexes, this helped them stick
• It’s easy to conceive of cruxes as a logical thing, and ignore implicit emotional biases. It is worth emphasising the introspective part, you can’t have a meaningful discussion without understanding your emotional biases.
• If doing again, I’d focus the class more explicitly on cruxes & paraphrasing, these seemed the most valuable takeaways. And give more time for exercises

I tried to heavily frame each workshop around building mental habits and reflexes. Ie, rather than the point being to learn a long, effortful algorithm, breaking the algorithm into bite-sized steps, learning the cues for when each step is relevant, and learning to bring them up in the moment. I think this is a skill best trained with TAPs. And based on the long-term feedback, this was an extremely successful approach! Few participants put in much effort to practice the techniques, but several managed to absorb these mental habits

Impact

People generally enjoyed the workshops, and when asked immediately afterwards gave highly positive feedback. I think the main source of impact is whether people absorb these techniques in the long-term, so I followed up 2-3 months after the workshops, and asked for qualitative feedback on how well the techniques had stuck. I heard back from about 85% of participants.

My best attempt to summarise this data was to loosely categorise people into neutral (no real impact), moderate successes (some long-term benefit) and strong successes (significant long-term benefit, regularly use the ideas in daily life). If you want to try analysing the feedback yourself, you can see my anonymised summaries of all testimonials.

• Productive disagreements: of 6 participants, 2 moderate and 2 strong successes
• Effective planning: of 12 participants, 2 moderate and 2 strong successes
• Building good habits: of 6 participants, 3 moderate successes, no strong successes
• I taught this workshop a few weeks before lockdown began in the UK, which I think was an unusually bad time. I think habits are very tied to your environment and daily routine, and easily break when that significantly changes. Several participants formed habits they found useful, that were then disrupted by lockdown
• Building useful systems: of 11 participants, 5 moderate and 1 strong success

Two highlighted testimonials from the productive disagreements workshop that I’m particularly excited about:

• Highlighted testimonial 1 (Strong):
• Has internalised the idea of cruxes and now does this naturally. Finds this useful for understanding what he believes and why. Finds it notably useful to process unpleasant personal decisions and to change his mind on this
• Intermittently notices “I am confused” in the moment and starts paraphrasing
• Intuitively notices overloaded words and taboos them
• Finds the techniques most useful when thinking alone, he generally finds arguments and debates unpleasant
• Highlighted testimonial 2 (Strong):
• Has internalised the idea that “other people’s minds should make sense” when talking to or thinking about other people
• Puts more effort into understanding other people’s models
• Regularly uses paraphrasing

In the interests of space, I give some highlights of the testimonials I am most excited about for the other workshops in Appendix B [LW · GW]. I think reading things like this is most interesting for gauging the impact of teaching rationality in this format. But I also find that seeing how other people engage with techniques in practice can help me understand them more deeply myself, and help me see how to put them into practice

My prior was that short, one-off classes would not have any noticeable long-term impact, because they would be too short and easily forgotten, and not have a surrounding context of self-improvement to reinforce the ideas and get people to practice. Seeing this long-term feedback has strongly updated me towards thinking these kinds of workshops are valuable. The workshops didn’t have a significant long-term impact on most attendees, but had an impact on enough attendees that it seems extremely worth the total time investment to run and attend them.

I expect that certain kinds of people will get much more benefit from these workshops than others, and I had the useful filters of:

• A Cambridge student (which I expect to somewhat correlate with work ethic and intelligence)
• Involved in EA Cambridge
• Interested in spending weekend afternoons learning about rationality

I expect these filters significantly increased expected benefit

I framed each workshop around a series of Trigger Action Pattern-style mental habits, eg “when I notice I am confused about what somebody is saying -> try paraphrasing it back to them”. I think this worked extremely well, and created a significant amount of the value of these workshops. A lot of the most successful feedback is people for whom these habits stuck, and are now regularly used. As far as I can tell, people didn’t put significant effort into deliberately retaining these habits, they just made intuitive sense and stuck. I am very pleasantly surprised that they stuck this easily. My rough model for this is that people remembered and used the habit shortly after the workshop, found it useful, and this reinforcement kept happening until the habit stuck.

One major weakness is that for the more practical workshops, people rarely put in meaningful effort to practice or retain the techniques after the workshop. Eg, in the systemisation workshop, I had participants design and implement a system in the workshop. Many found this useful, and had the system stick, but far fewer applied the ideas to design more systems afterwards. One guess for addressing this would be to have follow-up workshops entirely focused on applying and practising the techniques, as a form of group accountability. I’d be extremely interested in hearing any other ideas for addressing this problem!

In hindsight, I think most of the value of the workshops came from people doing exercises and practising the ideas, and less so from the content and theory. In future, I’d shift emphasis to spend less time talking, and spend more time on the exercises. Though I think there is still significant benefit to spending some time on theory, and trying to articulate the mindset behind why the techniques make sense and are useful.

Teaching Philosophy

I think teaching is an extremely important skill, and teaching skill and philosophy is responsible for a lot of the variance in how well people learn, so it’s something I try to think about a lot. In this section, I’m going to try to summarise my thoughts on teaching as relevant to applied rationality. If you’re interested, I go into a lot more detail on my thoughts on teaching generally in this blog post

One of the most important parts of my teaching philosophy is that learning is a process of information compression. We take in far more information than we retain. From a 90 minute workshop, most people will retain a few key points tops. This is important, because it means that I should be trying to choose those key points, and shaping the lesson around them. Fundamentally, the entire point of the workshop is to give context and reinforcement to those key points, everything else is irrelevant. Further, extracting those key points from a stream of information is significant intellectual labour. The student doesn’t know what is and is not important, and it takes effort to identify this, effort taken away from actually learning. Thus, as the teacher who does know what is and is not important, my role is to make it as easy as possible to identify these key points. Some strategies to achieve this:

• Explicitly write down the key takeaways I want people to get from each section of the workshop. Default to cutting anything that doesn’t help reinforce this takeaway.
• Summaries! Give an overview of the key points in the introduction and conclusion, and give a recap at the end of each section
• When I introduce an important point, explicitly say it’s important! Eg, saying “this is a really important point, if you don’t understand it, please flag this now so I can go into more detail.”
• Give examples and spend more time on the most important points.
• A good slogan: pace according to difficulty, not length - a common mistake is to just monologue until you get through all the ideas. But the real bottleneck is ensuring students can process and understand the points - it’s worth spending more time on important points, giving examples and alternate framings, even if this means you need to skip details on easier, less important points
• And if a point is hard and unimportant, why are you including it?!

Some ideas for teaching applied rationality specifically:

• Compress the key points into clear, concrete mental habits, of the form “If [condition], then [action]”. Eg, “when I intend to do something in future -> ask ‘would I be surprised if this doesn’t happen?’”
• There should be an explicit context for when the technique will be useful
• Examples are valuable for giving context, and making the idea feel more concrete
• The number one failure mode is that the ideas make sense, but feel abstract. It needs to be actionable, and clearly relevant to their life!
• Give a ton of examples. Illustrate every point with an example, distribute lists of more examples afterwards. Different things are actionable to different people, and examples help to give inspiration
• Every workshop I thought I had too many examples, and got the feedback to include more next time
• Be grounded in day-to-day life, and give relatable, everyday examples. Personal examples are a bonus
• This is part of why it’s important to have practiced a technique a bunch of times yourself before teaching it!
• Lead with relatable, motivating examples - make it clear that the technique can help achieve their goals, and help with their failures
• The goal is always to be immediately intuitive. When you’re trying to reframe how people interpret how their mind works, you’re fighting an uphill battle. There’s often a moment of snap judgement where the idea feels either intuitive or weird, and if it feels weird it’s very hard to be useful. And the exact framing can significantly change how this snap judgement goes
• First, use motivating examples to strongly relate the ideas to their lives. Ideally have several examples across a range of problems, and get people thinking “yes, that is a problem I have and struggle to fix”
• Then, when introducing the specific idea/technique, ensure every step feels motivated, intuitive and grounded in the audience’s internal experience.
• I found beta testing with friends valuable here - try out different framings on different people
• I think this part can determine a lot of the variance in how useful a workshop is, and it is worth putting a significant amount of effort in
• Note: People are complicated, and some people just won’t get value from a technique! This is fine. But I find there are also people who bounce off a technique framed one way, but get a lot of value from another framing
• Different framings work on different people. I like to invite questions after introducing key ideas, and to try different framings as appropriate if people didn’t like the original one
• Examples! I’ve said it a bunch of times already, but it’s so important it’s worth saying again. A class with only examples can work, a class with no examples will not.
• I think about examples in 3 different ways:
• Motivating examples - why you care about these ideas at all. These are often a good hook to begin the session with
• Illustrative examples - a long example that can be broken down to illustrate an idea
• Micro-examples - short examples given after a point to illustrate. This should be 1 clause (2 at worst) and be immediately intuitive - if you ever need to explain it, it’s a bad micro-example
• Teaching is hard, because it’s a lossy process of information transfer over a noisy channel. Examples are a different way of communicating information than explanations, and so are invaluable as a way to detect and correct errors
• The ideas should be embedded in their mind. It needs to be active, reflexive knowledge, not abstract academic knowledge. This is because most of the impact comes from recognising a problem in the moment and doing something about it
• Ask questions!
• It’s easy for people to be passive. I like to ask for a hand-poll (raise your hand to a point proportional to how much you agree with the statement) so everyone has to think
• Alternately, say you’ll pick on people at random
• Note: Ask questions one at a time - you want it to be clear what the attendees should be thinking about
• Give exercises! You want to make the ideas reflexive, so this needs to be something people practice and actually do. This lowers activation energy for actually using it in future
• Exercises also highlight misunderstandings, when it’s still easy to resolve them
• I like paired exercises, it’s far less easy to zone out (though warn people in advance!)
• It really helps to get people to think about the ideas, relate them to their life and their personal context
• One of my favourite exercises is getting people to generate ways the problems arise in their lives, or where they can apply the technique. This also means they leave the session with a list of other places to apply the techniques!
• You are teaching guidelines, not rules. Emphasise this frequently and repeatedly. People are complex, and different approaches work best for different people.
• When people to exercises, encourage them to experiment with and adapt the techniques, try to model this yourself

Doing These Yourself

Overall, I think these workshops were a major success at actually conveying the techniques to the audience. I’ve gotten a lot of value from these ideas in my personal life, empirically they’re teachable, and I’d be excited about seeing these insights spreading and helping others to become more effective! I also found teaching these to be valuable personally, because it significantly clarified the ideas in my head. I’ve noticed myself using the ideas much more often in normal life as a result.

My lesson plans are here. I expect these to work best as a source to ad-lib from and adapt, rather than to follow perfectly, but I designed them to be detailed and thorough, so I hope they can save a significant amount of work! EA Stanford have run two of these workshops based on my lesson plans, and seemed to think it went well and was much lower effort to run than if writing workshops from scratch.

If you want to run one of these, to prepare, I would recommend making a copy of these notes, reading through in detail, and editing it to be in your voice. Eg, noticing the framings you dislike and changing them, replacing my examples with ones you relate with, noticing the points you don’t understand and cutting or thinking more about them. I think it’s important to understand what’s in the plan and to have it be something that makes sense to you, before trying to teach it to others. I generally err on the side of putting too much content into lesson plans, and cutting things when presenting (or overrunning, or both). I’d recommend cutting the parts of the lesson that don’t seem as interesting or exciting to you.

I’d be especially excited to see the productive disagreements workshop done in EA groups. To me, a key part of EA culture is having good epistemics - taking other people’s ideas seriously, trying to understand them, and being open to changing your mind. But I rarely see this norm explicitly set or taught, it seems more something that people are either already on board with, or pick up by osmosis. And I’m excited about seeing attempts to explicitly set culture.

Further thoughts on how to actually do this:

• Who should teach this?
• I think the most important part is to understand the ideas well yourself. To have actually practiced and applied them in your life, to have a sense for the nuance, benefits and limitations.
• I think this is important because there’s a lot of subtlety and nuance to how the techniques are taught. This manifests in what parts of the techniques you emphasise, how you answer questions, help people struggling with the techniques, etc.
• I think it’d be especially good to have been to a CFAR workshop and connected with the ideas there. But I expect the more important part is to have actively practiced the techniques, found them useful in everyday life, and experienced the ways they can be hard to use or fail to be useful.
• A rough litmus test: Have you used the technique regularly for at least a month after you learned it, and have you found it useful?
• Downside risk
• Overall, I don’t think there’s high downside risk from other people teaching this stuff (if I did, I wouldn’t make this post!). My guess is that most of the downside risk comes from techniques that focus on introspection and motivational issues
• I expect the biggest downside of these techniques would be to use them to approach a naive idea of productivity in unhealthy ways. Eg, training a TAP for “when procrastinating, stop” and generally feel guiltier when procrastinating. Or to systematise your life in a way that doesn’t account for breaks and leads to longer term burn-out
• I don’t strongly trust my inside view on this, so I’d love to hear other people’s thoughts!
• Some guesses for ways teaching rationality could go badly:
• The instructor focuses on copying the structure of a CFAR class, rather than what’s real and useful, and participants learn a vague version of a technique that feels fake/forced. And this makes learning the technique well harder later (CFAR calls this ‘idea inoculation’)
• A student thinks rationality is all about trusting logic and ignoring emotions and anything that feels ‘irrational’, and makes unhealthy decisions (my take on why this is an error)
• A student labels part of their current mindset as ‘bad habits’ and tries to ‘fix’ them, and these habits are actually focused on protecting something important, which the student hasn’t dug deep enough to realise
• Remote settings
• I did these workshops in person. I tried re-running the disagreements one online, and it seemed to go well. Overall I think these are less ideal remotely, but still basically work
• Apparently the worst part of it being remote was my frequent complaints that remote workshops are terrible. I have since updated this belief!
• I think having Zoom breakout rooms (or equivalent) for paired exercises is essential for this to work well
• I think the most significant thing lost is the ability to easily judge audience reactions, people disengage more easily, and it’s harder to monitor how people are doing in the exercises
• I added an introductory question (“what are things you struggle with when having disagreements, and what do you find helpful?”), gave people a minute to think about it and told them I’d be picking people at random to answer. At least one person said the “I’ll pick somebody at random” part helped them feel engaged
• Encourage people to have video on, this significantly increases engagement.
• Have slides to illustrate key points/key algorithms/exercises (though err on the side of too little text, not too much)
• Further tips
• Beware the typical mind fallacy. People are complex and different, but it’s easy to implicitly assume they think exactly like you, find the same techniques useful, etc
• Encourage questions, to notice these differences!
• Do trial runs! I find doing a practice run with a friend valuable for getting more fluent, noticing exercises and framings that feel clunky or repetitive, etc
• Pace according to difficulty, not to length
• I think it’s easy to pace a lesson according to how long the content takes to say. This is an error, because the main bottleneck is how long students take to understand it.
• Rate each section/idea in the talk out of 5 for difficulty, and pad the hard ones with more examples, framings, etc, and cut things out of the easy ones
• Optimise for long-term impact
• I think most of the expected value of the workshops to come from techniques sticking super well for a handful of people, rather than weakly for everyone
• This means a lot of time should go for exercises, they help the techniques feel like useful knowledge, not academic knowledge

Conclusions

Overall, I think this was an excellent experiment! These ideas have become a powerful part of my mental toolkit, and I’ve been able to transfer them to others. My priors were against them being transferrable in this kind of context, but I’ve strongly updated in favour after doing long-term followups!

I’d be very excited to see other local groups trying to run these, and I hope my lesson plans can save some effort there. If you plan on organising these, I’d be very happy to chat and give any advice! Please feel totally free to reach out, and I’d be extremely interested in hearing how it goes if anybody does try them. My email is neelnanda27@gmail.com

Acknowledgements

Thanks to all of the many people who took the time to give feedback on the draft! Especially Dan Keys, Luca Righetti, Nora Ammann and Nathan Young, who helped make this significantly better. And thanks to CFAR for introducing me to these ideas in the first place! A year on, I've gotten a ton of value from my workshop

Appendix A: Lesson Content Continued

The continuation of the content section for my workshops on planning, habits and systems. I summarise the motivation, key takeaways and structure of the workshop, and try to give Pedagogical-Content Knowledge - common misconceptions and specific insights for teaching the ideas well.

Effective Planning

• Lesson Plan
• Motivation: Humans are systematically bad at planning. Eg, the student who leaves every essay to the last minute and needs to pull an all-nighter.
• This is a systematic failure in planning, because she had all the information she needed to conclude this would happen (historical data), but didn’t realise this in the moment. So the solution is to train your intuitions to better use your existing experiences to notice flaws in your plans
• Intended takeaways:
• Most of the things we want to do in the future will fail to happen. Often the key failure is that we never even get around to explicitly planning them, and leave it as a vague (“I should … some time”). To improve at planning, we need to improve our vague plans, not just improve our explicit planning skills
• We intuitively flinch away from making robust, working plans. This can be resolved by framing things differently, and correctly using our intuitive knowledge of how plans will fail, since we know what will go wrong on some level.
• This is a problem that needs to be solved in the moment, by building better reflexive reactions to planning and procrastination. This requires different solutions than just “trying harder” or “thinking differently”, engaging with intuitions takes a different framing
• Algorithm:
• Find a task/goal to plan
• Make a plan (it can be super vague & rough)
• Design a picture of failure - a concrete picture of yourself in the future, knowing that the plan failed, but without fleshing out why.
• Technique 1, Surprise: Ask yourself “am I surprised that this picture came about?”
• This is an introspective question - the surprise should be a visceral emotion
• Technique 2, Pre-hindsight: Ask yourself “suppose this picture did come about. What went wrong?”
• Emphasis: You assume it fails, and use hindsight to explain why. You aren’t asking whether it fails or not, it’s purely a hypothetical
• Using this information, patch the plan! Iterate until you feel surprise if it fails
• Structure:
• Introduce the ideas with relatable, motivating examples about how human intuitions suck at planning by default. Emphasise that anything you want done in future is a plan, even if you don’t put in effort to formally plan
• Introduce the idea of the Inner Simulator - a part of your unconscious mind that is excellent at modelling the world
• People generate a list of things they want to plan
• Conveniently, this means people leave with a list of other ways to practice the technique
• Explain the algorithm, lead people through step by step, and after each step, give them time to apply it to their plan
• Exercise: Pair people up, have them practice helping each other patch their plans and iterate
• I’m super excited about people getting better at planning - I think it’s a key subskill for making deliberate positive changes in your life (eg, from the other workshops!), career planning, overcoming procrastination etc. And that it can become a reflexive part of daily life, with enough practice.
• PCK:
• It’s super important to get people to practice the technique. Exercises should be over half the length of the workshop
• Paired exercises work well - it’s easier to spot flaws from the outside, easier to stick to the algorithm with somebody prompting you, and people take it more seriously
• People often think of planning as “only for high-effort things where I write out a plan”. Emphasise that hearing a cool idea and deciding to do it some day is a plan, it’s just a bad one
• The algorithm as framed feels high-effort and discouraging, people can’t imagine using it in everyday life. Emphasise that the end-goal is to build mental habits, applying surprise and pre-hindsight in the moment, and the algorithm is short-term effort to reinforce these habits, and for really important plans.
• Reflexive use of this technique looks like having much better calibrated expectations of yourself, and what you are and are not likely to actually do.
• It’s easy to frame it as only an anti-procrastination tool, but it’s more broadly useful. I lead with a motivating example about someone intensely preparing for a job interview, who gets stressed and screws up. And how they could have foreseen this
• Pre-hindsight feels kinda odd at first, like creating information from nothing. I motivate it with the example of regretting an email immediately after sending it, to highlight the difference between foresight and hindsight
• Some people bounce off the name Murphyjitsu, so I avoided it
• The technique can feel unnecessarily convoluted, and like a weird ritual. It helped to frame it as better accessing some intuitions, and that the interface required to do this well centres more on imagery and emotions, than it does on explicitly verbalised thoughts.

Building Good Habits

• Lesson Plan
• This was a class on Trigger-Action Plans (called implementation intentions in the psychological literature), a way to install if-then reflexes in your mind. Where you associate a concrete trigger with the reflex to perform a concrete action, eg “when I enter the library, turn off my phone”.
• Motivation: I think a core skill of rationality is adjusting your behaviour in the moment [LW · GW], because many errors happen without conscious attention. I am extremely excited about TAPs as a way to adjust your behaviour in the moment, and they’re my main tool for doing so
• Intended takeaways:
• It is valuable to think about how you reflexively react to things, and this is a key step to understanding and solving many problems. This requires a different approach to thinking analytically and abstractly about problems, since it’s about connecting with your System 1
• Most of our reflexive behaviour can be broken down simple if-then patterns. These have clear, concrete triggers and clear, concrete actions. This can help us to understand our behaviour, and to design new habits that stick
• More generally, habits are something you can cultivate and shape, rather than something that just happens to us
• Reflexive habits are most useful with leverage. Finding weak points early on, where a habit can be a small nudge and change default behaviour for the better
• Building and changing habits takes time, effort and maintenance in the short term, but this is worth it if it sticks in the long-term
• Structure:
• Introducing the idea of habits and TAPs. Making it intuitive that this is a natural part of how we think
• Framing them as small nudges, and talking about where they’re useful. Participants generate a list of problems to apply TAPs to
• Talk about what triggers work well, and have participants design a trigger
• Good triggers are obvious (it’s immediately clear when it happens), visceral (there’s clear sensory detail attached) and reliable (it happens when you want the behaviour change to happen)
• Talk about what actions work well, and have participants design one
• Good actions are obvious (you know exactly what to do), atomic (it’s short, simple and can be done in <5sec) and consistent (it’s something you can always do)
• You build a habit by explicitly practicing it 10 times, and then have a learning period where you keep it at the top of your mind and track it. Participants then build their habit
• PCK:
• TAPs are hard to teach, because they’re often a new concept without obvious analogies. The word habit often connotes routines, eg “I go to the gym once a week”. It helped to explicitly emphasis this distinction
• It also helps to ask the audience for examples of habits, and to repeat it back in the framing of “trigger -> action”
• Intuition pump: To shift from inaction to action, something must have changed, and that thing is the trigger. Eg the habit “stretching when I’m stiff” is “notice I am stiff -> stretch”
• Most people will not actually put in the effort to ensure habits stick. It helps to emphasise that it’s short-term effort to ensure it sticks long-term
• Habits break super easily with a change in life context, eg moving place. It’s useful to emphasise this, and say that it takes attention and care to avoid this failure mode
• Most habits built by participants broke after lockdown
• It’s easy to think TAPs are cool in the abstract but to never act upon this because it doesn’t feel relevant. This can be resolved by filling the workshop with examples, as diverse as possible. I try to illustrate every point with a different micro-example
• And link to a bunch more at the end
• Eg diverse examples like “take a bite of food -> appreciate the taste” were found helpful
• Even after all this, the biggest problem I found in long-term followup was participants not having clear ways to use it in their life
• The most common mistake is not being specific and concrete enough with the trigger & action
• Possible exercise idea: Pair people up, tell them to keep asking the other “can you make that more concrete”
• I think I’d add a paired exercise in future workshops
• Actions tend to be too ambitious, often high-effort, take more than 5 seconds, and aren’t always doable
• Emphasise it should be a reflex. If you need to ask “should I do this?”, it’s already not working
• This seems common with TAPs for social situations, people often feel uncomfortable taking an action the other person can perceive
• Emphasise that TAPs are small nudges, not brute force. A good litmus test is “if a friend tapped me on the shoulder and told me to take the action, would the problem feel solved?”
• There are many points about a good TAP. Emphasise that these are guidelines - they make it stick more easily, but you rarely satisfy all of them
• The idea of practicing 10 times can feel silly/unnecessary. I frame it as “carving a groove in your mind”, or as “your system 1 is good at pattern spotting. This gives it data to infer a reflex from”
• Emphasise that it’s totally fine to forget a habit sometimes, if it triggers sometimes that’s still a victory! Set the neutral point at “my habit never triggers”, not “my habit triggers perfectly”
• On the margin people are too ambitious and impatient, and will try to train many TAPs at once. Recommend doing one at a time
• A good framing is Tortoise Skills - a TAP is something you can have training in the background without much active effort. There’s no need for impatience, this is already a free win!
• Some people find it unintuitive that our minds work on such a computational ‘if X then Y’ level. I began with an example of this behaviour in wasps to make it clear that it happens in animals too
• Often people don’t get excited about habits, they feel too small and underwhelming to be worth the effort to work on. I like to emphasise the importance of tiny gains compounding over time, and having a major impact over your life. A good motivating article
• I’m most excited about TAPs being used to build mental habits, as I talk about in this post on Noticing [LW · GW]. I had a chapter on that, but didn’t have time to talk about it. In future, I think I’d spin that out into a separate workshop focusing more on Noticing (converting an emotion/mental phenomena into a visceral trigger)

Building Useful Systems

• Lesson Plan
• Motivation: A lot of important things in life take willpower and energy, eg remembering to exercise regularly and eat healthily. By building systems, eg a routine of going to the gym every Sunday morning, or having healthy food automatically delivered to your house, you can achieve these outcomes without needing to invest as much effort.
• More generally, the workshop was on the mindset of thinking in systems: thinking of your effort and willpower as a limited resource, and finding ways to shape your future actions so that you do the right thing without requiring as much willpower.
• Intended takeaways:
• Life is full of trade-offs. You have limited time, energy, attention etc, and need to consider opportunity costs when allocating them, rather than trying to do everything. Many problems can be best diagnosed by understanding how they’re spending your resources
• Willpower is a limited resource, and it is important to notice where it’s spent and which areas of your life consume too much of it
• It takes willpower to deviate from the default action. A sustainable, long-term solution needs to change your default action to be better, and you should pay a lot of attention to your default actions and how to shape them.
• If your default actions are bad, this is something to be changed, not something to feel guilty about
• It is not sustainable to fix your problems by just ‘trying harder’
• Building a good system takes iteration and creativity, there are many options beyond the obvious, and it’s worth looking for them
• To build good systems, you need to put a meaningful amount of effort into actually implementing it and reviewing it.
• Structure:
• Introduce the idea of thinking in your life as abstract resources: health, time, energy, attention, willpower, sleep, etc (what CFAR calls units of exchange)
• Emphasise that this is useful for identifying bottlenecks (resources you lack/are wasting) and for diagnosing problems (eg, am I short on time or just stressed)
• I think this was valuable & framed the class well, but was too long and not 100% necessary
• Introduce the mindset of thinking in systems. Define a system as “anything where you can spend willpower now to reduce willpower needed in future”
• Strongly emphasise friction and trivial inconveniences as important
• Strongly emphasise the importance of the default action, and systems as changing the default
• System design
• Exercise: Participants generate problems to systematise
• Exercise: Participants understand the problem better by generating examples and doing a mindful walk through
• Emphasise introspection and understanding the true behaviour, not the desired behaviour
• Exercise: Design a rough system. Emphasise that it should be reliable (happens by default) and effortless (doesn’t require much effort to follow)
• Reality check: Use pre-hindsight, run through the system, notice problems, iterate and patch
• Emphasise that making robust systems is hard, and that the first draft likely fails
• Exercise: Pair people up and have them practice
• System implementation
• Emphasise the importance of doing something to set up the system
• Exercise: Set a 5 minute timer, have participants start implementing their system
• PCK:
• There is a lot of room for creativity here
• Emphasise that systems are super personal and specific to your tastes & habits
• Emphasise that systems aren’t just routines. I give the examples of triage (quitting your least important course/commitment) and automating things
• It’s easy to agree in the abstract but for the ideas to not feel actionable. Fill the workshop with examples, as diverse as possible
• It’s easy to be neurotic and feel guilty at failure to meet your standards, and resolve to try harder. Emphasise the mindset of “failure is a problem with your default actions. Fix this by changing the default”
• It helps but is not essential to teach this after the planning workshop, pre-hindsight is a useful subskill
• The key idea of units of exchange is not to precisely measure things, it’s to highlight hidden costs and bottlenecks. If it motivates a decision, that decision should feel obvious
• A common failure mode is to fail to implement systems. The 5 minutes of “implement it now” time were valuable, and many participants left with worthwhile systems. Emphasise this
• A future idea would be follow-up workshops that are just practicing the ideas and making new systems, many participants never designed systems outside of the workshop
• There is unusually high value from pairing up to discuss systems and getting another perspective. I think I added significant value by suggesting tweaks to participant’s systems (eg using Toggl for time-tracking)
• Mention costs of systematising, it’s only worth doing for problems that come up frequently (relevant xkcd)
• Emphasise the importance of trivial inconveniences, this isn’t obvious to some people
• The idea of “make a system feel sacrosanct” resonated well with some people - shifting from “do I do this or not?” to it not even feeling like a decision
• The reality check is worth emphasising, it’s easy to design a system that works in an ideal world, but not in practice.
• The process outlined is high-effort, and this can put people off. Emphasise that the point is to practice and reinforce the mental habits of thinking in systems, and these are what matter:
• Noticing and fixing resource bottlenecks, and drains on your energy and attention
• Noticing and fixing inefficiencies
• Noticing and fixing minor inconveniences and trivial inefficiencies
• Noticing when a problem stems from a bad default action

Appendix B: Highlighted Testimonials Continued

Some more examples of the testimonials I’m most excited about from the other 3 workshops:

• Effective planning: of 12 participants, 2 moderate and 2 strong successes
• Highlighted testimonial 1 (Strong):
• Has formally applied the techniques to every important plan since (5 or 6 plans, took significant effort but felt valuable)
• Has absorbed the mental reflex of “if this plan fails, what happened?” and uses it on a regular basis when planning smaller things, and finds this valuable
• Found the ideas simple, intuitive and obviously powerful, so found them easy to absorb without much effort
• Building good habits: of 6 participants, 3 moderate successes, no strong successes
• Highlighted testimonial 1 (Moderate):
• Built several habits shortly after the workshop
• Successfully took effort to ensure they stuck, eg leaving themselves post-it note reminders
• These broke somewhat when life was disrupted by quarantine, only one has stuck since
• Feels like they have a better mindset around habits now, and that that has stuck
• I taught this workshop a few weeks before lockdown began in the UK, which I think was an unusually bad time. I think habits are very tied to your environment and daily routine, and easily break when that significantly changes. Several participants formed habits they found useful, that were then disrupted by lockdown
• Building useful systems: of 11 participants, 5 moderate and 1 strong success
• Highlighted testimonial 1 (Strong):
• Made a system in the workshop, it stuck moderately well
• Now feels significantly better at self-compassion: when he has an unproductive day, thinks about how to adapt the environment to avoid this happening again rather than blaming himself
• Eg he’s started using a LifeRPG app (Habitica) to track to-dos, finds this useful
• Eg he’s reduced procrastination by adding a 20 second delay to opening distracting websites
• Estimates a 30% probability a similar change would have happened without the workshop
• Also found reading the Replacing Guilt series made this shift easier
• Highlighted testimonial 2 (Strong):
• Internalised the idea of taking trivial inconveniences seriously
• Internalised the idea of setting things up to require minimal willpower
• Was nudged towards installing a fitness app with scheduled training sessions, and refusing to ever miss a session (going strong at 5 weeks!)
• Highlighted testimonial 3 (Moderate):
• Built a robust to-do list system in the workshop, still finding it notably useful months later
• Found the workshop valuable, but hasn’t explicitly tried designing another system since
• Highlighted testimonial 4 (Strong):
• Wanted to address the problem of being busy and having too many commitments, started using Toggl to track time and better evaluate opportunity costs
• 6 month follow-up: This has become a core part of their workflow, and been notably helpful at encouraging them to balance their life and say not to things
• Not sure how well the underlying ideas have stuck for developing future systems

2 comments

Comments sorted by top scores.