CFAR Takeaways: Andrew Critch

POV, this is the key observation that should've, and should still, instigate a basic attempt to model what humans actually are and what is actually up in today's humans.  It's too basic a confusion/surprise to respond to by patching the symptoms without understanding what's underneath.

On one hand, when you say it like that, it does seem pretty significant.

I'm not sure I think there's that much confusion to explain? Like, my mainline story here is:

1. Humans are mostly a kludge of impulses which vary in terms of how coherent / agentic they are. Most of them have wants that are fairly basic, and don't lend themselves super well to strategic thinking. (I think most of them also consider strategic thinking sort of uncomfortable/painful). This isn't that weird, because, like, having any kind of agency at all is an anomaly. Most animals have only limited agency and wanting-ness.
2. There's some selection effect where the people who might want to start Rationality Orgs are more agentic, have more complex goals, and find deliberate thinking about their wants and goals more natural/fun/rewarding.  
3. The "confusion" is mostly a typical mind error on the part of people like us, and if you look at evolution the state of most humans isn't actually that weird or surprising.

Perhaps something I'm missing or confused about is what exactly Critch (or, you, if applicable?) mean by "people don't seem to want things." I maybe am surprised that the filtering effect of people who showed up at CFAR workshops or similar still didn't want things.

Can you say a bit more about what you've experienced, and what felt surprising or confusing about it?

↑ comment by AnnaSalamon · 2024-02-25T21:09:26.354Z · LW(p) · GW(p)

Some partial responses (speaking only for myself):

1.  If humans are mostly a kludge of impulses, including the humans you are training, then... what exactly are you hoping to empower using "rationality training"?  I mean, what wants-or-whatever will they act on after your training?  What about your "rationality training" will lead them to take actions as though they want things?  What will the results be?

1b.  To illustrate what I mean: once I taught a rationality technique to SPARC high schoolers (probably the first year of SPARC, not sure; I was young and naive).  Once of the steps in the process involved picking a goal.  After walking them through all the steps, I asked for examples of how it had gone, and was surprised to find that almost all of them had picked such goals as "start my homework earlier, instead of successfully getting it done at the last minute and doing recreational math meanwhile"... which I'm pretty sure was not their goal in any wholesome sense, but was more like ambient words floating around that they had some social allegiance to.  I worry that if you "teach" "rationality" to adults who do not have wants, without properly noticing that they don't have wants, you set them up to be better-hijacked by the local memeset (and to better camouflage themselves as "really caring about AI risk" or whatever) in ways that won't do anybody any good because the words that are taking the place of wants don't have enough intelligence/depth/wisdom in them.

2.  My guess is that the degree of not-wanting that is seen among many members of the professional and managerial classes in today's anglosphere is more extreme than the historical normal, on some dimensions.  I think this partially because:

a.  IME, my friends and I as 8-year-olds had more wanting than I see in CFAR participants a lot of the time.  My friends were kids who happened to live on the same street as me growing up, so probably pretty normal.  We did have more free time than typical adults.

i.  I partially mean: we would've reported wanting things more often, and an observer with normal empathy would on my best guess have been like "yes it does seem like these kids wish they could go out and play 4-square" or whatever.  (Like, wanting you can feel in your body as you watch someone, as with a dog who really wants a bone or something).

ii.  I also mean: we tinkered, toward figuring out the things we wanted (e.g. rigging the rules different ways to try to make the 4-square game work in a way that was fun for kids of mixed ages, by figuring out laxer rules for the younger ones), and we had fun doing it.  (It's harder to claim this is different from the adults, but, like, it was fun and spontaneous and not because we were trying to mimic virtue; it was also this way when we saved up for toys we wanted.  I agree this point may not be super persuasive though.)

b.  IME, a lot of people act more like they/we want things when on a multi-day camping trip without phones/internet/work.  (Maybe like Critch's post about allowing oneself to get bored?)

c.  I myself have had periods of wanting things, and have had periods of long, bleached-out not-really-wanting-things-but-acting-pretty-"agentically"-anyway.  Burnout, I guess, though with all my CFAR techniques and such I could be pretty agentic-looking while quite burnt out.  The latter looks to me more like the worlds a lot of people today seem to me to be in, partly from talking to them about it, though people vary of course and hard to know.

d.  I have a theoretical model in which there are supposed to be cycles of yang and then yin, of goal-seeking effort and then finding the goal has become no-longer-compelling and resting / getting board / similar until a new goal comes along that is more compelling.  CFAR/AIRCS participants and similar people today seem to me to often try to stop this process -- people caffeinate, try to work full days, try to have goals all the time and make progress all the time, and on a large scale there's efforts to mess with the currency to prevent economic slumps.  I think there's a pattern to where good goals/wanting come from that isn't much respected.  I also think there's a lot of memes trying to hijack people [LW(p) · GW(p)], and a lot of memetic control structures that get upset when members of the professional and managerial classes think/talk/want without filtering their thoughts carefully through "will this be okay-looking" filters.

All of the above leaves me with a belief that the kinds of not-wanting we see are more "living human animals stuck in a matrix that leaves them very little slack to recover and have normal wants, with most of their 'conversation' and 'attempts to acquire rationality techniques' being hijacked by the matrix they're in rather than being earnest contact with the living animals inside" and less "this is simple ignorance from critters who're just barely figuring out intelligence but who will follow their hearts better and better as you give them more tools."

Apologies for how I'm probably not making much sense; happy to try other formats.

Replies from: AnnaSalamon, Wei_Dai, TekhneMakre

↑ comment by AnnaSalamon · 2024-02-26T04:04:00.796Z · LW(p) · GW(p)

A related tweet by Qiaochu:

(I don't necessarily agree with QC's interpretation of what was going on as people talked about "agency" -- I empathize some, but empathize also with e.g. Kaj's comment in a reply that Kaj doesn't recognize this at from Kaj's 2018 CFAR mentorship training, did not find pressures there to coerce particular kinds of thinking).

My point in quoting this is more like: if people don't have much wanting of their own, and are immersed in an ambient culture that has opinions on what they should "want," experiences such as QC's seem sorta like the thing to expect.  Which is at least a bit corroborated by QC reporting it.

Replies from: pktechgirl

↑ comment by Elizabeth (pktechgirl) · 2024-02-27T02:03:41.366Z · LW(p) · GW(p)

I'm not sure if this is a disagreement or supporting evidence, but: I remember you saying you didn't want to teach SPARC kids too much [word similar to agency but not quite that. Maybe good at executing plans?], because they'd just use it to [coerce] themselves more. This was definitely before covid, maybe as far back as 2015 or 2016. I'm almost certain it was before QC even joined CFAR. It was a helpful moment for me. 

↑ comment by Wei Dai (Wei_Dai) · 2024-02-28T07:18:12.131Z · LW(p) · GW(p)

If humans are mostly a kludge of impulses, including the humans you are training, then... what exactly are you hoping to empower using "rationality training"? I mean, what wants-or-whatever will they act on after your training? What about your "rationality training" will lead them to take actions as though they want things? What will the results be?

To give a straight answer to this, if I was doing rationality training (if I was agenty enough to do something like that), I'd have the goal that the trainees finish the training with the realization that they don't know what they want or don't currently want anything, but they may eventually figure out what they want or want something, and therefore in the interim they should accumulate resources/optionality, avoid doing harm (things that eventually might be considered irreversibly harmful), and push towards eventually figuring out what they want. And I'd probably also teach a bunch of things to mitigate the risk that the trainees too easily convince themselves that they've figured out what they want.

Replies from: Kaj_Sotala

↑ comment by Kaj_Sotala · 2024-02-28T18:29:57.948Z · LW(p) · GW(p)

There's something about this framing that feels off to me and makes me worry that it could be counterproductive. I think my main concerns are something like:

1) People often figure out what they want by pursuing things they think they want and then updating on the outcomes. So making them less certain about their wants might prevent them from pursuing the things that would give them the information for actually figuring it out.

2) I think that people's wants are often underdetermined and they could end up wanting many different things based on their choices. E.g. most people could probably be happy in many different kinds of careers that were almost entirely unlike each other, if they just picked one that offered decent working conditions and committed to it. I think this is true for a lot of things that people might potentially want, but to me the framing of "figure out what you want" implies that people's wants are a lot more static than this.

I think this 80K article expresses these kinds of ideas pretty well in the context of career choice:

The third problem [with the advice of "follow your passion"] is that it makes it sound like you can work out the right career for you in a flash of insight. Just think deeply about what truly most motivates you, and you’ll realise your “true calling”. However, research shows we’re bad at predicting what will make us happiest ahead of time, and where we’ll perform best. When it comes to career decisions, our gut is often unreliable. Rather than reflecting on your passions, if you want to find a great career, you need to go and try lots of things.

The fourth problem is that it can make people needlessly limit their options. If you’re interested in literature, it’s easy to think you must become a writer to have a satisfying career, and ignore other options.

But in fact, you can start a career in a new area. If your work helps others, you practice to get good at it, you work on engaging tasks, and you work with people you like, then you’ll become passionate about it. The ingredients of a dream job we’ve found are most supported by the evidence, are all about the context of the work, not the content. Ten years ago, we would have never imagined being passionate about giving career advice, but here we are, writing this article.

Many successful people are passionate, but often their passion developed alongside their success, rather than coming first. Steve Jobs started out passionate about zen buddhism. He got into technology as a way to make some quick cash. But as he became successful, his passion grew, until he became the most famous advocate of “doing what you love”.

↑ comment by TekhneMakre · 2024-02-27T06:30:30.659Z · LW(p) · GW(p)

It makes sense, but I think it's missing that adults who try to want in the current social world get triggered and/or traumatized as fuck because everyone else is behaving the way you describe.

Replies from: AnnaSalamon

↑ comment by AnnaSalamon · 2024-02-27T06:34:30.191Z · LW(p) · GW(p)

Totally.  Yes.

What do people here think about Robin Hanson's view, for example as elaborated by him and Kevin Simler in the book Elephant in the Brain? I've seen surprisingly few mentions/discussions of this over the years in the LW-adjacent sphere, despite Hanson being an important forerunner of the modern rationalist movement.

One of his main theses, that humans are strategic self-deceivers, seems particularly important (in the "big if true" way), yet downplayed/obscure.

I'd also broadly say of the meta update: I think human intuitions aren't fine tuned for happening to be in a +2-3sd brain loadout, so they aren't very good at cueing us to actually use those features reliably.

That's an interesting point I hadn't thought about before.

. The specific operation that happened was applying ooda loops to the concept of ooda loops.

I love this!

Yeah. This is a real problem and the world is full of people leaving $100 bills on the table, but I can think of situations I expect the other person would describe as "Elizabeth isn't even trying to solve her problem" and I would describe as "Alice jumped to idiot-level solutions without even a token attempt to understand the context, she is clearly not going to be helpful and there isn't a point in trying to bring her up to speed". I suspect this is common, which is why "the kind of person who would suggest yoga for depression" is a codified insult. 

And self-development instructors can be the absolute worst at this. They get used to helping people with low-hanging fruit and when their tricks don't work they blame the participant for not trying^[1], instead of their algorithm being inadequate to a particular person's problems. 

1. ^{^}

   I'm not asserting that's what happened with Critch- there definitely are lots of people who just aren't trying and that's worth pointing out. Although in many cases I suspect there are underlying reasons that at least felt rational at one point.  

Okay, maybe?  But I've also often been "real into that" in the sense that it resolves a dissonance in my ego-structure-or-something, or in the ego-structure-analog of CFAR or some other group-level structure I've been trying to defend, and I've been more into "so you don't get to claim I should do things differently" than into whether my so-called "goal" would work.  Cf "people don't seem to want things."

Beware of Other-Optimizing [? · GW]?

this narrative

Is this the narrative of "this person is using this as an excuse not to try more" or "this is a reason not to try more"?

I am pretty far from having fully solved this problem myself, but I think I'm better at this than most people, so I'll offer my thoughts.

My suggestion is to not attempt to "figure out goals and what to want," but to "figure out blockers that are making it hard to have things to want, and solve those blockers, and wait to let things emerge."

Some things this can look like:

1.  Critch's "boredom for healing from burnout" procedures.  Critch has some blog posts recommending boredom (and resting until quite bored) as a method for recovering one's ability to have wants after burnout:
   1. https://acritch.com/fun-does-not-preclude-burnout/
   2. https://acritch.com/boredom/
2. Physically cleaning things out.  David Allen recommends cleaning out one's literal garage (or, for those of us who don't have one, I'd suggest one's literal room, closet, inbox, etc.) so as to have many pieces of "stuck goal" that can resolve and leave more space in one's mind/heart (e.g., finding an old library book from a city you don't live in anymore, and either returning it anyhow somehow, or giving up on it and donating it to goodwill or whatever, thus freeing up whatever part of your psyche was still stuck in that goal).
3. Refusing that which does not "spark joy." Marie Kondo suggests getting in touch with a thing you want your house to be like (e.g., by looking through magazines and daydreaming about your desired vibe/life), and then throwing out whatever does not "spark joy", after thanking those objects for their service thus far.
   1. Analogously, a friend of mine has spent the last several months refusing all requests to which they are not a "hell yes," basically to get in touch with their ability to be a "hell yes" to things.
4. Repeatedly asking one's viscera "would there be anything wrong with just not doing this?".  I've personally gotten a fair bit of mileage from repeatedly dropping my goals and seeing if they regenerate.  For example, I would sit down at my desk, would notice at some point that I was trying to "do work" instead of to actually accomplish anything, and then I would vividly imagine simply ceasing work for the week, and would ask my viscera if there would be any trouble with that or if it would in fact be chill to simply go to the park and stare at clouds or whatever.  Generally I would get back some concrete answer my viscera cared about, such as "no! then there won't be any food at the upcoming workshop, which would be terrible," whereupon I could take that as a goal ("okay, new plan: I have an hour of chance to do actual work before becoming unable to do work for the rest of the week; I should let my goal of making sure there's food at the workshop come out through my fingertips and let me contact the caterers" or whatever.
5. Gendlin's "Focusing."  For me and at least some others I've watched, doing this procedure (which is easier with a skilled partner/facilitator -- consider the sessions or classes here if you're fairly new to Focusing and want to learn it well) is reliably useful for clearing out the barriers to wanting, if I do it regularly (once every week or two) for some period of time.
6. Grieving in general.  Not sure how to operationalize this one.  But allowing despair to be processed, and to leave my current conceptions of myself and of my identity and plans, is sort of the connecting thread through all of the above imo.  Letting go of that which I no longer believe in.

I think the above works much better in contact also with something beautiful or worth believing in, which for me can mean walking in nature, reading good books of any sort, having contact with people who are alive and not despairing, etc.  

This is often my problem - I think "I could probably do it if I really wanted to" and "I'm not doing it", and conclude "Therefore I don't really want to do it (strongly enough)."

I make space in my week to be bored, and where there are no options for short-term distractions that will zombify me (like videogames or YouTube). I usually find that ideas come to me then that I want that take a bit more work but will be more satisfying, like learning a song on the guitar or reading a book or doing something with a friend.

Chatting with friends who are alive and wanting things is another way I notice such things in myself, usually I catch some excitement from them as I'm empathizing with them.

I wrote the above before reading Anna's comment to see how our answers differed; seems like our number 1 recommendation is the same! 

Cleaning things out also works for me, I did that yesterday and it helped me believe in [LW · GW] my ability to make my world better. 

I also concur with the grieving one, but I never know how to communicate it. When I try, I come up with sentences like "Now vividly imagine failing to get the thing you want. Feel all the pain and sorrow and nausea associated with it. Great, now go get it!" but that doesn't seem to communicate why it helps and reads to me like unhelpful advice.

As someone with the opposite problem, who can babble countless goals that interest me, and is rigidly married to a few interrelated ones (i.e. make music videos, make films) but struggling to execute them anywhere near to my liking, I hope I can provide some insight into how to find goals or what you want.

I believe that big goals are no different than small goals in terms of finding them.

I'll be happy to write a post on this if any of this seems intriguing or useful but here's the dotpoints:

• If you have heroes, who are they? What adjectives would you use to describe them? For example, I would describe one of my heroes, Miuccia Prada, as "innovative" "insightful" "paradoxical" and "sophisticated". I could make it my goal to cultivate one of these qualities in myself, and that would require finding an exercise or even enrolling in a course or activity which would allow me to do so?
  • i.e. the go-to example would be if you want to be more 'charismatic' take up public speaking, join a amateur theatre troupe as that is meant to be a means of developing it.
• If you don't have heroes, who among your social group or friends have the traits or manners that you most envy (in a non-destructive way)? Same as above.
• Both of these exercises can be inverted by looking at people you detest or at least have a strong aversion to. Pride and Shame are good indicators too of what you want.
• Coming up with goals is easy, committing to them is hard. Just babble. Here's a template: "I would feel proud if I had a reputation based on fixing/making X". Prune out the ones that don't elicit a passionate response. 
• Analyze your Revealed Preferences on groceries as an Economist would. Everything from buying biodegradable dishwashing detergent to anti-aging wrinkle cream to tickets to a  UFC match are all commitments to a certain lifestyle or living with certain principals or values. Those commitments should point you towards broader patterns of goals

Okay sure? But what works for me? "How do you come up with goals". Here's how I do it.

I've known since I was a teenager that I've wanted to be involved in motion pictures however last year I asked myself "okay you say you want to make films, but what is a film you would be deliriously proud of look like?" so I brainstormed all the qualities and elements, and made a video-moodboard (a hour long montage of films, music videos, retro TV commercials, even Beckett plays and experimental animations that inspired me) and that too formed the vague outline of a story. Now my goal is to write a screenplay that incorporates all those elements seamlessly, and then the subsequent goal to make that screenplay into a film. There's certainly a lot of functionary goals and steppingstones that must be met to achieve those goals.

Do I know how to make that film? Not with any confidence. But that's because coming up with a goal, being specific about what I want, what I'm passionate and dedicated to is the easy part.

I assume he said something more nuanced and less prone to blindspots than that.

I think this is pretty close to what he said. I suspect he'd have some nuances if we drilled down into it.

This involves speculation on my part, but having known Critch moderately well, I'd personally bet that he has something like 10x to 30x fewer things lying around bothering him than most people (which maybe comes in waves where, first he had 10x fewer things lying around bothering him, then he set more ambitious goals for himself that created more bothersome things, then he fixed those, etc)

He probably does still have blindspots but I think the effect is real.

In my experience, I have not been able to reliably break even. This kind of estimate assumes a kind of fungibility that is sometimes correct and sometimes not. I think When to Get Compact [LW · GW] is relevant here--it can feel like my bottleneck is time, when in fact it is actually attentional agency or similar. There are black holes that will suck up as much of our available time as they can.

Yeah this sounds like a real/common problem, and dealing with that somehow seems like a necessary piece for this whole process to work.

I'm not Critch, but to speak my own defense of the numeracy/scope sensitivity point:

IMO, one of the hallmarks of a conscious process is that it can take different actions in different circumstances (in a useful fashion), rather than simply doing things the way that process does it (following its own habits, personality, etc.).  ("When the facts change, I change my mind [and actions]; what do you do, sir?")

Numeracy / scope sensitivity is involved in, and maybe required for, the ability to do this deeply (to change actions all the way up to one's entire life, when moved by a thing worth being moved by there).

Smaller-scale examples of scope sensitivity, such as noticing that a thing is wasting several minutes of your day each day and taking inconvenient, non-default action to fix it, can help build this power.

You might want to explore the Thinking Processes developed by Eliyahu Goldratt, which are part of his Theory of Constraints. These processes, such as the Goal Tree and the Current Reality Tree, are excellent tools for visualizing complex issues and identifying causal relationships between different elements of a system. Additionally, since you mentioned statistics, I recommend "The Book of Why" by Judea Pearl. 

I'm remembering the following excerpt from The Scout Mindset. I think it's similar to what I say above.

My path to this book began in 2009, after I quit graduate school and threw myself into a passion project that became a new career: helping people reason out tough questions in their personal and professional lives. At first I imagined that this would involve teaching people about things like probability, logic, and cognitive biases, and showing them how those subjects applied to everyday life. But after several years of running workshops, reading studies, doing consulting, and interviewing people, I finally came to accept that knowing how to reason wasn't the cure-all I thought it was.

Knowing that you should test your assumptions doesn't automatically improve your judgement, any more than knowing you should exercise automatically improves your health. Being able to rattle off a list of biases and fallacies doesn't help you unless you're willing to acknowledge those biases and fallacies in your own thinking. The biggest lesson I learned is something that's since been corroborated by researchers, as we'll see in this book: our judgment isn't limited by knowledge nearly as much as it's limited by attitude.

I totally agree that's a (the?) root level cause for most people.

My guess (although I'm not sure) is that in Critch's case working with CFAR participants he was filtering on people who were at least moderately strategic, and it turned out there was a third blocker.

The filtering described here seems moderately specific but not sensitive, whether or not you agree with the "income implies competence" relationship being strong.

It seems true that those who are interested in and can pay for a $4k course of this type are more likely to have 17 of the attributes in question than a person picked at random from the population. However, the filter tells you nothing about, and completely excludes, a large number of people who would fit the "have 17 of these attributes" criteria but not the "have $4k to spend on a course or the time to take it" criteria. 

The filter allows in a population of people with above-average chances of meeting the attribute criteria, but blocks a large and unknown number of other people who would also meet that criteria. 

It is potentially good for creating a desired environment in the course (having mostly people with a lot of the desired attributes), but is not a good filter for identifying the much larger population of people who might be interested in and benefitted by the course (as described in article as having 17 of the attributes and therefore capable of picking up the other two).

Well, you're filtering on both "can afford to pay for a workshop" and "wants to attend a workshop thay charges that much"...

Rationality seems to be missing an entire curriculum on "Eros" or True Desire.

I got this curriculum from other trainings, though. There are places where it's hugely emphasized and well-taught.

What are these places?

Yeah I don't know that I disagree with it (I think I maybe believe it less strongly than you atm, but it seems like a reasonable take)

I think the key issue here is that CFAR workshops were optimized around being 4 days long. I think teaching someone numeracy in 4 days is very hard, and the kind of things you end up being able to convey look different (and still pretty valuable, but I do think end up missing a large fraction of the art of the art of rationality).