"Fractal Strategy" workshop report

post by Raemon · 2024-04-06T21:26:53.263Z · LW · GW · 18 comments

Contents

    tl;dr on results 
  Workshop Outline
        Beforehand:
        Day 1: Practice skills on quick-feedback exercises
        Day 2: Big picture strategic thinking
        Day 3: Choose your own short exercises, and object-level work
        Day 4: Consolidation
  Core Workshop Skills
    "Metastrategic Brainstorming"
    "2+ Plans at 3 Strategic Levels"
    "Fluent, Cruxy Operationalization"
  Feedback
    "What changes did you make to you plan?"
    What's the most money you'd have paid for this workshop?
    What's the most money you'd have paid for the idealized version of this workshop that lives in your head, tailored for you?
    Overall, how worthwhile was the workshop compared to what else you could have been doing? (scale of 1-7)
    Rating individual activities/skills
  My own updates
  Would you like to attend a Ray Rationality Workshop?
      (or: what are your cruxes for doing so?)
None
18 comments

I just ran a workshop teaching the rationality [LW · GW] concepts [LW · GW] I've developed this year. 

If you're interested in paying money for a similar workshop, please fill out this form.

---

Six months ago, I started thinking about improving rationality. 

Originally my frame was "deliberate practice for confusing problems". For the past two months, I've been iterating on which skills seemed useful to me personally, and which I might convey to others in a short period of time. 

I settled into the frame "what skills are necessary for finding and pivoting to 10x better plans?". It's the area I most needed rationality for, myself, and it seemed generalizable to a lot of people I know.

I ended up with 5-10 skills I used on a regular basis, and I put together a workshop aiming to teach those skills in an immersive bootcamp environment. The skills wove together into a framework I'm tentatively called "Fractal Strategy", although I'm not thrilled with that name. 

Basically, whenever I spend a bunch of resources on something, I...

• Explicitly ask "what are my goals?"
• Generate 2-5 plans at 3 different strategic levels
• Identify my cruxes for choosing between plans
• Fluently operationalize fatebook predictions about those cruxes
• Check if I can cheaply reduce uncertainty on my cruxes

The framework applies to multiple timescales. I invest more in this meta-process when making expensive, longterm plans. But I often find it useful to do a quick version of it even on the ~30-60 minute timescale. 

I put together a workshop, aiming to:

1. help people improve their current, object level plan
2. help people improve their overall planmaking/OODA-loop process

tl;dr on results 

I didn't obviously succeed at #1 (I think people made some reasonable plan updates, but not enough to immediately say an equivalent of "Hot Damn, look at that graph". See the Feedback section for more detail). 

I think many people made conceptual and practical updates to their planning process, but it's too early to tell if it'll stick, or help.

Nonetheless, everyone at the workshop said it seemed like at least a good use of their time as what they'd normally have been doing. I asked "how much would you have paid for this?" and the average answer was $800 (range from $300 to $1,500).

When I was applying these techniques to myself, it took me more like ~3 weeks to update my plans in a significant way. My guess is that the mature version of the workshop comes with more explicit followup-coaching. 

Workshop Outline

First, here's a quick overview of what happened.

Beforehand: 

• People sent me a short writeup of their current plans for the next 1-2 weeks, and broader plans for the next 1-6 months.

Day 1: Practice skills on quick-feedback exercises

• Everyone installs the fatebook chrome/firefox extension
• Solve a puzzle with Dots and a Grid with an unspecified goal
• Solve a GPQA question with 95% confidence
• Try to one-shot a Baba is You puzzle [LW · GW]
• For both of those puzzles (Baba and GPQA), ask "How could I have thought that faster? [LW · GW]"
• Play a videogame like Luck Be a Landlord, and make fermi-calculations about your choices within the game.
• For all exercises, make lots of fatebook predictions about how the exercise will go.

Day 2: Big picture strategic thinking

• Work through a series of prompts about your big picture plans.
• Write up at least two different big-picture plans that seem compelling
• Think about short-feedback exercises you could do on Day 3

Day 3: Choose your own short exercises, and object-level work

• Morning: Do concrete exercises/games/puzzles that require some kind of meta-planning skill, that feels useful to you.
• Afternoon: Do object-level work on your best alternative big picture plan, 
  • You get to practice "applying the method" on the ~hour timescale
  • You flesh out your second favorite plan, helping you treat it as "more real"

Day 4: Consolidation

• Write up your plan for the next week (considering at least two alternative plans or frames)
• Review how the workshop went together
• Consolidate your takeaways and Murphitjsu [LW · GW]. 
  • What practices do you hope to still be trying a week, month, or year from now? Do you predict you'll actually stick with them? Do you endorse that? What can you do to help make things stick.
• Fill out a feedback form

Most of the workshop had people working independently, with me and Eli Tyre cycling through, chatting with people about what they were currently working on.

I expect to change the workshop structure a fair amount if/when I run it again, but, I feel pretty good about the overall approach. The simplest change I'd make is dropping the "Fermi calculations on Luck Be a Landlord" section, unless I can think of a better way to build on it. (I have aspirations of learning/teaching how to make real fermi estimates about your plans, but this feels like my weakest skill in the skill tree)

Core Workshop Skills

A good explanation of each concept at the workshop would be an entire sequence of posts. I hope to write that sequence, but for now, here are some less obvious things I want to spell out.

"Metastrategic Brainstorming"

 I didn't actually do a great job teaching this at the workshop, but, a core building block skill upon which all others rest, is:

"Be able to generate lots of ideas for strategies to try."

If you get stuck, or if you have a suspicion there might be a much better path towards solving a problem, you should be able to generate lots of ideas on how to approach the problem. If you can't, you should be able to generate metastrategies that would help you fix that. 

This is building off of "Babble [LW · GW]"-esque skillset. (See the Babble challenges [LW · GW], which encourages generating 50 ways of accomplishing things, to develop the muscle of "not getting stuck.")

A few examples of what I mean:

• Break a problem into simpler pieces
• Take a nap, or get a drink of water
• Ask yourself what a smart friend would do

I have more suggestions for good meta-strategies that often work for me. However, I prefer not to give people many strategies at first: I think one of the most important skills is to figure out how to generate new strategies, in novel situations, where none of your existing strategies is quite the right fit.

---

"2+ Plans at 3 Strategic Levels"

The central move of the workshop is "come up with at least 2 plans, at multiple strategic levels." 

Reasons I think this is important include:

Escaping local optima. If you want to find plans that are 10x better than your current plan, you probably have to cast a fairly broad search process.

Don't get trapped by tunnel vision. It's easier to notice it's time to pivot if you've explicitly considered your best alternatives. Leave yourself a line of retreat [LW · GW]. Avoid "idea scarcity [LW · GW]."

Practice having lots of good ideas. Strategic creativity is a muscle you can train. If it feels like a struggle to come up with more than one good idea, I think it's probably worthwhile to invest in more practice in metastrategic brainstorming. (Meanwhile, if you're good at it, it shouldn't take long)

For context, when I'm doing this process for myself, even on the hour-long timescale, I basically always am able to generate 2 alternative ways to achieve my goal that feel "real". And, quite often when I force myself to generate alternatives, I discover a better way to accomplish my goal than whatever I was going to do by default. This process takes a couple minutes, so it pays for itself on smallish timescales.

Originally, when I first conceptualized this I wrote down "have at least 2 plans at basically every level of meta." As soon as I wrote it I was like "well, that's too many meta levels. That can't possibly be practical." But what can be practical is to track:

1. The object level goal you're currently planning around
2. The higher level strategic awareness that informs your current plan
3. The narrower tactical level of how to implement your plans.

I'll followup more on this in a future post, but here's an example of what this looked like for me, when I was planning for the workshop and it was about 2 weeks away:

In this example, I had previously decided to run the 10x planning workshop. The most natural thing to do was write out the outline of the existing pieces I had and put them into a schedule. But, I was pretty sure I could get away with doing that at the last minute. Meanwhile, the workshop still had some major unsolved conceptual problems to sort out, that I wanted to allocate some spacious time towards figuring out.

---

"Fluent, Cruxy Operationalization"

Since 2020, I've been interested in calibration training for "real life." I had done simple calibration games, but the exercises felt contrived and fake and I wasn't sure they generalized to the real world.

I've intermittently practiced since then. But, only 3 weeks ago did I feel like I "got good enough to matter."

The problem with making predictions is that the most natural-to-predict things aren't actually that helpful. Forming a useful prediction is clunky and labor intensive. If it's clunky and labor-intensive, it can't be part of a day-to-day process. And if it can't be part of a day-to-day process that connects with your real goals, it's hard to practice enough to get calibrated on things that matter.

I have a lot of tacit, hard-to-convey knowledge about "how to make a good prediction-operationalization". For now, some prompts I find helpful are:

• What would I observe (physically, in the world) if Plan A turned out to fail?
• What would I observe if Plan A turned out to not matter as much as I thought?
• What would I observe in worlds where Plan B turned out to be way better than plan A?
• What's the ideal (even if stupidly expensive) experiment I could run that would inform my plan?

I also have a couple "basic predictions" that are usually relevant to most plans, such as:

• How likely am I to be "surprised in a surprising-way", which ends up being strategically relevant?
• How likely am I to "operationally screw up" my plan? (i.e. the basic plan was good but I made a basic error).

Note that most of the immediate value comes from the operationalization itself, as opposed to putting a probability on it. Often, simply asking the right question is enough to prompt your internal surprise-anticipator [LW · GW]to notice "oh, yeah I basically don't expect this to work" or "oh, when you put it like that, obviously X will happen."

But, longterm, a major point is to get calibrated. It's unclear to me how much calibration "transfers" between domains, but I think that at least if you practice predicting and calibrating on "whatever sorts of plans you actually make", you'll get more grounded intuitions in those domains, which help your internal surprise anticipator be more confidently calibrated.

A major point of the workshop is to just grind on making cruxy-predictions for 4 days, and hopefully reach some kind of "fluency escape velocity", where it feels easy enough that you'll keep doing it.

---

Feedback

Nine people came to the workshop. Two of them dropped out over the course of the workshop. I've got substantial feedback from the other seven, and a few quick notes from the remaining two.

The most obvious effects I was aiming at with workshop were:

• Help people come up with better object level plans.
• Help people learn skills, or changing their planning/OODA-loop process

Before people came to the workshop, I asked them to submit a few paragraphs about their current ~6 month plans. Some people had explicit plans, some people have "processes they expected to follow" (which would hopefully later generate more fleshed out plans), and some people didn't have much in the way of plans.

Here are feedback questions I asked, and some anonymized and/or aggregated answers:

 

"What changes did you make to you plan?"

I think nobody made a major change to a longterm plan (although I don't know if that was actually a realistic goal for one workshop). 

But answers included:

• Two people said "I basically no longer have an object level plan". (They both spent much of the workshop thinking about what their meta level planning process should look like)
• One person said "My original plan is now a substep in my new plan, which I might not take for a month or two."
• One person aid "I'm placing a much higher premium on maintaining optionality"
• One person didn't make changes to their ~6 months plan but seemed like the workshop helped them realize they really need a vacation, which came up during the "make your plan for next week" phase.
• Two people upweighted some ideas they'd been thinking about 

 

What's the most money you'd have paid for this workshop?

Average was ~$800. Answers were:

• $300-400
• ~$300 (would pay $2000 more for more help/attention/coaching)
• $800
• $1500, maybe $2000
• $1500, $2500 if at a better time/place
• ~$500, maybe more if they had more disposable income

I think there were some fairly distinct clusters here, where the people who got more value were also at stages of their life where the "planmaking skillset" was obviously a bottleneck for them. 

 

What's the most money you'd have paid for the idealized version of this workshop that lives in your head, tailored for you?

Average ~$5000.

• $3,500
• $10,000
• ~$3,500
• $4,000
• $5,000 ($10,000 if they had more discretionary income)
• $5,000
• $2,000 (maybe $3,000 if their company was paying)

 

Overall, how worthwhile was the workshop compared to what else you could have been doing? (scale of 1-7)

Average was 5.5 (where 4 was "about the same"). 

Results were: 5, 4, 5, 7, 6, 7, 6, 4.

 

Rating individual activities/skills

I had people rate activity and skills, with options "not valuable", "somewhat/mixed valuable", or "lots!". When reviewing that feedback, I counted "some" as 1 point, and "lots!" as 3 points.

Here was how the activities rated

 

 

My own updates

I feel like this was a solid Draft 1 of the workshop format. Eli Tyre, who I hired to collaborate on it, says his experience is that most "workshops" tend to cohere in the third iteration. 

Major things I'd like to change are:

• Figure out which skills end up actually help people, and focus more on those.
• Figure out how to teach quantitative estimating (i.e. fermi-calculations) that allow you to compare plans. (I feel like I got some handle on this recently, but it's still pretty messy)
• Find better prompts to help people with operationalizing predictions. 
• Make a better environment for people giving each other nuanced feedback on their plans. (I think this requires selecting participants for having useful technical background, so they can actually engage with the details).
• Do better at creating a safe space for people really question their life plan, and/or confront very difficult things. (Related: teach "grieving [LW · GW]")

My overall sense was that for most people the different skills didn't really gel into a cohesive framework (though 2 people found it quite cohesive, and another 2 people said something like "I can see the edges of how it might fit together but haven't really grokked it"). I came up with the "fractal strategy" framing the day before the workshop, and then was kinda scrambling to figure out how to convey it clearly over the 4 days. 

I came up with better phrasings over the course of the workshop, but I think there's a bunch of room to improve there.

One interesting thing is that the person who was "the most experienced planmaker" IMO, still got benefit from "semi-structured deliberate practice on the skills they found most important." AKA "basically the original version of my plan."

Would you like to attend a Ray Rationality Workshop?

(or: what are your cruxes for doing so?)

Whether I continue working on this depends largely on whether people are willing to pay for it at sustainable prices. "People" can either be individuals, or grantmakers who want to subsidize upskilling for x-risk researchers (and similar).

Reasons I want to charge money, rather than provide this at subsidized rates:

• "Lightcone just actually does need to pay the bills somehow, and running workshops like this isn't free." 
• These workshops only actually make sense if they are really changing people's plans/skills in radical ways. "Make sure they seem valuable enough that people are willing to pay for them" is a good way to check that this is actually worth doing.
• As Andrew Critch [LW · GW] notes, there are some good selection effects on charging money, such that attendees are people who have most of the pieces necessary to achieve cool things, and you're filling in a few missing pieces that hopefully can unlock a lot of value.

 

The workshop format I'm actually most excited about is "team workshop", where an organization sends a team of decisionmakers together. I feel more optimistic about this resulting in "actual new plans that people act on", since:

• Everyone on the team will actually share context with each other, and they can evolve their planmaking together based on what seems actually likely to work.
• Since they're an existing org with some resources, they're more likely to be able to put longterm plans into action.

 

If you're interested in any of that enough to pay money for it, fill out this interest form.

 

Meanwhile, I'll be trying to write up posts that go into more detail about what I learned here.

18 comments

Comments sorted by top scores.