A 'Practice of Rationality' Sequence?

I think we're overdue for a general overhaul of "applied epistemic rationality".

Superforecasting and adjacent skills were, in retrospect, the wrong places to put the bulk of the focus. General epistemic hygiene is a necessary foundational element, but predictive power is only one piece of what makes a model useful. It's a necessary condition, not a sufficient one.

Personally, I expect/hope that the next generation of applied rationality will be more explicitly centered around gears-level models [LW · GW]. The goal of epistemic rationality 2.0 will be, not just a predictively-accurate model, but an accurate gears-level understanding.

I've been trying to push in this direction for a few months now. Gears vs Behavior [LW · GW] talked about why we want gears-level models rather than generic predictively-powerful models. Gears-Level Models are Capital Investments [LW · GW] talked more about the tradeoffs involved. And a [LW · GW] bunch [? · GW] of [? · GW] posts [LW · GW] showed how to build gears-level models in various contexts.

Some differences I expect compared to prediction-focused epistemic rationality:

• Much more focus on the object level. A lot of predictive power comes from general outside-view knowledge about biases and uncertainty; gears-level model-building benefits much more from knowing a whole lot about the gears of a very wide variety of systems in the world.
• Much more focus on causality, rather than just correlations and extrapolations.
• Less outsourcing of knowledge/thinking to experts, but much more effort trying to extract [LW · GW] experts' models, and to figure out where the models came from and how reliable the model-sources are.

In my first CFAR experience (which was an MSFP, fwiw), the phrase "It's not about the exercises!" was kind of a motto. It was explained at the beginning that CFAR teaches exercises not because people learn the exercises and then go out and use the exercises, but rather, going through the exercises a few times changes how you think about things. (The story was that people often go to a CFAR workshop and then improve a bunch of things in their life, but say "but I haven't been doing the exercises!".)

I want to flag that this is not a universal attitude at CFAR.

CFAR workshops have long taught that "The point of the workshop is not the techniques", but I, at least, have long been frustrated that almost no one actually uses the techniques, even though they provide a lot of value. It seems to me that folks sometimes use "the techniques are not the point" or the concept of "five second versions" as an excuse for not sitting down with paper to goal factor, or do IDC, or whatever, even though those mental processes are demonstrably useful for actually solving problems on the object level, and even getting at the mindset requires some dozens (at least) of mindful reps, which few people ever do.

Duncan once taught a mini-workshop where the moto was explicitly "People don't use the tools they have", and I've been trying to push a theme of something like "actually practice", in CFAR's recent instructor training, and with a new class on training regimes, for instance.

[The following is a bit of a ramble. I'm making a couple of different points, and have differing levels of confidence in each one, so I've segregated them, to avoid everything getting mixed together]

Point 1:

I think there's something that's a little bit missing in how this question is asked. Meditation (to my understanding) is the sort of thing where one can do basically the same activity for many millions of reps, and over time, one will build capacity along some dimension. This kind of training is quantitative, you keep getting a little better at a particular activity. (Another activity that is like this is literal weightlifting.)

But I think that quantitative training is the exception rather than the rule.

I currently think that rationality (and most skills for that matter), is better thought of as qualitative: made up of a bunch of very distinct, discrete, micro-skills, or TAPs. Each micro-skill requires a specific training exercise, but after you've robustly ingrained the TAP, you reliably execute the relevant motion when confronted with the relevant stimuli, you're done. You don't get meaningfully stronger at applying each one. You maybe have to do the exercise again on a review schedule. But once you train the TAPs of (for instance) "notice that my thoughts are feeling fake -> ask what do I actually believe?" or "feel frustrated -> orient on what's happening in this situation" [1] to reliability, it doesn't really make sense to try and get better at that kind of TAP. You just move on to training the next one to reliability.

Point 2:

That said, in order to have this not spin off into worthlessness, there needs to be grounded in some particular real world "activity" that you do reliably enough to get feedback (in the same way that if you're trying to get better at playing football, mostly you're doing drills, doing deliberate practice on specific low level skills, but you also want those skills to come together in the activity of "playing football games." This is closer to a "practice" like meditation.

Some "practices" in that vein, that come to mind:

• Doing research and theoretical work
• Doing math proofs
• Doing MIRI like research
• Effectively resolving disagreements
• Doing actual science?
• Debugging code?
• Running a hedge fund
• Writing and discussing Slate Star Codex articles

Point 3:

But all of these (except maybe the first one?) are too narrow to be "rationality practice." Unless you care about any of these in particular, I think the main thing is just trying ambitious things in the world. Your low level skills should "come together" in making more money, or getting better grades, or finding the sort of significant other that you want, or successfully executing on some project, or something.

I think this tension is at the core of why we are not really a community of practice. Every community of practice that I know about, be that the Parkour community, or the Circlers, or your local football club, has some specific activity, that one 1) could reasonably spend hours doing, 2) could enjoy doing for its own sake and, 3) can meaningfully get better at. We decided that our thing was "winning", in general, so any particular activity will always be too narrow to capture what we care about.

This dynamic makes me sympathetic to this [LW(p) · GW(p)] comment: I think if you try and have a community who's central activity is "winning", you're going to find that "winning" is not the sort of thing that you can easily set up a practice regime at. But if you make your community about figuring out confusing questions, that is in fact something that you can do many reps of and get a lot better at.

[1] - Note that you have to train the true, sub-verbal, versions of these TAPs, not the words I'm using here.

I usefully demonstrated rationality superpowers yesterday by bringing a power strip to a group project with limited power outlets.

Now, you could try to grind this ability by playing improv games with the situations around you, looking for affordances, needs, and solutions. But this is only a sub-skill, and I think most of my utility comes from things that are more like mindset technology.

A personal analogy: If I want to learn the notes of a tune on the flute, it works fine to just play it repeatedly - highly grindable. If I want to make that tune sound better, this is harder to grind but still doable; it involves more skillful listening to others, listening to yourself, talking, trial and error. If I want to improve the skills I use to make tunes sound better, I can make lots of tunes sound better, but less of my skill is coming from grinding now, because that accumulation is slower than other methods of learning. And if I want to improve my ability to learn the skills used in making tunes sound better...

Well, first off, that's a rationality-adjacent skill, innit? But second, grinding that is so slow, and so stochastic, that it's hard to distinguish from just living my life, but just happening to try to learn things, and accepting that I might learn one-time things that obviate a lot of grinding.

So maybe the real grinding was bringing the power strip all along.

Another example pointing at what I want here is bewelltuned.com. The content may or may not be right, but the sort of thing seems exactly right to me -- actionable skills you can keep working on regularly after getting simple explanations of how to do it. And furthermore, the presentation seems exactly right. LessWrong has a tendency to focus on wordy explanations of intellectual topics, which is great, but the bewelltuned style seems like an excellent counterbalance.

Somewhat side note, but I think bewelltuned is deeply great, and a minor obstacle I've found is that because the author is dead, and there's no clear steward of the content, and the content is blatantly incomplete, I'm sort of unsure what to do with it. 

One option would be for someone else to rewrite it in their own voice, but honestly in most cases the original writing was quite clear, and rewriting it would just make it worse. I think some of parts of the models might turn out to be false and some instructions suboptimal, in which case rewriting makes sense, but for now AFAICT they're just sort of the best version of themselves around. 

edit to be clearer/more-actionable: 

I'd like to be able to copy posts over to LessWrong so they can be more tightly integrated into other sequences, but I don't know if that's reasonable, and don't know how to get into an epistemic state where it's either clearly reasonable (and I can do it) or clearly unreasonable (and I can move on). Does anyone know anyone who knew squirrel-in-hell/Maia/Chris Pasek well enough to figure out an answer to that?

Curated. "How do we actually practice rationality" seems like one of the key questions we should be collectively trying to answer. It seems good to both collect/distill our existing resources, as well as to periodically remind people that creating practical exercises is a useful thing to do.

The TL:DR comment on this is also the conclusion.

It was a group of rather committed and also individually competent rationalists, but they quickly came to the conclusion that while they could put in the effort to become much better at forecasting, the actual skills they'd learn would be highly specific to the task of winning points in prediction tasks, and they abandoned the project, concluding that it would not meaningfully improve their general capability to accomplish things!!

What you (can) learn from something might not be obvious in advance. While it's possible they were right, it's possible they were wrong.

And if you're right, then doing the thing is a waste, but if you are wrong then it's not.*

*Technically the benefit of something can equal the cost.

U(x) = Benefit - Cost. The first is probabilistic - in the mind, if not in the world. (The second may be as well, but to lesser extent.)

If this is instead modeled using a binary variable 'really good (RG)', the expected utility of x is roughly:

Outcome_RG*p_RG + Outcome_not*(1-p_RG) - cost

But this supposes that the action is done or not done, ignoring continuity. You to superforecaster you, is a continuum. If this is broken up into into intervals of hours then there may exist both a number of hours x and y, such that U(x)-cost >0, but U(y)-cost < 0. The continuous generalization is the derivative of 'U(x hours) - cost', and it becomes zero where the utility has stopped increasing and started increasing (or when the reverse holds). This leaves the question of how U(x) is calculated, or estimated. One might imagine that this group could have been right - perhaps the low hanging of fruit of forecasting/planning is Fermi estimates, and they already had that skill/tool.

Forecasting (predicting the future) is all well and good if you can't affect something, but if you can then perhaps planning (creating the desired future) is better. The first counterexample that comes to mind is that if you could predict the stock market in advance, then you might be able to make money off of that. This example seems unlikely, but it suggests a relationship between the two - some information about the future is useful for 'making plans'. However, while part of what information that will/could be important in the future may be obvious, that leaves:

• how to forecast information about the future that's obviously useful (if the forecast is correct)
• the information that's not obviously useful, but turns out to be important later (This is usually lumped under 'unknown unknowns', but while Moravec's paradox** can be cast as an unknown unknown, the fact that no one had built a machine/robot that did x yet, could be considered known.)

**Moving is harder than calculating.