When is rationality useful?

I do disagree about rationality not being useful to for "positive things" (such as creativity). I think I have become more intellectually generative as I've gained rationality skill. A lot of this does have to do with avoiding mistakes in the creative/generative process, but

a) that seems close enough to "just getting better at generation" that I'm not sure it makes much sense to refer to it as "just fixing mistakes",

b) I also think rationality involves reinforcing the skills about what to do right.

See Tuning Your Cognitive Strategies for what I think of as the most distilled essence of that skill.

I also think using rationality to clarify your goals, and to figure out which subgoals are most relevant to them, is once of the most important skills. That's not (quite) being creative/generative, but seems related.

We can model success as a combination of doing useful things and avoiding making mistakes. As a particular example, we can model intellectual success as a combination of coming up with good ideas and avoiding bad ideas. I claim that rationality helps us avoid mistakes and bad ideas, but doesn’t help much in generating good ideas and useful work.

Eliezer Yudkowsky has made similar points in e.g. "Unteachable Excellence" [LW · GW] ("much of the most important information we can learn from history is about how to not lose, rather than how to win", "It's easier to avoid duplicating spectacular failures than to duplicate spectacular successes. And it's often easier to generalize failure between domains.") and "Teaching the Unteachable" [LW · GW].

We can model success as a combination of doing useful things and avoiding making mistakes. As a particular example, we can model intellectual success as a combination of coming up with good ideas and avoiding bad ideas. I claim that rationality helps us avoid mistakes and bad ideas, but doesn’t help much in generating good ideas and useful work.

I think this might be sort of right, but note that since plans are hierarchical, rationality can help you avoid mistakes (e.g. failing to spend 5 minutes thinking about good ways to do something important, or assuming that your first impression of a field is right) that would have prevented you from generating good ideas.

It’s least useful in domains where making mistakes is a more effective way to learn than reasoning things out in advance, and so there’s less advantage in avoiding them.

[...]

It’s least useful in domains where making mistakes is a more effective way to learn than reasoning things out in advance, and so there’s less advantage in avoiding them.

This seems to me like you equate instrumental rationality with long explicit reasoning.

If you look at rationality!CFAR there are many techniques that are of a different nature. Resolving internal conflicts via Gendlin's Focusing or Internal Double Crux is helpful for building intrinsic motivation.

I think my views are somewhat similar. Let me crosspost a comment I made in a private conversation a while ago:

I think the main reason why people are asking "Why aren't Rationalists winning?" is because Rationality was simply being oversold.

Yeah, seems like it. I was thinking: why would you expect rationality to make you exceptionally high status and high income?^[1] And I think rationality was sold as general-purpose optimal decision-making, so once you have that, you can reach any goals which are theoretically reachable from your starting point by some hypothetical optimal decision-maker—and if not, that's only because the Art is not fully mature yet.

Now, in reality, rationality was something like:

• a collection of mental movements centered around answering difficult/philosophical questions—with the soft implication that you should ingrain them, but not a clear guide on how (aside from CFAR workshops);
• a mindset of transhumanism and literally-saving-the-world, doing-the-impossible ambition, delivered via powerfully motivational writing;
• a community of (1) nerds who (2) pathologically overthink absolutely everything.

I definitely would expect rationalists to do better at some things than the reference class of {nerds who pathologically overthink everything}:

I would expect them not to get tripped up if explicitly prompted to consider confusing philosophical topics like meaning or free will, because the mental movement of {difficult philosophical question → activate Rationality™} is pretty easy and straightforward.

Same thing if they encounter e.g. a different political opinions or worldviews: I'd expect them to be much better at reconsidering their dogmas, if, again, externally prompted. I'd even expect them to do better evaluating strategies.

But I don't think there's a good reason to expect rationalists to do better unprompted—to have more unprompted imagination, creativity, to generate strategies—or to notice things better: their blind spots, additional dimensions in the solution space.

Rationality also won't help you with inherent traits like conscientiousness, recklessness, tendency for leadership, the biological component of charisma (beyond what reading self-help literature might do for you).

I also wouldn't expect rationalists to be able to dig their way through arbitrarily many layers of Resistance on their own. They might notice that they want to do a thing T and are not doing it, but then instead of doing it, they might start brainstorming ways how to make themselves do T. And then they might notice that they're overthinking things, but instead of doing T, they start thinking about how to stop overthinking and instead start doing. And then they might notice that and pat themselves on the back and everything and think, "hey, that would make a great post on LW", and so they write a post on LW about overthinking things instead of fucking doing the fucking thing already.

Rationality is great for critical thinking, for evaluating whatever inputs you get; so that helps you to productively consider good external ideas, not get tripped by bad ideas, and not waste your time being confused. In the ideal case. (It might even make you receptive to personal feedback in the extreme case. Depending on your personality traits, I guess.)

On the other hand, rationality doesn't help you with exactly those things that might lead to status and wealth: generating new ideas, changing your biological proclivities, noticing massive gaps in your epistemology, or overturning that heavily selected-for tendency to overthink and just stumbling ass-first out into the world and doing things.

I personally define rationality as "the study of how to make robustly good decisions" (with epistemic and instrumental rationality being two arenas that you need to be skilled at for it to matter)

Some things I see rationality for are:

• Performing robustly even if you situation changes. You can make a bunch of money by finding a locally good job and getting skills to do it pretty well, without relying much on rationality. But, if you are dissatisfied with the prospect that your situation might change (your job stops being lucrative, you stop being emotionally satisfied with it, you realize that you have other goals, you realize that the world might be about to end), you want rationality, so that you can figure out what other goals to pursue, and how to pursue them, and what facts about reality might help or hinder you.
• Thinking about problems when it's hard to find good evidence
• Gaining a deeper understanding of thinking, well enough to contribute to the cutting edge of how to think well, which can then either be used to think about particularly-tricky problems, or distilled down into more general lessons than can be applied with less effort.

For example, I'm glad someone studied Bayes theorem well enough to distill it down into the general two heuristics of "remember base rates" and "remember relative likelihood ratios" when considering evidence. That person/people may or may not have gotten benefits for themselves commesurate with the effort they put in.

Many life skills don't show benefit until they become internalized enough to be deployed organically in response to circumstance. Probabilistic reasoning, factor analysis, noticing selection effects, noticing type errors, etc. are 'rationalist' examples of this, but it applies to many if not most skills.

https://en.wikipedia.org/wiki/Shuhari

I think rationality ought to encompass more than explicit decision making (and I think there are plenty of writing on this website that show it does even within the community).

If you think of instrumental rationality of the science of how to win, then necessarily it entails considering things like how to setup your environment, unthinking habits, how to "hack" into your psyche/emotions.

Put otherwise, it seems you share your definition of Rationality with David Chapman (of https://meaningness.com/ ) — and I'm thinking of that + what he calls "meta-rationality".

So when is rationality relevant? Always! It's literally the science of how to make your life better / achieving your values.

Of course I'm setting that up by definition... And if you look at what's actually available community-wise, we still have a long way to go. But still, there is quite a bit of content about fundamentals ways in which to improve not all of which have to do with explicit decision making or an explicit step-by-step plan where each step is an action to carry explicitly.

Interesting take on things. I think I'd want a more specific definition of "rationality" to really debate, but I'll make a few counter-arguments that the study and practice of rationality can improve one's capability to choose and achieve desirable outcomes.

"Doing good things and avoiding mistakes" doesn't really match my model, but let's run with it. I'll even grant that achieving this by luck (including the luck of having the right personality traits and being inexplicably drawn to the good things) is probably just as good as doing it by choice. I do _NOT_ grant that it happens by luck with the same probability as by choice (or by choice + luck). Some effort spent in determining which things are good, and in which things lead to more opportunity for good is going to be rewarded (statistically) with better outcomes.

The question you don't ask, but should, is "what does rationality cost, and in what cases is the cost higher than the benefit"? I'll grant that this set may be non-zero.

I'll also wave at the recursion problem: "when is rationality useful" is a fundamentally rationalist question.

This seems largely correct, so long as by "rationality", you mean the social movement. The sort of stuff taught on this website, within the context of human society and psychology. Human rationality would not apply to aliens or arbitrary AI's.

Some people use the word "rationality" to refer to the abstract logical structure of expected utility maximization, baysian updating, ect, as exemplified by AIXI, mathematical rationality does not have anything to do with humans in particular.

Your post is quite good at describing the usefulness of human rationality. Although I would say it was more useful in research. Without being good at spotting wrong Ideas, you can make a mistake on the first line, and produce a Lot of nonsense. (See most branches of philosophy, and all theology)