Extreme Rationality: It's Not That Great

So the second factor is that most people are rational enough for their own purposes. Oh, they go on wild flights of fancy when discussing politics or religion or philosophy, but when it comes to business they suddenly become cold and calculating. This relates to Robin Hanson on Near and Far modes of thinking. Near Mode thinking is actually pretty good at a lot of things, and Near Mode thinking is the thinking whose accuracy gives us practical benefits.

Seems to me that most of us make predictably dumb decisions in quite a variety of contexts, and that by becoming extra bonus sane (more sane/rational than your average “intelligent science-literate person without formal rationalist training”), we really should be able to do better.

Some examples of the “predictably dumb decisions” that an art of rationality should let us improve on:

• Dale Carnegie says (correctly, AFAIK) that most of us try to persuade others by explaining the benefits from our point of view (“I want you to play basketball with me because I don’t have enough people to play basketball with”), even though it works better to explain the benefits from their points of view. Matches my experiences, and matches also many/most of the local craigslist ads. The gains if we notice and change this one would be significant.
• Lots of people decide to take a job “to make more money”, but don’t bother to actually research the odds of getting that job, and the average payoff from that job (the latter, at least, is easy to look up on the internet) before spending literally years training for the job. Even in cases like med school. Again, significant payoff here, and in this case fairly minimal willpower requirements.
• Lots of us tend to mostly stick to our own opinions in conversations, even in cases where our impressions are no better data than our interlocutor’s impressions, and where the correct opinion can actually impact the goodness of our lives (e.g., which course to take on a work project whose outcome matters; which driving route is faster; which carwash to try) (these latter decisions are small, but small decisions add up).
• Similarly, lots of us decide we’re “good at X and bad at Y”, or that we’re “the sort of people who do A in such-and-such a specific manner”, and quit learning in a particular domain, quit updating our skill-sets, keep suboptimal beliefs or practices glued to our identities instead of looking around to see how others do things and what methods might achieve greater success. Lots of us spend far more of our thinking time noting all the reasons why we’re best off doing what we’re doing than we do looking for new ways to do things, even when such looking has tended to give us useful improvements.
• Lots of people run more risk of death by car than they would upon consideration choose, e.g. by driving too close to the car in front of them (the half-second earlier that you get home isn’t worth it) or by driving while tired. At the same time, lots of people refrain from enjoyable activities such as walking around at night or swimming off the coast of Florida despite the occasional sharks, in cases where the activities in fact pose nearly negligible danger, but the dangers in question are vivid and easy to over-estimate.

Imagine a world where the only way to become really rich is to win the lottery (and everybody is either risk averse or at least risk neutral). With an expected return of less than $1 per $1 spent on tickets, rational people don't buy lottery tickets. Only irrational people do that. As a result, all the really rich people in this world must be irrational.

In other words, it is possible to have situations where being rational increases your expected performance, but at the same time reduces your changes of being a super achiever. Thus, the claim that "rationalists should win" is not necessarily true, even in theory, if "winning" is taken to mean being among the top performers. A more accurate statement would be, "In a world with both rational and irrational agents, the rational agents should perform better on average than the population average."

And this is why I am not so impressed by Eliezer's claim that an x-rationality instructor should be successful in their non-rationality life. Yes, there probably are some x-rationalists who will also be successful people. But again, correlation 0.1. Stop saying only practically successful people could be good x-rationality teachers! Stop saying we need to start having huge real-life victories or our art is useless! Stop calling x-rationality the Art of Winning! Stop saying I must be engaged in some sort of weird signalling effort for saying I'm here because I like mental clarity instead of because I want to be the next Bill Gates! It trivializes the very virtues that brought most of us to Overcoming Bias, and replaces them with what sounds a lot like a pitch for some weird self-help cult...

I think the truth is non-symmetrical: rationalism is the art of not failing, of not being stupid. I agree with you that "rationalists should win big" is not true in the sense Eliezer claims. However, rationalists should be generally above average by virtue of never failing big, never losing too much, e.g. not buying every vitamin at the health food store, not in cults, not bemoaning ancient relationships, etc.

I'm not sure if it was your intent to point this out by contrast, but I would like to point out that a reasonable art of "kicking" would not rely on you making conscious decisions, let alone explicitly rational ones. Rather, it would rely on you ensuring that your subconscious has been freed from sources of bias ahead of time, and is therefore able to safely leap to conclusions in its usual fashion. An art that requires you to think at the time things are actually happening is not much of an art.

Case in point: when reading "Stuck In The Middle With Bruce", I became aware of a subconsciously self-sabotaging behavior I'd done recently. So I "kicked" it out by crosslinking the behavior with its goal-satisfaction state. It would be crazy to wait until the next occasion for that behavior to strike, and then try to reason my way around it, when I can just fix the bloody thing in the first place. (Interestingly, I mentioned the story to my wife, and described how it related to my own behavior... and she thought of a different sort of self-sabotage she was doing, and applied the same mindhack. So, as of now, I'd say that story was one of the top 5 most valuable things I've gotten from LW.)

Now, in the case of extinguishing a behavior, there's no way you can absolutely prove you've fixed something permanently; the best you can do is show that the thought process that you use to produce an autonomous response before applying a technique, no longer produces the same response afterward. Also, sometimes you catch a break: you find yourself in a situation, expecting yourself to do the same old stupid thing you've been doing before, and then you find you don't need to, or notice a few seconds later that you already did something completely different, and a much better choice.

Truth is, our brains really aren't that bad at making decisions, once you take out the "priority overrides" that mess things up.

Anyway, I'm rambling a bit now. The point is, "kicking" is generally not something you do at the time -- you do it in advance of the next time....

Because your brain is faster than you are.

And for this post, I use "benefits" or "practical benefits" to mean anything not relating to philosophy, truth, winning debates, or a sense of personal satisfaction from understanding things better. Money, status, popularity, and scientific discovery all count.

In my life, I've used rationality to tackle some pretty tough practical problems. The type of rationality I have been successful with hasn't been the debiasing program of Overcoming Bias, yet I have been employing scientific thinking, induction, and heuristic to certain problems in ways that are atypical for the category of people you are calling normal rationalists. I don't know whether to call this "x-rationality" or not, partly because I'm not sure the boundaries between rationality and x-rationality are always obvious, but it's certainly more advanced rationality than what people usually apply in the domains below.

On a general level, I've been studying how to get good (or at least, dramatically better) at things. Here are some areas where I've been successful using rationality:

• Recovering from social anxiety disorder and depression
• Social skills
• Fashion sense
• Popularity / social status in peer group
• Dating

I'm not using success necessarily to mean mastery, but around 1-2 standard deviations of improvement from where I started.

I do find it interesting that many people are not achieving practical benefits from their studies of more advanced rationalities. I agree with you that akrasia is a large factor in why they do not get significant practical benefits out of rationality. I am going to hypothesize an additional factor:

The practical benefits of x-rationality are constrained because students of x-rationality (such as the Overcoming Bias / Less Wrong) schools of thought focus on critical rationality, yet critical rationality is only good for solving certain types of problems.

In my post on heuristic, I drew a distinction between what I'm calling "critical rationality" (consisting of logic, skepticism, and bias-reduction) and "creative rationality" (consisting of heuristic and inference). Critical rationality concerns itself with idea validation, while creative rationality concerns itself with idea creation (specifically, of ideas that map onto the territory).

Critical rationality is necessary to avoid many mistakes in life (e.g. spending all your money on lottery tickets, high-interest credit card debt, Scientology), yet perhaps it runs into diminishing returns for success in most people's lives. For developing new ideas and skills that would lead people to success above a mundane level, critical rationality is necessary but not sufficient, and creative rationality is also required.

If in 1660 you'd asked the first members of the Royal Society to list the ways in which natural philosophy had tangibly improved their lives, you probably wouldn't have gotten a very impressive list.

Looking over history, you would not have found any tendency for successful people to have made a formal study of natural philosophy.

Michael Vassar:

nerds, scientists, skeptics and the like who like to describe their membership in terms of rationality are [not] noticibly better than average at behavioral rationality, as opposed to epistemic rationality where they are obviously better than average but still just hideously bad.

Simply applying "ordinary rationality" to behavior is extreme. People don't use reason to decide if fashion is important, they just copy. Eliezer's Secret Identities post seems to make a very similar point, which seemed to largely match this post. One point was to get rationality advice from people who actually found it useful, rather than ordinary nerds who fetishize it.

An understanding of 'x-rationality' has helped me find the world a little less depressing and a little less frustrating. Previously when observing world events, politics and some behaviours in social interactions that seemed incomprehensible without assuming depressing levels of stupidity, incompetence or malice I despaired at the state of humanity. An appreciation of human biases and evolutionary psychology (some of which stems from an interest in both going back well before I ever started reading OB) gives me a framework in which to understand events in the world which I find a lot more productive and optimistic.

An example from politics: it is hard to make any rational sense of drug prohibition when looking at the evidence of the costs and benefits. This would tend to lead to an inevitable conclusion that politicians and the voting public are either irredeemably stupid or actively seeking negative outcomes. Understanding how institutional incentives to maintain the status quo, confirmation bias and signaling effects (politicians and voters needing to be 'seen to care' and/or 'seen to disapprove') can lead to basically intelligent and well meaning people maintaining catastrophically wrong beliefs at worst allows for accepting the status quo without assuming the worst about one's fellow man and at best maps out plausible paths for achieving political change by recognizing the true nature of the obstacles.

An example from social interactions: I suffered a fair amount of personal emotional stress reconciling what I had been led to believe 'ought' to work when interacting with others and the apparently much less pleasant realities of what seemed to be successful in reality. The only conclusion I could draw was that everyone deliberately lied about the way human interactions worked for their own mysterious and possibly malicious reasons. Coming to an understanding of evolutionary psychology and signaling explanations for many common patterns of human behaviour allows me to reconcile 'doing what works' with a belief that most people are not consciously misleading or malicious most of the time. Many people don't appear to be aware of the contradictions inherent in social interactions but as someone who saw them but could not explain them without assuming the worst, discovering explanations that did not require imputing conscious malice to others allowed for a much more positive outlook on the world.

I could give a number of examples of how 'regular' rationality rigorously applied to areas of life where it is often absent have also directly helped me in my life but they seem slightly off topic for this thread.

I’m partly echoing badger here, but it’s worth distinguishing between three possible claims:
(1) An “art of rationality” that we do not yet have, but that we could plausibly develop with experimentation, measurements, community, etc., can help people.
(2) The “art of rationality” that one can obtain by reading OB/LW and trying to really apply its contents to one’s life, can help people.
(3) The “art of rationality” that one is likely to accidentally obtain by reading articles about it, e.g. on OB/LW, and seeing what happens to rubs off, can help people.

There are also different notions of “help people” that are worth distinguishing. I’ll share my anticipations for each separately. Yvain or others, tell me where your anticipations match or differ.

Regarding claim (3):
My impression is that even the art of rationality one obtains by reading articles about it for entertainment, does have some positive effects on the accuracy of peoples’ beliefs. A couple people reported leaving their religions. Many of us have probably discarded random political or other opinions that we had due to social signaling or happenstance. Yvain and others report “clarity-of-mind benefits”. I’d give reasonable odds that there’s somewhat more benefit than this -- some unreliable improvement in peoples’ occasional, major, practical decisions, e.g. about which career track to pursue, and some unreliable improvement in peoples’ ability to see past their own rationalizations in interpersonal conflicts -- but (at least with hindsight bias?) probably no improvements in practical skills large enough to show up on Vladimir Nesov’s poll. Does anyone’s anticipations differ, here?

Regarding claim (2):
I’d a priori expect better effects from attempts to really practice rationality, and to integrate its thinking skills into one’s bones, than from enjoying chatting about rationality from time to time. A community that reads articles about skateboarding, and discusses skateboarding, will probably still fall over when they try to skateboard twenty feet unless they’ve also actually spent time on skateboards.

As to the empirical data: who here has in fact practiced (2) (e.g., has tried to integrate x-rationality into their actual practical decision-making, as in Yvain’s experiment/technique, or has used x-rationality to make major life decisions, or has spent time listing out their strengths and weaknesses as a rationalist with specific thinking habits that they really work to integrate in different weeks, or etc.)? This is a real question; I’d love data. Eliezer is an obvious example; Yvain cites the impressiveness of Eliezer’s 2001 writings as counter-evidence (and it is some counter-evidence), but: (1) Eliezer, in 2001, had already spent a lot of time learning rationality (though without the heuristics and biases literature); and (2) Eliezer was at that time busy with a course of action that, as he now understands things, would have tended to destroy the world rather than to save it. Due to insufficient rationality, apparently.

I’ve practiced a fair amount of (2), but much less than I could imagine some practicing; and, as I noted in the comment Yvain cited, it seems to have done me some good. Broadly similar results for the handful of others I know who try to get rationality into their bones. Less impressive than I’d like, but I tend to interpret this a a sign we should spend more time on skateboards, and I anticipate that we’ll see more real improvement as we do.

The most important actual helps involve that topic we’re not supposed to discuss here until May, but I’d say we were able to choose a much higher-impact way to help the world than people without x-rationality standardly choose, and that we’re able to actually think usefully about a subject where most conversations degenerate into storytelling, availability heuristics, attaching overmuch weight to specific conjunctions, etc. Which, if there’s any non-negligible chance we’re right, is immensely practical. But we’re also somewhat better at strategicness about actually exercising, about using social interaction patterns that work better than the ones we were accidentally using previously (though far from as well as the ones the best people use), about choosing college or career tracks that have better expected results, etc.

Folks with more data here (positive or negative), please share.

Regarding claim (1):
I guess I wouldn’t be surprised by anything from “massive practical help, at least from particular skilled/lucky dojos that get on good tracks” to “not much help at all”. But if we do get “not much help at all”, I’ll feel like there was a thing we could have done, and we didn’t manage to do it. There are loads of ridiculously stupid kinds of decision-making that most people do, and it would be strange if there were no way we could get visible practical benefit from improving on that. Details in later comments.

By "decision", I don't mean the decision to get up in the morning, I mean the sort that's made on a conscious level and requires at least a few seconds' serious thought.

Consider yourself lucky if that doesn't describe getting up in the morning for you.

Anyway, not that this counts at all (availability bias), but I made a rational decision a couple of days ago to get some sleep instead of working later into the night on homework. I did exactly that.

In fact, I just made a rational decision-- just now-- to quit reading the article I was reading, work on homework for a few minutes and then go to bed. I haven't gotten to bed yet. Otherwise, that's going well.

...but you will disagree with me. And we are both aspiring rationalists, and therefore we resolve disagreements by experiments. I propose one.

I'm surprised you expected most of your readers to disagree. I think it's pretty clear that the techniques we work on here aren't making us much more successful than most people.

Humans aren't naturally well equipped to be extreme rationalists. The techniques themselves may be correct, but that doesn't mean we can realistically expect many people to apply them. To use the rationality-as-martial art metaphor, if you taught Shaolin kung fu to a population of fifty year old couch potatoes, they would not be able to perform most of the techniques correctly, and you should not expect to hear many true accounts of them winning fights with their skills.

Perhaps with enough work we could refine the art of human instrumental rationality into something much better than what we've got, maybe achieve a .3 correlation with success rather than a .1, but while a fighting style developed explicitly for 50 year old couch potatoes might give your class better results than other styles, you can only expect so much out of it.

Sometimes, people do worse when they try to be rational because they have a poor model of rationality.

One error I commonly see is the belief that rationality means using logic, and that logic means not believing things unless they are proven. So someone tries to be "rational" by demanding proof of X before changing their behavior, even in a case where neither priors nor utilities favor not X. The untrained person may be doing something as naive as argument-counting (how many arguments in favor of X vs. not X), and is still likely to come out ahead of the person who requires proof.

A related error is using Boolean models where they are inappropriate. The most common error of this type is believing that a phenomenon, or a class of phenomena, can have only one explanation.

Am I the only one who is isn't entirely positive towards the heavy use of language identifying the LW community as "rationalists", including terms like "rationalist training" etc.? (Though he is by far the heaviest user of this kind of language, I'm not really talking about Eliezer here, his language use is whole topic on its own - I'm restricting this particular concern to other people, to the general LW non-Eliezer jargon). Is strongly self-identifying as a "rationalist" really such a good thing? Does it really help you solve problems? (I second the questions raised by Yvain). Though perhaps small, isn't there still a risk that the focus becomes too much on "being a rationalist" instead of on actually solving problems?

Of course, this is a blog about rationality and not about specific problems, so this kind of language is not suprising and sometimes might even be necessary. I'm just a bit hesitant towards it when the community hasn't actually shown that it's better at solving problems than people who don't self-identify as rationalists and haven't had "rationalist training", or shown that the techniques fostered here have such a high cross-domain applicability as seems to be assumed. Maybe after it has been shown that "rationalists" do better than other people, people who just solve problems, I would feel better about this kind of jargon.

Here's one example of a change I've made recently, which I think qualifies as x-rationality. When I need to make a decision that depends on a particular piece of data, I now commit to a decision threshold before I look at the data. (I feel like I took this strategy from a LW article, but I don't remember where now.)

For example, I recently had to decide whether it would be worth the potential savings in time and money to commute by motorcycle instead of by car. I set a threshold for what I considered an appropriate level of risk beforehand, and then looked up the accident statistics. The actual risk turned out to be several times larger than that.

Had I looked at the data first, I would have been tempted to find an excuse to go with my gut anyway, which simply says that motorcycles are cool. (I'm a 23-year-old guy, after all.) A high percentage of motorcyclists experience a serious or even fatal accident, so there's a decent chance that x-rationality saved me from that.

This experiment seems easy to rig4; merely doing it should increase your level of conscious rational decisions quite a bit. And yet I have been trying it for the past few days, and the results have not been pretty. .... [O]ne way to fail your Art is to expect more of it than it can deliver.... Perhaps there are developments of the Art of Rationality or its associated Arts that can turn us into a Kellhus or a Galt, but they will not be reached by trying to overcome biases really really hard.

To make a somewhat uncharitable paraphrase: you read many articles on rationality, did not actually use them to change the way you make decisions, and found that the rationality hasn’t changed the results of your decisions much. You conclude, not that you aren’t practicing rationality, but that rationality can’t deliver practical goods at all, at least not as taught.

I agree we need practices for better incorporating OB/LW/new techniques of rationality into our actual practice of inference and decision-making. But it seems like the “and I’m not actually using this stuff much” result of your experiment should prevent “it hasn’t made my life much better” from telling you all that much about whether the OB/LW inference or decision-making techniques, if one does practice them, could make one’s life better.

Only a handful responded

I am reserving my judgment for a couple of years. See how I'm doing then.

Would Newton have gone even further if he'd known Bayes theory? Probably it would've been like telling the world pool champion to try using more calculus in his shots: not a pretty sight.

An interesting choice of example, given that Bayesian probability theory as we know it (inverse inference) was more or less invented by Laplace and used to address specific astronomical controversies surrounding the introduction of Newton's Laws, having to do with combining multiple uncertain observations.

I have yet to hear what anyone even means by "rationalism" or "rationalist," let alone "x-rationality." People often refer to the "techniques" or "Art of rationality" (a particularly irksome phrase), though as best I can tell, these consist of Bayes theorem and a half-dozen or so logical fallacies that were likely known since the time of Aristotle. Now, I've had an intuitive handle on Bayes theorem since learning of it in high school pre-calc, and spotting a logical fallacy isn't particularly tough for anyone accustomed to close reading of philosophy or doing science (or who's studied for the LSAT). So apart from simply calling oneself a "rationalist" and feeling really good about being a part of some "rationalist community" (much like Dennett's tone-deaf coining of the term "brights" to describe atheists), is there actually anything to this?

If I was rational enough to pick only stocks that would go up, I'd become successful regardless of how little willpower I had, as long as it was enough to pick up the phone and call my broker.

Rationality is not enough to pick the right stocks. You need to have the willpower to read the vast amount of material to enable you to do that pick.

Eliezer considers fighting akrasia to be part of the art of rationality; he compares it to "kicking" to our "punching". I'm not sure why he considers them to be the same Art rather than two related Arts.

Remember your post on haunted rationalists, and Eliezer’s reply about how it’s possible to successfully work to accept rational beliefs even with the not-so-conscious, not-so-verbal parts of oneself that might be continue to believe in ghosts after one rationally understands the arguments against?

It sounds like maybe you mean “rationality” (or “x-rationality”) to include only “conscious processes that one employs to route around natural biases, with one’s verbal centers, on purpose”, while Eliezer is using “rationality” to mean “extra bonus sanity” or “trying to get one’s whole mind, impression-making-systems, decision-making-systems, etc., in good contact with all the evidence and with one’s own real concerns” (e.g., in the manner RichardKennaway describes changing his decision-making). It’s this latter art that I’d like to improve in, at least.

Extreme rationality is for important decisions, not for choosing your breakfast cereal. Really important decisions - by which I mean those that you'd sleep on, and allocate more than ten minutes of thought - typically coincide with changes in habits and routine, which don't happen more often than once in several months. For more common decisions, we only have time and energy for ordinary rationality.

2\. Two

Foo

1\.  One

Yes, yes, yes, yes, yes. And also yes.

I had a similar reaction to the fictional rationalist initiation ceremony.

That said, on further consideration, I'm not sure the "Bayesian Conspiracy" has a choice, given its goals.

It's possible that, even though these sorts of policies do turn away perfectly competant rationalists, they are the only alternative to ending up with a comfortable community of one-or-two-sigmas-above-the-mean rationalists rather than an ultra-elite x-rationality club that can bootstrap itself into the sort of excellence that we enjoy labelling in Japanese.

This has nothing to do with rationality, extreme or otherwise. For any property P, if you want to maintain a minimum standard of P within in a group, you need some way to test for P. If you have a highly reliable test for P that has negligible false positives and few false negatives, great, use that! But lacking one, a test with negligible false positives and lots of false negatives might be good enough, if you can afford to exclude a lot of potential. (Or even a series of mostly independent tests might work well enough, even when no individual test is highly reliable, as long as you exclude anyone who fails any of the tests... which also raises your false-negative rate.)

So, hey, if the ultra-elite rationality club is meeting somewhere and plotting optimized utility, great. More power to them. The most sensible thing for them to do is not even let me waste their time by knowing they exist. LessWrong certainly isn't that club; it's at best a self-selecting group of people who think that club would be a good thing if it existed, plus some others who think they ought to be in it.

Which is OK with me.

Unrelatedly, a specific quibble:

If I was rational enough to pick only stocks that would go up, I'd become successful regardless of how little willpower I had, as long as it was enough to pick up the phone and call my broker.

Well, and enough to actually do all the necessary research to obtain the data from which you could rationally conclude that a given stock will go up. And to continue attending to that data (and researching additionally relevant data) so that you can make rational decisions about whether to sell or hold those stocks tomorrow, next week, next month, etc.

If I was rational enough to pick only stocks that would go up, I'd become successful regardless of how little willpower I had, as long as it was enough to pick up the phone and call my broker.

Availability of investing is NOT a disproof that akrasia is NOT the complete explanation. Successful investing is rationality+financial education+a lot of work (Buffett is rumored to read an incredible amount of accounting statements), and hence subject to akrasia.

Better decisions are clearly one possible positive outcome of rationality training. But another significant positive outcome is reaching the same decision faster. In my work, there are a number of rationality techniques that I have learned that have not necessarily changed the end result I have come to, but that have contributed to me spending less time confused, and getting to the right result more quickly than I otherwise would have.

Anything that frees up time in this way, has real, positive, and measurable effects on my life. (Also, confusion, and things-not-working are frustrating and stressful; so the less time I spend confused, the better)

I can't think of any people who started out merely above-average, developed an interest in x-rationality, and then became smart and successful because of that x-rationality.

I'm working on this.

In the spirit of concrete reductions, I have a question for everyone here:

Let's say we took a random but very large sample of students from prestigious colleges, split them into two groups, and made Group A take a year-long class based on Overcoming Bias, in which students read the posts and then (intelligent, engaging) professors explained anything the students didn't understand. Wherever a specific technique was mentioned, students were asked to try that technique as homework.

Group B took a placebo statistics class similar to every other college statistics class, also with intelligent and engaging professors.

Twenty-five years later, how would you expect the salaries of students in Group A to compare to the salaries of students in Group B? The same? 1.1 times greater? Twice as great? What about self-reported happiness? Amount of money donated to charity per year?

The most effective way for you to internally understand the world and make good decisions is to be super rational. However, the most effective way to get other people to aid you on your quest for success is to practice the dark arts. The degree to which the latter matters is determined by the mean rationality of the people you need to draw support from, and how important this support is for your particular ambitions.

I strongly suspect that it is unreasonable to expect people to actively apply x-rationality on a frequent, conscious basis--to do so would be to fight against human cognitive architecture, and that won't end well.

Most of our decisions are subconscious. We won't be changing this. The place of x-rationality is not to make on-the-spot decisions, it's to provide a sanity check on those decisions and, as necessary, retrain the subconscious decision making processes to better approximate rationality.

I think you are right that x-rationality doesn't help an individual win much on a day to day basis. But there are some very important challenges that humanity as a whole is failing for lack of x-rationality.

The current depression. The fact that we aren't adequately protecting the earth from asteroids. DDT being banned. Nobody's getting froze. Religion. First-past-the post elections. Most wars.

I think one reason might be that the vast majority of the decisions we make are not going to make a significant difference as to our overall success by themselves; or rather, not as significant a difference as chance or other factors (e.g., native talent) could. For example, take the example about not buying into a snake-oil health product lessdazed uses above: you've benefited from your rationality, but it's still small potatoes compared to the amount of benefit you could get from being in the right place at the right time and becoming a pop star... or getting lucky with the stock market... or starting a software company at just the right time. These people, who have to have varying degrees of something besides luck and a small amount of rationality to capitalize on it, are much more visible; even if their decisions were less-than-optimal, the other factors make up for it. Nothing's stopping them from poisoning themselves with a scam miracle-elixir, though.

This ties in with the point lessdazed was making, that the rational person most likely loses less rather than wins big - that is, makes a large number of small decisions well, rather than a single important one extremely well. That's not to be despised; I wonder what the results if we ask about overall well-being and happiness rather than fame and fortune.

[W]e should generally expect more people to claim benefits than to actually experience them.

I don't think this claim is supported. There are reasons (some presented) why we should expect this. There are also reasons (a few listed below) why we should expect the opposite. I don't see at all why we should expect either set to dominate.

Reasons I might not post a benefit I've accrued:

1) I'm too busy out enjoying my improved life. 2) The self-congratulatory thread smells too much of an affective death spiral. 3) I am unsure how much of the benefit was actually from x-rationality and how much was from other sources. 3.1) 3, plus overcompensation for cognitive dissonance and similar biases. 4) It feels like bragging - and in fact, it seems to sometimes be interpreted this way; look at some of the reaction Luke has got for some of his posts. 5) I'm busy focusing on improving further; posting a comment listing benefits I've derived so far might not be an effective means to this goal.

"Eliezer considers fighting akrasia to be part of the art of rationality; he compares it to "kicking" to our "punching". I'm not sure why he considers them to be the same Art rather than two related Arts."

I don't understand the implications of seeing it as part of the same art or a different art altogether.

study evolutionary psychology in some depth, which has been useful in social situations

Could you elaborate on this?

I doubt that it directly told you anything useful, but it was more likely helpful in telling you to pay attention and not to interpret things through your usual beliefs.

X-Rationality can help you succeed. But so can excellent fashion sense. It's not clear in real-world terms that x-rationality has more of an effect than fashion. And don't dismiss that with "A good x-rationalist will know if fashion is important, and study fashion." A good normal rationalist could do that too; it's not a specific advantage of x-rationalism, just of having a general rational outlook.

Yet many highly intelligent people with normal rationality have terrible fashion sense, particularly males, at least in my anecdotal experience. Ditto for social skills, dating skills, etc... (fashion is really a subset of social skills, combined with aesthetics). (a) Are these people not really rationalists, because they haven't figured out how to improve themselves in those areas, or (b) do ordinary rationalists have trouble figuring out that they would benefit from improvement in those areas, and how to do it? Or perhaps (c), they recognize the benefits of greater social abilities, but they do not believe that the effort is worth it?

In principle, normal intelligent rationalists could figure out how to improve their fashion skills and social skills deliberately and systematically. But if indeed so few people in that category do so, I would take it as evidence that a systematic approach to developing interpersonal skills and style actually requires a higher level of rationality that what normal rationalists possess (perhaps x-rationality, depending on what we mean by that).

In the case of Hubbard, preaching irrationality and being irrational is different. Hubbard went genuinely crazy in his later years, but when he knew what he was doing when he invented Scientology. He even said in an interview once "I'm tired of writing for a penny a page. If a man really wanted to make a million dollars, he would invent a religion."

Reading OB/LW forced me to look hard at my contradictory beliefs about politics, and admit that I no longer believed certain things I used to believe, particularly about the market. If I don't get anything else out of it, that alone would be a large bonus.

Even before reading, I formulated an explanation for myself "all people who are too stupid not to jump off the roof will simply die out", market mechanisms and natural selection will remove all the really destructive consequences of everyday stupidity available to them, will collect all the low-hanging fruits, so the study of rationality will help only in rare or individually weak negative consequences issues. On average, rationalists will be more successful than non-rationalists, but differences between individuals will be greater than differences between groups. Nevertheless, it seems to me that knowledge of techniques would make it possible to achieve the level of Yudkovsky, but only in the case of really good knowledge of them, solving problems with acrasia and setting a serious goal.

I find my personal experience in accord with the evidence from Vladimir's thread. I've gotten countless clarity-of-mind benefits from Overcoming Bias' x-rationality, but practical benefits? Aside from some peripheral disciplines¹, I can't think of any.

Well, it did ultimately help you make SlateStarCodex and Astral Codex Ten successful, which provided a haven for non-extremist thought to thousands of people. And since the latter earned hundreds of thousands in annual revenue, you were able to create the ACX grants program that will probably make the world a better place in various small ways. Plus, people will look back on you as one of the world's most influential philosophers.

As for me, I hope to make various EA proposals informed by rationalist thought, especially relating to the cause area of "improving human intellectual efficiency", e.g. maybe sort of evidence-Wikipedia [EA(p) · GW(p)]. Mind you, the seeds of this idea came before I discovered rationalism, and "Rationality A-Z" was, for me, just adding clarity to a worldview I already already developed vaguely in my head.

But yes, I'm finding that rationalism isn't much use unless you can spread it to others. My former legal guardian recently died with Covid, but my anti-vax father believes he was killed by a stroke that coincidentally happened at the same time as his Covid infection. Sending him a copy of Scout Mindset was a notably ineffective tactic; in fact, it turned out that one month before I sent him the book, the host and founder of his favorite source of anti-vax information, Daystar, died of Covid. There's conviction—and then there's my dad. My skill at persuading this kind of person is virtually zero even though I have lots of experience attempting it, and I think the reason is that I have never been the kind of person that these people are, so my sense of empathy fails me, and I do not know how to model them. Evidence has no effect, and raising them into a meta-level of discussion is extremely hard at best. Winnifred Louis suggests (among other things) that people need to hear messages from their own tribe, and obviously "family" is not the relevant tribe in this case! So one of the first things I sent him was pro-vax messaging from the big man himself, Donald Trump... I'm not sure he even read that email, though (he has trouble with modern technology, hence the paper copy of Scout Mindset).

Anyway, while human brain plasticity isn't what we might like it to be, new generations are being born all the time, and I think on the whole, you and this community have been successful at spreading rationalist philosophy, and it is starting to become clear that this is having an effect on the broader world, particularly on the EA side of things. This makes sense! LessWrong is focused on epistemic rationality and not so much on instrumental rationality, while the EA community is focused on action; drawing accurate maps of reality isn't useful until somebody does something with those maps. And while the EA community is not focused on epistemic rationality, many of its leaders are familiar with the most common ideas from LessWorng, and so rationalism is indeed making its mark on the world.

I think a key problem with early rationalist thought is a lack of regard for coordination and communities. Single humans are small, slow, and intellectually limited, so there is little that a single human can do with rationalism all by zimself. Yudkowsky envisioned "rationality dojos" where individuals would individually strengthen their rationality—which is okay I guess—but he didn't present a vision of how to solve coordination problems, or how to construct large new communities and systems guided in their design by nuanced rational thought. Are we starting to look at such things more seriously these days? I like to think so.

1: Specifically, reading Overcoming Bias convinced me to study evolutionary psychology in some depth, which has been useful in social situations. As far as I know. I'd probably be biased into thinking it had been even if it hadn't, because I like evo psych and it's very hard to measure.

Oooh! I realize this is an old post, but I'm desperately curious for some concrete examples of this.

as robin has pointed out on numerous occasions, in many situations it is in our best interest to believe, or profess to believe, things that are false. because we cannot deceive others very well, and because we are penalized for lying about our beliefs, it is often in our best interest to not know how to believe things more likely to be true. refusing to believe popular lies forces you to either lie continually or to constantly risk your relative status within a potentially useful affiliative network by professing contrarian beliefs or, almost as bad, no beliefs at all. you're better off if you only apply ''epistemic rationality techniques'' within domains where true beliefs are more frequently or largely rewarded, i.e., where they lead to winning strategies.

trying to suppress or correct your unconscious judgments (often) requires willpower. indiscriminately applying ''epistemic rationality techniques'' may have the unintended consequence of draining your willpower more quickly (and needlessly).

winning takes time. few of us have gotten rich yet.

May I humbly suggest changing the title to "Extreme Rationality: It's Not That Great"? (This will not break any links!)

It actually just occurred to me that the intelligence professions might benefit greatly from some x-rationality. We may not have to derive gravity from an apple, but the closer we come to that ideal, the less likely failures of intelligence become.

Intelligence professionals are constantly engaged a very Bayesian activity, incorporating new data into estimates of probabilities and patterns. An ideal Bayesian would be a fantastic analyst.

If people typically found great personal benefits from reading OB/LW type material, then we would not be such a minority.

We hope that that rationality is increasing, and it could be, but I don't have much confidence that 30 years from now people, even people in positions of power, will be much more rational than they are now.

2: Eliezer considers fighting akrasia to be part of the art of rationality; he compares it to "kicking" to our "punching". I'm not sure why he considers them to be the same Art rather than two related Arts.

Winning.

Your post is a great improvement on mine. Thanks, esp. for the "limiting factor" riff.

Am I alone in thinking the word "akrasia" doesn't quite describe our problem? Isn't it more like "apathy"? Some people wish to be able to do the things they want; lucky them! Me, I just wish to want to do the things I'm able to do.

I will list the only example that comes to my mind : better x-rationality techniques have actually helped me get my university diploma : not a few times getting out of a difficult situation, where I used what I knew of heuristics, biases, the limits and usual mistakes in normal rationality, how one can sound rational regardless of whether he really is ... to give off that impressive aura of someone who knows what he's doing at little cost. To sound rational when facing an audience.

To my defense, I actually faked the cues and tells of my rationality, skill etc. to adapt them to what I did estimate to be my real level of rationality and skill, but which I also decided wouldn't be signaled correctly if I didn't actively do the job.

Now, that'd be the only time I used that knowledge in real life. But I'm not really an x-rationalist either. Even the best of us is still but a student in x-rationality. Personally, I'd just consider myself as a normal rationalist with some x-rationality ideas, someone on the transition, getting there. I mean, I can't even do the math, after all. It's all very intuitive so far, to me, not really formalized. But what I have so far, intuitively, tells me that I should seek that conversion, so as to eventually be able to use the math, and formalize that rationality. I ain't even saying that once (if ever) I'm there, I'll only used formalized x-rationality. But I'll have a new, powerful tool, maybe the most powerful, to help me wherever it may (should it be everywhere ? Perhaps not, for as long as we're meatware human; it'll still be easier to hit the ball when it feels right, rather than when you've solved its differential equations.)

And so in our present days, I wonder just how much of our art could be really be used formally, so far, as opposed to all that is still only present and usable on an intuitive level. And what part of it is really more a part of x-rationality, rather than something borrowed from someplace else.

...but you will disagree with me. And we are both aspiring rationalists, and therefore we resolve disagreements by experiments.

Your suggested experiment is good. But in this particular case, let's also try to employ the power of positivist thinking on your thesis as a whole. That is, let's break it up into a bunch of specific anticipations, and see what parts there is and isn't disagreement on, before we try to resolve those disagreements. I'll take my own stab at this with a number of short comments in a moment.

For me, the core of the rationality project is something like a determination to make your beliefs completely subservient to reality, at all costs, fighting your natural instincts for defending your beliefs, trying to win debates, etc. Not trusting your beliefs just because they are yours. Approaching the most controversial and divisive subjects with a curiosity mindset. Looking forward to changing your mind.

Most "normally rational" people can do this in technical and scientific matters, but in other domains, such as politics, philosophy, society, economics, religion and their personal lives (to some extent), such a stance is extremely uncommon, even among very "normally rational" and highly-educated people -- thus according to your taxonomy, it would fall under x-rationality, I guess (even though I'm not sure whether that's what you intended). In these areas, tribalism, confirmation bias, trying to win debates and talking past each other usually wins out.

All the other parts of x-rationality -- all the techniques and tools -- are just that, tools (of varying levels of usefulness, of course). I am not going to argue much for the value of the tools, mostly for the value of The Determination. When you have The Determination, you can seek out the tools you need.

The Determination is valuable in many ways.

People with The Determination make a better society, coordinate better, choose better policies because they actually listen to arguments from all sides and don't just fight with each other etc. And because high-level societal problems are messy and complicated, ordinary rationality is usually not enough. I know, this is not what you are aiming for, it is not a direct benefit to the practicing individual. But I think it is important and overlooked nonetheless.

Especially in light of the COVID pandemic, it is very visible how desperately we would need more Determined people. Even well-educated rational people such as doctors and biochemists sometimes spread dangerous falsehoods (and stuck with them) because they overlooked some simple fact (e.g. how an exponential growth really behaves). When confronted, they just rehearse their arguments, which make perfect sense themselves, except for the fact that they overlook a crucial consideration. When you have The Determination, you are always on the lookout for crucial considerations missed.

This is a tangible practical benefit (e.g. there would be much less dead) but it is a group benefit rather than an individual one.

As for truly individual benefits, I'm sure there are many, but they differ from person to person. For me, the most obvious one is that I can much better talk with people of widely differing opinions, I am much less confrontational and generally relate to others better. This has been hugely valuable to me.

It has also greatly helped me in matters of relationships and emotional regulation. I can for example handle my emotions in a similar way to my beliefs: not taking them at face value, treating them as a hypothesis. I can notice when I fall into harmful patterns, react poorly because of something completely unrelated, start forming an anxious-preoccupied / codependent attachment, etc. and correct myself, no matter how real it feels. This is the same mindset of never fully trusting yourself, just applied differently.

As for achieving mundane life goals such as getting jobs and money, I'd concede that x-rationality does not help that much. On the other hand, I think it can for example help you notice that you chose the wrong goals (e.g. you believe something will make you happy when in fact it won't; people are generally lousy at predicting what will make them happy). Again, the same stance helps. Treating everything, including what you want, as a hypothesis.

I think the problem with practising rationality as on LessWrong is that people end up not doing perfectly rational actions and strategies the rationale behind which they did not understand or had explained to them (usually people pick those up from environment without the explanation attached). Furthermore intelligence (as in e.g. ability to search in a big solution space for solutions) is the key requirement as well, and intelligence is hard to improve with training, especially for already well trained individuals.

I want to master x-rationality because I want to teach it. I value rational behavior in my fellow human because the historical record is clear: rational behavior is correlated with increased safety, health, and wealth of a society. I want to live in an increasingly safe, healthy, and wealthy society. I understand that "rational" behavior has a saturating plateau, or that it is only so effective, but the masters of rationality must continue to exist in every society, scientific skeptics must be cultivated. I enjoy working with the rational arts because, frankly, I grew up with a very irrational mother and I have lived ever since to do everything in my power (of almost a half a century now) to be everything she was (and still is) NOT. I want to be one of the rationality masters and teachers in my society because I enjoy those arts, find value in them, and find value in safety, health, and wealth.

Pretty selfish, but there it is.

5: In which case it will have ceased to be an experiment and become a technique instead. I've noticed this happening a lot over the past few days, and I may continue doing it.

It would still be an experiment, just a test of x-rationality (including the new technique) rather than a test of x-rationality (excluding the new technique)

And why would you want to test a version of x-rationality less than the best you have?

"techniques and theories from Overcoming Bias, Less Wrong, or similar deliberate formal rationality study programs, above and beyond the standard level of rationality possessed by an intelligent science-literate person without formal rationalist training."

Here is my definition, I attempted to rely less on CapitalNames in it.

X-Rationality:

Ability to behave according to dictates of rationality in situations where such behaviors would be highly discomforting/counter-intuitive.

As an aside the discomforting part provides an escape hatch for people not to be rational, because they can always claim high utility value of being emotionally comfortable.

X-rationality is the kind you do with math, and humans are crap at casual math, so no surprise it becomes a weapon of last resort. (We ought to be using a damn sight more math for more or less everything - the fact our cognitive architecture doesn't support it will not persuade the universe to let us off lightly).

(Edit: I removed the second half of this comment because if after a day of thinking I can't pin down what I thought I was referring to, then I'm talking nonsense. Sorry. Next time: engage brain, open mouth, in that order.)

As far as I can tell, epistemic and instrumental rationality are two related Arts, even to EY, but are both under the banner of "Rationality" because they both work towards the same goal, that of optimal thinking (I can't cite any specific examples right now, but I'll throw it out there anyway).

Also, another reason for the comparative inefficiency of x-rationality could be lack of information. Epistemic rationality is the Art of filtering/modifying information for greater accuracy. Instrumental rationality is the Art of using all available information to maximize your values. So both techniques increase the amount of benefit you gain from information. But when you don't know all that much, the fine-tuning techniques, x-rationality, would have an extremely low return since they increase your benefit by such a small percentage. There IS an element of akrasia here, in that we could go learn more if we weren't so lazy, but it's not really the same thing.

Goals are yet another problem, which you mentioned already. People just don't need rationality in routine tasks, that's what we have habits for! Would you think rationally about how to brush your teeth? More than once, then? And many of our plans for the future take large amounts of patience but not much thinking to get a 'good enough' result, so most of our focus is on being patient, the rational course of action.

There's no reason for you to change your goals just for the sake of getting to use rationality, but some other ways of getting more out of it (not necessarily the best ones, of course) could be:

• Low-"short-term"-investment tasks that would force you to study (like installing a productivity program that only allows you to access certain sites, as many people have already done)

• Increasing the entertainment value of studying, the clichéd option (OpenStax CNX has made textbooks that are slightly more interesting than normal, but I don't think it will be enough for most of the population)

• Meditation, another cliché. It increases patience, and you can work on analyzing and fixing stray beliefs you find floating around your brain

• Recording your thoughts, observations, actions, reasons for those actions, etc. in some sort of portable device (like a notebook or phone). I know this was already mentioned by Yvain, but I just want to make a single list here.

• (If you're willing to do so) Putting those recorded thoughts on LessWrong, especially the actions and their reasons, for critical review

Any other ideas?

Note: Markdown was acting up, but I've fixed it now

Also relevant

We simply don't have the time and computing power to use full rigor on our individual decisions, so we need an alternative strategy. As I understand it, the human brain operates largely on caching. X-rationality allows us to clean and maintain our caches more thoroughly than does traditional rationality. At first, it seems reasonable to expect this to yield higher success rates.

However, our intuition vastly underestimates the size of our personal caches. Furthermore, traditional rationality is simply faster at cleaning, even if it leaves a lot of junk behind. So it would appear that we should do most of the work with traditional rationality, then apply the slower x-rationality process for subtle refinement. But since x-rationality is so much slower and more difficult to run, it takes a whole giant heap of time and effort to get through a significant portion of the cache, and along the way many potential corrections will have already been achieved in the traditional rationality first pass.

But if we leave out the more rigorous methods entirely, deeming them too expensive, we're doomed to hit a pitfall where traditional rationality will not save us from thirty years of pursuing a bad idea. If we can notice these pitfalls quickly, we can apply the slow x-rationality process to that part of the cache right away, and we might only pursue the bad idea for thirty minutes instead.

We need to be able to reason clearly, to identify opportunities for clearer reasoning, and to identify our own terminal goals. A flaw in any of these pieces can limit our effectiveness, in addition to the limits of just being human. What other limiting factors might there be? What methods can we use to improve them? I keep coming back to Less Wrong because I imagine this is the most likely site to provide me with discourse on the matter.

You say that rationality only slightly correlates with winning. I think that's because incremental increases in rationality don't necessarily lead to incremental increases in winning. Winning is governed by lots of factors, and sometimes you have to get over a critical threshold of rationality to see the results you want.

I am curious about a finer distinction of what you expect to see as evidence of instrumental benefits of x-rationality:

• increased individual success/happiness
• remarkable achievement by some x-rationalists
• remarkable achievement by LW (or SIAI) as a group

"Extreme rationality" is good - but "x-rationality" is pretty jargon-dense.

Yes, it's shorter - but x-sports and x-programming are not common for a good reason - nobody will know what you are talking about. I recommend caution with the use of "x-rationality".

And this is why I am not so impressed by Eliezer's claim that an x-rationality instructor should be successful in their non-rationality life. Yes, there probably are some x-rationalists who will also be successful people. But again, correlation 0.1. Stop saying only practically successful people could be good x-rationality teachers! Stop saying we need to start having huge real-life victories or our art is useless! Stop calling x-rationality the Art of Winning! Stop saying I must be engaged in some sort of weird signalling effort for saying I'm here because I like mental clarity instead of because I want to be the next Bill Gates! It trivializes the very virtues that brought most of us to Overcoming Bias, and replaces them with what sounds a lot like a pitch for some weird self-help cult...

Yay! Cheer! Woohooo!

Thankyou Yvain! I appreciate seeing the value of epistimic rationality for humans put in a somewhat plausable perspective.

To be honest some of the greatest practical benefits I've taken from reading OB have been along the lines of "Oh, so that's what people are doing. Now I get it. I had better start doing that more". Not with all biasses mind you, but there are some biasses that are just worth implementing. This is along the lines of Eleizer's recommendation to apply 'ethics' for a pure selfish reasons. Sure, you are adding a bias, but you are at the same time accounting for calculations that are simply beyond your abilities.

he could have explained that the kids were doing some very impressive mathematics on a subconscious level

But he would have been wrong. It really is in the arm more than in the head.

EDIT: Okay, there's evidence it's largely in the brain. But talking about a 'subconscious level' isn't helpful.

A guy takes some golf lessons. Convinced he's got the mechanics of the swing down, he takes on a pro at a golf course, and has his ass handed to him. "Those golf lessons did me no good", he says. "Do golf lessons even correlate with being good at the sport?".

fantastic !!!

Check out Gladwell's new book - Outliers. Our success cannot be attributed to our individuality to the degree that most American's think it can. There are huge cultural influences, arbitrary society rules, birth year, etc... There's a chapter on why high IQ only matters to a certain point. Once you're "inteligent enough", practical wisdom takes over in determination of success. I don't think akrasia has that much to do with it. We live in a world of lower inteligence and have to play by those rules. It pays to be ONE step ahead of the mob, not 10! You cannot make money in the stock market by being 10 steps ahead. You'd be shorting stocks into oblivious in the 90's and 2006-2007, while the mob was getting more and mroe exuberant. By the same token i don't think success is a good measuring stick for rationality. Success depends on irrational subjects' interpretation and understanding of your actions, which cannot happen by definition. Unless in your endevours outside your brain you dumb it down to be one step ahead instead of ten, but then you gotta think like a dummy, which is not a skill of rationality.

I second the entire post except for the Galt-worshiping. If good and evil worked like they do in D&D and were real ontological categories, Ayn Rand would be the highest dark priestess in recorded history.