The Martial Art of Rationality
post by Eliezer Yudkowsky (Eliezer_Yudkowsky)
I often use the metaphor that rationality is the martial art of mind. You don’t need huge, bulging muscles to learn martial arts—there’s a tendency toward more athletic people being more likely to learn martial arts, but that may be a matter of enjoyment as much as anything else. If you have a hand, with tendons and muscles in the appropriate places, then you can learn to make a fist.
Similarly, if you have a brain, with cortical and subcortical areas in the appropriate places, you might be able to learn to use it properly. If you’re a fast learner, you might learn faster—but the art of rationality isn’t about that; it’s about training brain machinery we all have in common. And where there are systematic errors human brains tend to make—like an insensitivity to scope—rationality is about fixing those mistakes, or finding work-arounds.
Alas, our minds respond less readily to our will than our hands. Our ability to control our muscles is evolutionarily ancient; our ability to reason about our own reasoning processes is a much more recent innovation. We shouldn’t be surprised, then, that muscles are easier to use than brains. But it is not wise to neglect the latter training because it is more difficult. It is not by bigger muscles that the human species rose to prominence upon Earth.
If you live in an urban area, you probably don’t need to walk very far to find a martial arts dojo. Why aren’t there dojos that teach rationality?
One reason, perhaps, is that it’s harder to verify skill. To rise a level in Tae Kwon Do, you might need to break a board of a certain width. If you succeed, all the onlookers can see and applaud. If you fail, your teacher can watch how you shape a fist, and check if you shape it correctly. If not, the teacher holds out a hand and makes a fist correctly, so that you can observe how to do so.
Within martial arts schools, techniques of muscle have been refined and elaborated over generations. Techniques of rationality are harder to pass on, even to the most willing student.
Very recently—in just the last few decades—the human species has acquired a great deal of new knowledge about human rationality. The most salient example would be the heuristics and biases program in experimental psychology. There is also the Bayesian systematization of probability theory and statistics; evolutionary psychology; social psychology. Experimental investigations of empirical human psychology; and theoretical probability theory to interpret what our experiments tell us; and evolutionary theory to explain the conclusions. These fields give us new focusing lenses through which to view the landscape of our own minds. With their aid, we may be able to see more clearly the muscles of our brains, the fingers of thought as they move. We have a shared vocabulary in which to describe problems and solutions. Humanity may finally be ready to synthesize the martial art of mind: to refine, share, systematize, and pass on techniques of personal rationality.
Such understanding as I have of rationality, I acquired in the course of wrestling with the challenge of artificial general intelligence (an endeavor which, to actually succeed, would require sufficient mastery of rationality to build a complete working rationalist out of toothpicks and rubber bands). In most ways the AI problem is enormously more demanding than the personal art of rationality, but in some ways it is actually easier. In the martial art of mind, we need to acquire the realtime procedural skill of pulling the right levers at the right time on a large, pre-existing thinking machine whose innards are not end-user-modifiable. Some of the machinery is optimized for evolutionary selection pressures that run directly counter to our declared goals in using it. Deliberately we decide that we want to seek only the truth; but our brains have hardwired support for rationalizing falsehoods. We can try to compensate for what we choose to regard as flaws of the machinery; but we can’t actually rewire the neural circuitry. Nor may martial artists plate titanium over their bones—not today, at any rate.
Trying to synthesize a personal art of rationality, using the science of rationality, may prove awkward: One imagines trying to invent a martial art using an abstract theory of physics, game theory, and human anatomy.
But humans arent reflectively blind. We do have a native instinct for introspection. The inner eye isnt sightless, though it sees blurrily, with systematic distortions. We need, then, to apply the science to our intuitions, to use the abstract knowledge to correct our mental movements and augment our metacognitive skills.
We aren't writing a computer program to make a string puppet execute martial arts forms; it is our own mental limbs that we must move. Therefore we must connect theory to practice. We must come to see what the science means, for ourselves, for our daily inner life.
Comments sorted by oldest first, as this post is from before comment nesting was available (around 2009-02-27).
comment by Robin_Hanson2 ·
2006-11-22T21:02:34.000Z · LW(p) · GW(p)
To continue with this metaphor, it seems what we need is a good set of problems to test our rationality (i.e., ability to resist common biases). As with any cognitive test, the details of the test can't be known in advance, or people would just memorize answers instead of developing skills. So we need to collect a rather large set of good test questions, or create a generators to create sets of them automatically. Not easy, but perhaps worth doing.
comment by MichaelAnissimov ·
2006-11-23T01:14:29.000Z · LW(p) · GW(p)
It seems that calibrating personal judgements using decision markets could in principle be a way to measure rationality. You make a guess, and if your guess is close to what the market predicts, you can be slightly more confident about yourself. Of course, for this to work, you'd have to avoid prior knowledge of the market, and skip participating in betting on topics that you want to consider unassisted first.
Great post btw.
comment by Pensans ·
2006-11-23T02:02:18.000Z · LW(p) · GW(p)
Do we want "techniques" for finding the truth? I suppose if the truth is "the weight of this fruit for sale." But if the truth is more like "what I ought to be doing," there is something closer to the essence of personality involved. If my personality is gained as a technique at a truth dojo, then it does not feel like mine. The necessity of discovering certain truths personally rather through technical instruction is a limit on the usefulness of teaching.
Ultimately, the nature of rationality is a question that goes to essence of personality. If I am willing to receive only deductively derived formula as rational, then I have -- not only a technique of rationality -- but a particularly well-known and annoying personality. If I am willing to act on any emotional or intuitive vapor, then I have another well-known and annoying personality. We are all aware that the grounds upon which I am willing to believe and act are not just techniques of rationality -- they are part of our fundamental identities and determine the way that we form or do not form social relations with others.
comment by Robin_Hanson2 ·
2006-11-23T02:07:34.000Z · LW(p) · GW(p)
Michael, you can't have a prediction market without a way to pay off the bets, and if you have that you can measure personal accuracy directly, if you just wait.
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) ·
2006-11-23T03:51:14.000Z · LW(p) · GW(p)
Robin, I agree that the main difficulty is figuring out how to pay off the bets, but it seems to me that - given such a measure - playing a prediction market around the measure makes the game more complex, and hopefully more of a lesson, and more socially involving and personally intriguing. In other words, it's the difference between "Guess whether it will rain tomorrow?" and "Bob is smiling evilly; are you willing to bet $50 that his probability estimate of 36.3% is too low?" Or to look at it another way, fewer people would play poker if the whole theme was just "Estimate the probability that you can fill an inside straight." I think Anissimov has a valid fun-amplifying suggestion here.
comment by Robin_Hanson2 ·
2006-11-23T13:13:05.000Z · LW(p) · GW(p)
Psychologists have developed a self-deception test, which includes questions like "Did you ever hate your mother," on the assumption that most everyone did but few want to admit it. See:
Paulhus, Delroy L. "Self-deception and Impression Management in Test Responses." In Angleitner, A. & Wiggins, J. S., Personality assessment Via Questionnaires. New York, NY: Springer, 1986, 143-165.
Perhaps questions like those might be a good part of a rationality test and practice regime .
Replies from: wizzwizz4
↑ comment by wizzwizz4 ·
2019-07-15T21:22:18.592Z · LW(p) · GW(p)
I'm not so sure. Such "probabilistic" tests are good for aggregate testing, but not for personal testing. We want to minimise false positives and false negatives.
comment by Joshua_Fox ·
2006-11-23T15:40:26.000Z · LW(p) · GW(p)
Why aren't there dojos that teach rationality?
They're called universities. My PhD adviser taught me to think rationally and caught many of the errors in my thinking, including those caused by biases. This eventually allowed me to catch many of my own errors.
No, it wasn't perfect. It was limited to one academic area and was not explicitly aimed at teaching rationality and overcoming biases. (Then again, martial-arts senseis do not teach their skills "explicitly", but by repetition and training.) Moreover, gaining subject-matter expertise took as much of the effort as learning rational thinking processes. But a PhD in a rigorous field from a good university is as close as you'll find to a "color belt" (dare I say "black belt"?) in rationality.
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) ·
2006-11-23T20:14:28.000Z · LW(p) · GW(p)
Joshua, the thought had occurred to me, but with all due respect to universities, that's the same sort of training-in-passing that you get from reading "Surely You're Joking, Mr. Feynman" as a kid. It's not systematic, and it's not grounded in the recent advances in cognitive psychology or probability theory. If we continue with the muscle metaphor, then I would say that - if judged by the sole criterion of improving personal skills of rationality - then studying physics is the equivalent of playing tennis. If you actually want to do physics, of course, that's a whole separate issue. But if you want to study rationality, you really have to study rationality; just as if you wanted to study physics it wouldn't do to go off and study psychology instead.
comment by Steve_Sailer ·
2006-11-24T19:48:48.000Z · LW(p) · GW(p)
I think you are accepting the advertising hooey of martial arts dojos too much at face value. Most people don't take martial arts for all the abstracted reasons that appear in the brochures, but because they want to feel they can win fights with other people.
Similarly, people don't want to learn how to be rational for the sake of being rational. Instead, you have to sell rationality for more human ends, such as being able to win arguments, or make money, or understand a particular field such as baseball statistics. You can learn a lot of general lessons about rationality from reading Bill James on baseball statistics, but it's not very exciting to study rationality for the sake of being rational.
Replies from: johnlawrenceaspden
↑ comment by johnlawrenceaspden ·
2012-10-09T20:01:44.033Z · LW(p) · GW(p)
I did judo as a child. At first it was because my parents thought I'd develop physical self-confidence that way. It worked (and judo's something I'd recommend to any boy), but it doesn't take much training before you can win fights against most people. After that you stay interested because you want to beat other people at judo, just like any other sport.
So you might sell your rationality dojos as being about winning arguments, but pretty soon I imagine that the practitioners might get more interested in being right.
Most people will get very competitive the minute they've got something they can measure. Is there any form of transport which isn't raced? And most non-competitive people will get very competitive the moment they're in a fight they can win.
Suddenly I'm imagining a room where the sensei is describing the Amanda Knox case, and the students are asking him questions and debating with each other, and after all the talking is done, people place and take bets on the result at various odds, and then the sensei reveals the actual answer and the correct reasoning. And ranking points are transferred accordingly, and there is a ladder on the wall where names are listed in order with the scores.
I already reckon I could seriously enjoy such a game. Who wants to play?
Replies from: None, Morendil
↑ comment by [deleted] ·
2012-10-09T20:04:57.740Z · LW(p) · GW(p)
I'm in as long as we don't discuss the Amanda Knox case.
Replies from: johnlawrenceaspden
↑ comment by johnlawrenceaspden ·
2012-10-10T19:18:19.338Z · LW(p) · GW(p)
I mentioned it only because it seems to have been a unique triumph for Less Wrong. I'd read about the case, and thought nothing of it particularly. And then people here started saying "Look at it from a probabilistic point of view", and so I did, and after a few hours head-scratching and diagram-drawing I realized that it was almost certainly a miscarriage.
I mentioned this to a few people I know, and they reacted pretty well as you'd expect to a middle-aged man suddenly getting a bee in his bonnet about a high-profile sex murder case involving pretty girls.
When she was eventually acquitted, various people said "How did you do that?". And the mathematically minded types were quite impressed with the answer, while the muggles think I've got some sort of incomprehensible maths-witchcraft thing that I can do to find out the truth.
Which is exactly the sort of thing you might want to sell, if you can find a way to teach it.
Replies from: Eliezer_Yudkowsky, None, shminux
↑ comment by shminux ·
2012-10-10T22:38:32.305Z · LW(p) · GW(p)
I mentioned it only because it seems to have been a unique triumph for Less Wrong.
Why has it been so unique? Surely there are plenty of high-profile predictions one can make using the same Bayesian techniques? (Or one can simply ask gwern, who is apparently well calibrated after a thousand or so recorded predictions.)
Replies from: somervta, johnlawrenceaspden
↑ comment by johnlawrenceaspden ·
2012-10-12T17:49:53.937Z · LW(p) · GW(p)
Well actually I was just wondering about that.
What other claims like 'Amanda Knox is innocent' can we make, in the sense that (a) they're counter common thinking (b) we're pretty sure we're right (c) there's likely to be a resolution in our favour soon?
The Amanda Knox thing was a surprising prediction that came true. More of those would be neat.
comment by Paul_Gowder ·
2006-11-24T21:33:27.000Z · LW(p) · GW(p)
Steve: Wasn't that the claim of the sophists? "We'll teach you how to win arguments so you can prevail in politics." The problem is that the skills for winning arguments aren't necessarily the schools for rationality in general. Probably the easiest way to learn the skills for winning arguments is to go to law school and "learn to think like a lawyer."
comment by pdf23ds ·
2006-11-28T02:18:56.000Z · LW(p) · GW(p)
Steve, you're probably right, but I doubt that's really a big obstacle. Rationality has plenty of uses, and if anyone wants to teach it for money, they shouldn't have problems finding ways to promote it, in the long run. Besides, if people want to learn karate to beat people up, why don't dojos advertise based on that?
But I think that the analogy to martial arts is ill-advised for other reasons. Martial arts is a mostly anachronistic practice when evaluated against its original purpose. If you're really serious about self-defense nowadays, you get weapons. (Weapons have always been an advantage, but it's only been for a few hundred years now that they've become such a decisive advantage.)
Where are the tools for giving us a rational advantage? Sure, we have plenty of them for bringing knowledge to us quickly. The internet has been pretty successful at that. But what about other prostheses that specifically target our rational acuity?
comment by Dylan ·
2009-07-22T22:24:54.672Z · LW(p) · GW(p)
I'll go with the martial arts metaphor, it has at least limited application, and is already the reference point for the discussion. As far as there being no rationality dojos, what about Buddhism, or any other mystic or contemplative path along religious lineages?
As far as measuring rationality or rational skills, the main problem is there is no generally agreed upon Truth or even agreement as to whether Truth even exists. This discussion is not about the actual content of truth per se but about the integrity of the approach to truth. This may be soundly based on the idea that we are only approaching knowledge of truth, but truth itself is still not known. So we cannot measure the soundness as a function of measurable results, since the results are either non-existent,unknown or not agreed upon.
It seems that if this is the case then any form of measurement cannot be objective or externally made, but must be subjective and internally assessed. So the question would become not what is true as fact, but what is true as honest. My vote would be for rational integrity as the context within which skill or understanding would be viewed.
Replies from: orthonormal, jonathan-doolin
↑ comment by orthonormal ·
2009-07-22T22:44:21.815Z · LW(p) · GW(p)
Welcome to Less Wrong, Dylan! Check out the welcome thread and introduce yourself.
I agree that one can develop a pretty impressive Art based on destroying self-deception rather than on seeking truth, particularly when 'seeking truth' sounds like an exceptionally problematic phrase.
But it might not have to be that problematic. Read The Simple Truth if you want to know what Eliezer means by truth; it's much less naive than I'd expected when I was new around here.
comment by [deleted] ·
2010-05-26T02:01:08.037Z · LW(p) · GW(p)
Just as one style of martial arts benefits mainly one area, say the feet, another is totally dedicated to the fists. To measure the growth in each of these forms, there are different 'belts' or levels of mastery based on what you have learned. So perhaps rationality cannot be measured in just one way; the different applications would have to have different levels, just as schools have letter grades on different subjects, and then an overall GPA. Practical usage, such as in the workplace, would have to be 'graded' differently than the same practical usage in a social setting.
comment by thomblake ·
2011-05-11T22:37:59.093Z · LW(p) · GW(p)
Re-reading this brings to mind the history of the Logicians in ancient China. Apparently, they fell out of favor because all that their skills seemed to be good for was winning arguments, and others already had much more effective ways of winning arguments. Advertising rationality as a way to win arguments, as suggested by some of the original commenters, could cause rationality to fall into a similar trap.
I think this problem is avoided by your (later) idea of a Bayesian Conspiracy; if practitioners of beisudo were demonstrably better at the various instrumental tasks they undertook, then that might be enough advertising to attract people to rationality.
I also wonder if cargo-cult rationality would be good or bad. It seems like it could go either way.
comment by chatquitevoit ·
2011-07-18T16:08:46.749Z · LW(p) · GW(p)
I wonder how much of a correlation there is between people who put effort into self-training in rationality (or communal training, a la Less Wrong) and those who actually train a martial art. And I don't mean in the "Now I'lll be able to beat up people Hoo-AH" three-week-course training - I mean real, long-term, long-rewards-curve training. I've done aikido on and off for years (my life's been too hectic to settle down to a single dojo, sadly), and it takes a similar sort of dedication, determination, and self-reflection as a serious foray into training your mind to rationality. And, I'd go so far as to say, a similar 'predilection of mind and preference' (and I'll let you LWers go to town on that one).
Replies from: Navanen
↑ comment by Navanen ·
2012-12-08T04:52:16.437Z · LW(p) · GW(p)
What are you meaning by correlation? Do you mean how similar the thinking of those who mindfully approach rationality training is to the thinking of people who seriously dedicate themselves to practicing a martial art? Or do you mean something else?
comment by boilingsambar (AbyCodes) ·
2011-07-24T11:23:11.796Z · LW(p) · GW(p)
Why aren't there dojos that teach rationality?
There is. Buddhist temples in the Himalayas (Bhutan and the neighbouring countries) and remote china. I lived in Bhutan as a child. All buddhist monks lived their lifes' in monasteries; meditating, contemplating, doing the daily chores, in introspection and following the teachings of Buddha. Isn't that a "dojo of rationality"?
How to communicate procedural skills of rationality, or measure them, is probably the single largest open issue that stands between humanity and rationality dojos - at least it's the part of the problem that most baffles me.
Discussion; as is with every other field.
Replies from: TheWakalix
↑ comment by TheWakalix ·
2019-02-19T17:59:20.987Z · LW(p) · GW(p)
Isn't that a "dojo of rationality"?
How is that "rationality"? They spent a lot of time thinking, yes, but not all thought is rational. Not even all introspection is rational.
comment by Arandur ·
2011-08-13T01:06:56.275Z · LW(p) · GW(p)
Well, if we have a Bayesian Conspiracy in place, as it were, then I reckon the test would be derivation of hidden knowledge from available priors. This would require that the Conspiracy have access to answers reasoned out ahead of time, and also that the actual testing was kept secret. Perhaps an aspiring rationalist may only be tested once. If we have that strict a measure, though, we might consider allowing a certain amount of leniency... but then again, perhaps only those who strictly adhere to the Art deserve to move further into its ranks.
I am quite aware that my idea is based largely on a post you made about 16 months after this one; in that sense, I'm simply perpetuating a feedback loop, and may be acting in redundancy. But the more I think on this problem, the more convinced I am that it is a Good Idea: test, ceremony, Conspiracy, and all.
comment by witzvo ·
2012-05-30T03:03:26.645Z · LW(p) · GW(p)
Computer games, done properly, could be a good teacher of rationality. If you fail to observe the laws of nature that happen to apply in the game and act accordingly, you die / can't get to the goal. Also, debugging programs has been good for me: plenty of times I could not see an error in my algorithm, but knew there had to be one (modulo system bugs), and so I learned what does and does not constitute valid argument that way. Put these ideas together (games where you have to program things that work in the game universe) and it'd be interesting. Some AI competitions work that way, but we could probably build an iPad app for kids with the right ideas. For example, I'm not talking about teaching basic or some terrible "easy" language. Why not base the programming language of the game off some other Turing-complete system, like, say, cellular automata. The automata rules could change from time to time to keep them guessing. The game could be a sequence of problems: fill in these squares correctly in order to get the desired result and move to the next level. Actually, maybe Markov algorithms would be better. Or a version of the untyped lambda calculus (http://worrydream.com/AlligatorEggs/). Having a good baseline programming "world" is an important part of getting them to think about the right kinds of abstractions.
Replies from: RichardKennaway, Desrtopa, DanWilliams
↑ comment by RichardKennaway ·
2012-05-30T10:45:55.827Z · LW(p) · GW(p)
Computer games, done properly, could be a good teacher of rationality. If you fail to observe the laws of nature that happen to apply in the game and act accordingly, you die / can't get to the goal.
The laws may carry their own message.
Space Invaders: However high you build your score, the alien horde (i.e. Death) crushes you in the end.
First Person Shooters: Killing your enemies is fun!
Tetris, Kingdom of Loathing, Evony, etc.: Clicky-clicky-clicky! Better than life!
Replies from: witzvo
↑ comment by witzvo ·
2012-06-01T00:46:17.540Z · LW(p) · GW(p)
That's not quite what I had in mind, but, well, if we make our own game it can have it's own laws. For example, you gain power by understanding the rules of the game and acting accordingly, not just by reflexes or level grinding. If you understand the rules keenly, you act in subtly different ways that are vastly more effective, perhaps. Maybe you can collaborate to achieve more than you can individually, etc....
Here's a game that, at least, makes you figure out the game mechanics a bit to live:
Variations on this requiring more clever tool building, say, would be interesting. But I think it is possible to build a game around a turing machine if we are clever about it.
comment by Lander7 ·
2017-07-01T14:05:45.318Z · LW(p) · GW(p)
An AI psychologist would stand the best chance of learning human interactions and rational. An AI in that position could quickly learn how we work and understand things. The programmers would also have a great view of how the AI responds.
As for the Dojo idea, we may now have that with, "luminosity.com", possibly the world's first online mental Dojo. A local Dojo would lack the visibility needed to ensure the correct training was being applied. If it were just local it would be more like theology rather than rational learning.
I try to explore ideas and concepts that put people outside the realm or normality on my site thinkonyourown.com, I've found that challenging people's limits is a fantastic way of exercising the mental muscles.
comment by Jonathan Doolin (jonathan-doolin) ·
2018-05-26T14:47:29.995Z · LW(p) · GW(p)
I was arguing the other day with someone online, and I said to him " I think you're trying to throw me off your scent here... You are cornered, and trying to confuse the issue by claiming I'm arguing for this idea, and you've been arguing against it. "
The truth is, that I had not cornered him. The reason I couldn't corner him was in order to corner someone in a rational debate, they have to allow themselves to become cornered... That is, they have to explain themselves so well, what their thoughts and ideas actually are that they are "nailed down".
If their ideas are valid, from a scientific perspective, then this position of being cornered and nailed down, is actually the position of ultimate strength. It is exactly the opposite of where you would want to be in a physical fight.
What is often done, in an argument with a rationalist, who is trying to be cornered and nailed down, so that he can argue from his position of greatest strength, is that the attacker, knowing that he cannot actually attack the argument being defended, must create a strawman argument.
While this does nothing to the beliefs of the person defending the argument, it does confuse other participants or bystanders in the argument, and may succeed in impugning the integrity of a person defending a true idea... e.g., put him in an illusory box that gives others an excuse not to listen to him.
So, as a rule, one can be more successful in seeking truth, by looking for the participant in the debate who is defending and explaining their own ideas, with an honest desire to be cornered and nailed down. They will not feint, or dodge, but will stand where they are, defending their ground, until such time as they are exposed to a better idea.