Rational Me or We?

Robin Hanson has identified a breakdown in the metaphor of rationality as martial art: skillful violence can be more or less entirely deferred to specialists, but rationality is one of the things that everyone should know how to do, even if specialists do it better. Even though paramedics are better trained and equipped than civilians at the scene of a heart attack, a CPR-trained bystander can do more to save the life of the victim due to the paramedics' response time. Prediction markets are great for governments, corporations, or communities, but if an individual's personal life has gotten bad enough to need the help of a professional rationalist, a little training in "cartography" could have nipped the problem in the bud.

To put it another way, thinking rationally is something I want to do, not have done for me. I would bet that Robin Hanson, and indeed most people, respect the opinions of others in proportion to the extent that they are rational. So the individual impulse toward learning to be less wrong is not only a path to winning, but a basic value of a rationalist community.

Yes, it would be silly to think of ourselves as isolated survivalists in a society where so many people are signed up for cryonics, where Many-Worlds was seen as retrospectively obvious as soon as it was proposed, and no one can be elected to public office if they openly admit to believing in God. But let us be realistic about which Earth we actually live in.

I too am greatly interested in group mechanisms of rationality - though I admit I put more emphasis on individuals; I suspect you can build more interesting systems out of smarter bricks. The obstacles are in many ways the same: testing the group, incentivizing the people in it. In most cases if you can test a group you can test an individual and vice versa.

But any group mechanism of that sort will have the character of a band of survivalists getting together to grow carrots. Prediction markets are lonely outposts of light in a world that isn't so much "gone dark" as having never been illuminated to begin with; and the Policy Analysis Markets were burned by a horde of outraged barbarians.

We have always been in the Post-Apocalyptic Rationalist Environment, where even scientists and academics are doing it wrong and Dark Side Epistemology howls through the street; I don't even angst about this, I just take it for granted. Any proposals for getting a civilization started need to take into account that it doesn't already exist.

For example, with subsidized prediction markets, we can each specialize on the topics where we contribute best, relying on market consensus on all other topics.

If no one person has a good grasp of all the material, then there will be significant insights that are missed. Science in our era is already dominated by dumb specialists who know everything about nothing. EY's work has been so good precisely because he took the effort to understand so many different subjects. I'll bet at long odds that a prediction market containing an expert on evo-psych, an expert on each of five narrow AI specialisms, an expert on quantum mechanics, an expert on human biases, an expert on ethics and an expert on mathametical logic would not even have produced FAI as an idea to be bet upon.

We don't each need to train to identify and fix each possible kind of bias; each bias can instead have specialists who look for where that bias appears and then
correct it.

If people could see inside each others' heads and bet on (combinations of) people's thoughts, this would work.

In reality, what will happen is that a singly debiased single subject specialist will simply not produce any ideas for the prediction market that (a) involve more than his specialism and (b) would require him to debias in more than one way.

For example, a logic expert who suffers from overconfidence in the effectiveness of logic in AI will not hypothesize that maybe something other than a logical KR is appropriate for the semantic web. [people in my research group were shocked when I produced this hypothesis] A bayesian stats researcher will not produce this hypothesis because he doen't know the semantic web exists; it isn't part of his world.

What I am driving at with this comment is that the strength of connection between thoughts held in one mind is much greater than the strength of connection between thoughts in a market. In a market, two distinct predictions interact in a very simple way: their price. In a mind, two or more insights can be combined. If no individual mind is bias-free, then we lose this "single mind" advantage. [Apologies for comment deletion. It would be nice to have a preview button...]

One problem with trusting the experts rather than trying to think things through for yourself is that you need a certain amount of expertise just to understand what the experts are saying. The experts might be able to tell you that "all symmetric matrices are orthonormally diagonalizable," and you might have perfect trust in them, but without a lot of personal study and inquiry, the mere words don't help you very much.

Following the martial arts analogy, I guess that makes Robin a supporter of "Rationalist Gangs".

One of the ways that I think that OB could have been better, and that I think LW could be more helpful, is to put a greater emphasis on practice and practical techniques for improving rationality in the writings here and to give many more real-life examples than we do.

When making a post that hints at any kind of a practical technique, posters could really make an effort to clearly identify the practical implications and techniques, to put all the practical parts together in the essay rather than mixing them throughout 15 paragraphs of justification and reasoning, and to highlight that practical part of the post.

The practical parts could be extracted and placed together somewhere in order to have one single place that people can go to easily find them. Perhaps the LW software could provide some kind of support for distinguishing the practice sections of a post, and the extraction and aggregation of the practical howto sections could be automated.

Robin was kind enough not to say what overemphasizing the heroic individual rationalist implies about our true motivations.

I love your thesis and metaphor, that the goal is for us all jointly to become rational, seek, and find truth. But I do not "respect the opinions of enough others." I have political/scientific disagreements so deep and frequent that I frequently just hide them and. worry. I resonated best with your penultimate sentence: "humanity's vast stunning cluelessness" does seem to be the problem. Has someone written on the consequences of taking over the world? The human genome, presumptively adapted to forward it's best interests in a competitive world, may have only limited rationality, inadequate to the tasks of altruism, global thinking, and numerical analysis. By this last phrase I refer to our overreaction to a burning skyscraper, when an equal number of deaths monthly on freeways, by being less spectacular or poignant, motivates a disproportionately low response. Surely the difference there is a "gut" reaction, not a cogent one. We need to change what we care about, but we're hardwired to worry about spectacle, perhaps?

Maybe personal finance is a better analogy than Martial Arts. It's useful for nearly anybody to know about personal finance, yet many people are lacking even in the basics. Some high-falutin stock market concepts may not be useful to the average joe, the same way advanced rationality ("better then Einstein") may not be needed, but still, education about the basics is useful.

For whatever reason, the community here (so-called "rationalists") is heavily influenced by overly-individualistic ideologies (libertarianism, or in its more extreme forms, objectivism). This leads to ignoring entire realms of human phenomena (social cognition) and the people who have studied them (Vygotsky, sociologists of science, ethnomethodology). It's not that social approaches to cognition provide a magic bullet -- they just provide a very different perspective on how minds work. Imagine if you stop believing that beliefs are in the head and locate themselves in a community or institution. If interested, you could start with How Institutions Think by Mary Douglas.

Martial arts can be a good training to ensure your personal security, if you assume the worst about your tools and environment. If you expect to find yourself unarmed in a dark alley, or fighting hand to hand in a war, it makes sense. But most people do a lot better at ensuring their personal security by coordinating to live in peaceful societies and neighborhoods; they pay someone else to learn martial arts. Similarly, while "survivalists" plan and train to stay warm, dry, and fed given worst case assumptions about the world around them, most people achieve these goals by participating in a modern economy.

As a martial arts enthusiast I have to concur that the practical survivability impact of my training is somewhat limited. In fact, I would go as far as to say that my martial art training is far less likely to save my life than is my previous sporting hobby, running.

The martial arts metaphor for rationality training applies as much to my motives for participation as it does for the training itself. I don't expect to beat many armed assailants to a pulp in a dark alley and nor do I expect elimitating biasses from my cognition to make a dramatic impact on my success or life satisfaction. However, I relish every opportunity to push both my body and mind to their limits in elegance and performance. I am also attracted to subcultures that tend to non-exclusivity with skill based elitism.

I unashamedly confess that I'd be a rationalist even if it had absolutely no direct benefit (over participation in the activities of any other arbitrary non-rationilist subculture to a similar degree). But at the same time I have to concur with Robin on the best way to go about finding truth.

But for those of us who respect the opinions of enough others to want to work with them to find truth, it makes more sense to design institutions which give each person better incentives to update a common consensus.

Absolutely. There is just something comforting in knowing that if the information I am relying apon is flawed, someone is losing money because of it. It's even better to know that if you do find flaws you'll be rewarded for doing so, not hunted down and persecuted as a 'whistleblower' or a heretic.

Unfortunately 'designing institutions' doesn't sound like the hard part. The hard part is taking these institutions and making them an active reality. Diluting the influence of authority tends to go against the interests of those in authority, at least how they perceive it. Of course, that particular robotic rampage of human stupidity is not something I personally need to overcome with my own rationalist-fu. I can respect the opinions of Robin et. al. and eagerly keep abreast of their instights and practical solutions.

On this point, we should also be talking about effective evangelism for rationality.

We should learn how to identify trustworthy experts. Is there some general way, or do you have to rely on specific rules for each category of knowledge?

Two examples of rules are never trust someone's advice about which specific stocks you should buy unless the advisor has material non-public information, and be extremely skeptical of statistical evidence presented in Women Studies' journals. Although both rules are probably true you obviously couldn't trust financial advisers or Women Studies' professors to give them to you.

Another good example is the legal system. Individually it serves many participants poorly on a truth-seeking level; it encourages them to commit strongly to an initial position and make only those arguments that advance their cases, while doing everything they can to conceal their cases' flaws short of explicit misrepresentation. They are rewarded for winning, whether or not their position is correct. On the other hand, this set-up (in combined with modern liberalized disclosure rules) works fairly well as a way of aggregating all the relevant evidence and arguments before a decisionmaker. And that decisionmaker is subject to strong social pressures not to seek to affiliate with the biased parties. Finally, in many instances the decisionmaker must provide specific reasons for rejecting the parties' evidence and arguments, and make this reasoning available for public scrutiny.

The system, in short, works by encouraging individual bias in service of greater systemic rationality.

The rationality dojo seems to be part of a world where "we" work together for truth, at least if you don't take the dojo metaphor too seriously. I assume that training individuals to be more rational is part of your optimal strategy. So I take it that you argument is that we should emphasize individual training less relative to designing institutions which facilitate truth-finding despite our biases. Am I understanding you correctly?

Robin wrote: "Martial arts can be a good training to ensure your personal security, if you assume the worst about your tools and environment." But this does not mean that martial arts cannot also be good training if you assume a more benign environment. Environments are known to be unpredictable.

One of the most important insights a person gains from martial arts training is to understand one's limits--which relates directly to the bias of overconfidence. If martial arts training enables a person to project an honestly greater degree of self confidence, then the signaling benefit alone may merit the effort. Does rationality training confer analogous signaling benefits?

An attempt to even find Einsteins is doomed unless the number of them is large enough as a fraction of the population. (cf: Eliezer's introduction to Bayes.)

On the other hand, a purely aggregate approach is a dirty hack that somehow assumes no (irrational) individual is ever able to be a bottleneck to (aggregate) good sense. It's also fragile to societal breakdown.

It seems evident to me that what's really urgent is to "raise the tide" and have it "lift all boats". Because then, tests start working and the individual bottleneck is rational.

"How can we join together to believe truth?"

Yes!

I am being deluged here on LW by all the posts and comments. Spending so much time in front of the screen does not seem sensible or rational.

What to do?

For example, with subsidized prediction markets, we can each specialize on the topics where we contribute best, relying on market consensus on all other topics.

If no one person has a good grasp of all the material, then there will be insights that are missed. Science in our era is already dominated by dumb specialists who know everything about nothing. EY's work has been so good precisely because he took the effort to understand so many different subjects. I'll bet at long odds that a prediction market containing an expert on evo-psych, an expert on each of five narrow AI specialisms, an expert on quantum mechanics, an expert on human biases, an expert on ethics and an expert on mathametical logic would not even have produced FAI as an idea to be bet upon.

I was just about to respond by asking if you would advocate a website in the beliefs of the members are aggregated based on their reliability, then I remembered: prediction markets.

I'm guessing you're not pushing a real prediction market due to legal issues, but why not create one that uses token points instead of real money?

My first thought was slightly different: have testable predictions, as in a market, but the system treats each persons' likelihood ratios as evidence (as well as the tags for the prediction, to account for each person's area of expertise)

It seems to me that the real issue still is a supply of testable problems.

Aren't we the supposed martial rationalists of the humanity? Aren't we the ones being paid (I wish) to protect the neighborhoods from the marauding apologists? Aren't we the ones to go to the wild places and battle dragons?

Division of labour

I would guess that martial arts are so frequently used as a metaphor for things like rationality because their value is in the meta-skills learned by becoming good at them. Someone who becomes a competent martial artist in the modern world is:

• Patient enough to practice things they're not good at. Many techniques in effective martial arts require some counter-intuitive use of body mechanics that takes non-trivial practice to get down, and involve a lot of failure before you achieve success. This is also true of a variety of other tasks.

• Possessing the fine balance of humility and confidence required to learn skills from other people. Generally if you're going to get anywhere in martial arts, you're not going to derive it from first principles. This is true of most human knowledge domains. Learning to be a student or apprentice is valuable, as is learning to respect the opinions of others when they demonstrate their competence.

• Practiced in remaining calm and thinking strategically under pressure. If one is taught to competently handle a high-stress situation such as a physical fight, one can make decisions quickly and confidently even when stressed. This skill is useful for reasons I hope I don't have to go into depth on.

• Able to engage mirror neurons to understand and reason about the nonverbal behavior of other humans, and somewhat understand their intentions and strategies. This is useful in a fight and taught by many martial arts, but extremely useful in other contexts, not the least of which being negotiation with semi-cooperative individuals.

• Probably pretty physically fit. It's a decent whole-body exercise regimen, and there are numerous benefits to exercising frequently and keeping in good shape. It is probably not the most efficient exercise regimen out there by a long shot, but it may be one that is intrinsically fun to do for a lot of people, and thus it's likely that they'll stick with it.

• Almost incidentally, reasonably capable of defending oneself in one of the few instances where civilized behavior temporarily breaks down (An argument with a seemingly reasonable person who quickly becomes unreasonable, perhaps alcohol is involved? I don't know. Fights are low-stakes and uncommon these days but they still happen). This is kind of a weird edge case in a modern society but might non-trivially prevent injury or gain you status when it comes up.

Note that there are a lot of vectors by which one can gain these meta-skills. While there are a bunch of martial arts enthusiasts out there who would probably claim that martial arts have the exclusive ability to grant you one or more of these, I really doubt that's the case. However, martial arts get a pretty good amount of coverage in real and fictional cultural reference frames that we can be reasonably confident most people are familiar with, and it's not a bad example of a holistic activity that can hone a lot of these meta-skills.

It's also worth noting that while the skills involved in interacting with a society of people you trust and want to work with are often different from the skills involved in becoming a competent individual, many of the latter can be helpful in the former. I would much rather be on a team with a bunch of people who understand the meta-skill of staying calm under pressure, or the meta-skill of making their beliefs pay rent, than be on a team with a bunch of people who don't. Aggregated individual prowess isn't the only factor for group success, and it may not even be the most important one, but it certainly doesn't hurt.

For example, with subsidized prediction markets, we can each specialize on the topics where we contribute best, relying on market consensus on all other topics.

Why do these prediction markets have to be subsidized? In the U.S., online prediction markets are currently considered internet gambling and are hampered. Is there a reason legal, laissez-faire prediction markets couldn't take hold?

The martial arts metaphor for rationality training seems popular at this website, and most discussions here about how to believe the truth seem to assume an environmental worst case: how to figure out everything for yourself given fixed info ...

A very good point!

But I can't easily explain why it is a good point without violating the ban on mention of AI.

This observation doesn't invalidate Less Wrong. Someone still has to study these things. But the emphasis on individualism here can diminish awareness of the big picture.

I think it was just brainstorming based on Eliezer's post; he also wrote about the sanity water line, which I see your rational society approach fitting in with. Maybe a dojo is a bit extreme, but I think a zendo isn't implausible, with people working on rationality koans. Or maybe rationality group therapy, where people can express potential irrationality that they can receive non-judgemental feedback on. Grassroots bottom up approaches could work with larger top down approaches to create the rational society, or whatever word Yvain might find less taboo :)

If one goes off the notions of others without coming to conclusions for themselves they're just as blind as an evangelical christian. True insight can only come from within. That's why reason is of premium importance.

It is important to denote the difference between insight and belief, however; for insight is based off of rationality and logic whereas belief is based on primal emotions and instincts.

I should also add that I think group rationality techniques are important. We've already seen that being a good group rationalist means acting differently than just trying to be individually as accurate as possible. [in particular, you should not be swayed by what the rest of the group thinks].

Silly robot, Less Wrong is for kids!