What is Rationality?

post by Roko · 2010-04-01T20:14:09.309Z · LW · GW · Legacy · 22 comments

Contents

22 comments

This article is an attempt to summarize basic material, and thus probably won't have anything new for the experienced crowd.

Related: 11 Core Rationalist Skills, What is Bayesianism?

Less Wrong is a blog devoted to refining the art of human rationality, but what is rationality? Rationality is unlike any subject I studied at school or university, and it is probably the case that the synthesis of subjects and ideas here on Less Wrong is fairly unique. 

Fundamentally, rationality is the study of general methods for good decision-making, especially where the decision is hard to get right. When an individual is considering whether to get a cryonics policy, or when a country is trying to work out what to do about global warming, one is within the realm of decision-making that we can use rationality to improve. People do badly on hard decision problems for a variety of reasons, including: that they are not born with the ability to deal with the scientific knowledge and complex systems that our modern world runs on, that they haven't been warned that they should think critically about their own reasoning, that they belong to groups that collectively hold faulty beliefs, and that their emotions and biases skew their reasoning process.

• Rationality is the ability to do well on hard decision problems.

Another central theme of rationality is truth-seeking. Truth-seeking is often used as an aid to decision-making: if you're trying to decide whether to get a cryonics policy, you might want to find out whether the technology has any good evidence suggesting that it might work. We can make good decisions by getting an accurate estimate of the relevant facts and parameters, and then choosing the best option according to our understanding of things; if our understanding is more accurate, this will tend to work better.

• Rationality is also the art of how to systematically come to know what is true.

Often, the processes of truth-seeking and decision-making, both on the individual level and the group level are subject to biases: systematic failures to get to the truth or to make good decisions. Biases in individual humans are an extremely serious problem - most people make important life-decisions without even realizing the extent and severity of the cognitive biases they were born with. Therefore rational thought requires a good deal of critical thinking - analyzing and reflecting on your own thought processes in order to iron out the many flaws they contain. Group dynamics can introduce mechanisms of irrationality above and beyond the individual biases and failings of members of the group, and often good decision-making in groups is most severely hampered by flawed social epistemology. An acute example of this phenomenon is The Pope telling HIV infested Africa to stop using condoms; a social phenomenon (religion) was responsible for a failure to make good decisions.

Perhaps the best way to understand rationality is to see some techniques that are used, and some examples of its use.

Rationality techniques and topics include:

• Following through with simple logical inferences and numerical calculations - A surprising number of bad decisions and conclusions can be avoided by doing relatively simple pieces of logical reasoning without error or flinching in the face of the conclusion. Common general examples include non sequiturs such as affirming the consequent, argument from fallacy, and taking absence of evidence as certitude of absence ("I haven't found any evidence for it therefore it can never happen" type reasoning). Many bad decisions also result from not doing simple arithmetic, or not taking into account quantitative reasoning. See this website on environmentalism gone wrong due to a failure to reason quantitatively.

• Heuristics and biases - Perhaps the key insight that started the Overcoming Bias and Less Wrong blogs was the mounting case from experimental psychologists that real human decision-making and Belief formation is far from the ideals of economic rationality and Bayesian probability. A key reference on the subject is Judgment under Uncertainty: Heuristics and Biases. For those who prefer the web, Less Wrong has a set of articles tagged "standard biases". If you know your own flaws, you may be able to correct for them - this is known as debiasing.

• Evolutionary psychology and Evolutionary theory - In his bestselling book Fooled by Randomness, Nassim Taleb writes "Our minds are not quite designed to understand how the world works, but, rather, to get out of trouble rapidly and have progeny". Understanding that the process that produced you cared only about inclusive genetic fitness in the ancestral environment, rather than your welfare or ability to believe the truth can help to identify and iron out flaws in your decision-making. There is a good sequence on evolution on Less Wrong. Perhaps the most important piece of work on the implications of evolutionary theory for decision-making and rationality is Bostrom and Sandberg's Wisdom of Nature; although it purportedly aims at assessing human enhancement options, the style of reasoning is highly applicable to thinking about how to deal with the mixed blessings that evolution put inside our skulls.

• Defeating motivated cognition - Many specific instances and types of biased reasoning are probably created by the same set of sources, often processes deeply intertwined with our evolved psychology. The most pernicious of these "sources of biased reasoning" is motivated cognition, the king of biases. The human mind seems to have a way of short-circuiting itself whereby happy emotions come when you visualize an outcome that is good for you, and this causes you to search for arguments that support the conclusion that that good outcome will occur. This kind of "bottom line reasoning" is insidious, and decreasing the extent to which you suffer from it is a key way to increase your rationality. Leaving a line of retreat is one good antidote. There is a whole sequence on how to actually change your mind that attempts to beat this problem.

• Techniques of analytic philosophy - Analytic philosophers have spent a long time honing techniques to promote better thinking, especially about conceptually confusing subjects. They will often be very careful to explicitly define key terms they use, and be open and upfront about terms that they take as primitive, as well as being clear about the structure of their arguments.

• Bayesian statistics and the Bayesian mindset - Covered expertly in the article "What is Bayesianism?" - briefly, the idea is that the beliefs of an ideal rational agent are formed by a process of formulating hypotheses, assigning prior credence to each, and then using Bayes' theorem to work backwards from the data to work out how likely various hypotheses are, given the data. In cases where there is "overwhelming evidence", the strictures of Bayes' theorem are unnecessary: it will be obvious which hypothesis is true. For example, you do not need Bayes' theorem to deduce that Gary Kasparov would beat your grandmother at chess. Related to this are various errors and lies that can arise from bad (or deliberately misleading) statistical analyses.

• Microeconomic ways of thinking - Microeconomics models rational agents as aiming to make good personal choices subject to resource constraints. Von Neumann and Morgenstern proved an important theorem stating that the preferences of a "rational" agent can be expressed as a utility function. Other researchers in microeconomics made significant advances by considering the marginal utility of actions - how much better do things get if one shifts one dollar of one's expenditure from buying ice-cream to buying clothes? The notion of opportunity cost is a classic example of a microeconomic concept. Value of information is another. In recent times, microeconomics has taken human psychology into account more, leading to formal theories of boundedly rational and irrational agents, such as prospect theory.

• Game theory and signaling games - a sub-field of microeconomics so important that it deserves a separate mention, game theory analyzes the interactions between competing rational agents in a formal way. The key intuition pump is the prisoner's dilemma, but I think that the the formal analysis of signaling games is even more important for rationality, as signaling games explain so much about why people verbally endorse statements (the statement is there as a signal, not as an indicator of rational belief). Robin Hanson of Overcoming Bias has posted many times on how the subconscious human desire to signal affects our decision-making in weird ways.

• Creating good social epistemology and norms of rationality - several sequences on Less Wrong are about how to create an atmosphere that encourages honest and productive social epistemology. Resisting groupthink and cultishness is important step, as is dealing with the problem that politics tends to make humans stupid; this is covered in the "politics is the mind killer" sequence. Finally there is a sequence on creating good rationalist communities.

 

22 comments

Comments sorted by top scores.