What is Rationality?

post by Roko · 2010-04-01T20:14:09.309Z · LW · GW · Legacy · 22 comments

Contents

22 comments

This article is an attempt to summarize basic material, and thus probably won't have anything new for the experienced crowd.

Related: 11 Core Rationalist Skills, What is Bayesianism?

Less Wrong is a blog devoted to refining the art of human rationality, but what is rationality? Rationality is unlike any subject I studied at school or university, and it is probably the case that the synthesis of subjects and ideas here on Less Wrong is fairly unique. 

Fundamentally, rationality is the study of general methods for good decision-making, especially where the decision is hard to get right. When an individual is considering whether to get a cryonics policy, or when a country is trying to work out what to do about global warming, one is within the realm of decision-making that we can use rationality to improve. People do badly on hard decision problems for a variety of reasons, including: that they are not born with the ability to deal with the scientific knowledge and complex systems that our modern world runs on, that they haven't been warned that they should think critically about their own reasoning, that they belong to groups that collectively hold faulty beliefs, and that their emotions and biases skew their reasoning process.

Another central theme of rationality is truth-seeking. Truth-seeking is often used as an aid to decision-making: if you're trying to decide whether to get a cryonics policy, you might want to find out whether the technology has any good evidence suggesting that it might work. We can make good decisions by getting an accurate estimate of the relevant facts and parameters, and then choosing the best option according to our understanding of things; if our understanding is more accurate, this will tend to work better.

Often, the processes of truth-seeking and decision-making, both on the individual level and the group level are subject to biases: systematic failures to get to the truth or to make good decisions. Biases in individual humans are an extremely serious problem - most people make important life-decisions without even realizing the extent and severity of the cognitive biases they were born with. Therefore rational thought requires a good deal of critical thinking - analyzing and reflecting on your own thought processes in order to iron out the many flaws they contain. Group dynamics can introduce mechanisms of irrationality above and beyond the individual biases and failings of members of the group, and often good decision-making in groups is most severely hampered by flawed social epistemology. An acute example of this phenomenon is The Pope telling HIV infested Africa to stop using condoms; a social phenomenon (religion) was responsible for a failure to make good decisions.

Perhaps the best way to understand rationality is to see some techniques that are used, and some examples of its use.

Rationality techniques and topics include:

 

22 comments

Comments sorted by top scores.

comment by timtyler · 2010-04-01T21:58:15.777Z · LW(p) · GW(p)

Re: Rationality is the ability to do well on hard decision problems.

That sounds like the definition of intelligence - though it skips the "range of problems" bit. The "range of problems" qualification seems to be doing useful and desirable work there - though do we really want two separate terms meaning practically the same thing?

Replies from: Roko
comment by Roko · 2010-04-01T23:02:00.115Z · LW(p) · GW(p)

Certainly rationality as defined here is within the fuzzy cloud of groundings of the rather vague word "intelligence". However, it is probably closer in meaning to "wisdom".

Rationality differs from intelligence as commonly used in that intelligence in humans is commonly judged on abstract problems in situations of certainty, such as IQ tests, and frequently involves comparative assesments (IQ, exams) under time pressure and with tuition and preparation. Rationality typically deals with real-world problems with all the open-endedness that entails, in situations of uncertainty and without tuition or practice.

Replies from: timtyler, steven0461
comment by timtyler · 2010-04-02T10:08:30.256Z · LW(p) · GW(p)

My take on this issue is as follows:

There are a whole bunch of mental faculties - skill at which scientists have demonstrated is represented pretty well by a single quantity, g. That result leaves remarkably little space for a different, but nearby concept of instrumental rationality.

So: terminology in this area appears to be in quite a mess - with two historically well-known terms jostling together in a space where scientists are telling us there is only really one thing. Maybe there are three jostling terms - if you include "reason".

So: we need some philosophers of science to wade in and propose some resolutions to this mess.

Replies from: Wei_Dai, Roko, timtyler, timtyler
comment by Wei Dai (Wei_Dai) · 2010-04-02T10:39:16.702Z · LW(p) · GW(p)

There are a whole bunch of mental faculties - skill at which scientists have demonstrated is represented pretty well by a single quantity, g. That result leaves remarkably little space for a different, but nearby concept of instrumental rationality.

There is a recent book, What Intelligence Tests Miss: The Psychology of Rational Thought, by Keith Stanovich which argues that the g measured by IQ tests actually leaves out many other cognitive skills that are necessary for epistemic and instrumental rationality.

So: we need some philosophers of science to wade in and propose some resolutions to this mess.

He proposes to use "intelligence" to refer to g, and "rationality" for epistemic and instrumental rationality, but since our community is perhaps more closely linked to the field of AI than to psychology, I don't know if we want to follow that advice.

Replies from: timtyler, Roko
comment by timtyler · 2010-04-02T10:46:48.427Z · LW(p) · GW(p)

The certainly looks like a relevant book. I didn't like some of Keith Stanovich's earlier offerings much, and so probably won't get on with it, though :-(

Reading summaries makes me wonder whether Stanovich have any actual evidence of important general intellectual capabilities that are not strongly correlated with Spearman's g.

It is easy to bitch about IQ tests missing things. The philosophy behind IQ tests is that most significant mental abilities are strongly correlated - so you don't have to measure everything. Instead people deliberately measure using a subset of tests that are "g-loaded" - to maximise signal and minimise noise. E.g. see:

http://en.wikipedia.org/wiki/General_intelligence_factor#Mental_testing_and_g

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2010-04-02T11:03:08.878Z · LW(p) · GW(p)

He cites a large number of studies that show low or no correlation between IQ and various cognitive biases (the book has very extensive footnotes and bibliographies), but I haven't looked into the studies themselves to check their quality.

Replies from: timtyler
comment by timtyler · 2010-04-02T11:33:16.217Z · LW(p) · GW(p)

Right - well, there are plenty of individual skills which are poorly correlated with g.

If you selected a whole bunch of tests that are not g-loaded, you would have similar results.

What you would normally want to do is to see what they have in common (r) and then see how much variation in common cognitive functioning is explained by r.

The classical expectation would be: not very much: other general factors are normally thought to be of low significance - relative to g.

The other thing to mention is that many so-called "cognitive biases" are actually adaptive. Groupthink, the planning fallacy, restraint bias, optimism bias, etc. One would expect that many of these would - if anything - be negatively correlated with other measures of ability.

comment by Roko · 2010-04-02T15:43:17.447Z · LW(p) · GW(p)

Thanks, Wei, that's very useful.

comment by Roko · 2010-04-02T15:42:53.338Z · LW(p) · GW(p)

There are a whole bunch of mental faculties - skill at which scientists have demonstrated is represented pretty well by a single quantity, g. That result leaves remarkably little space for a different, but nearby concept of instrumental rationality.

What is the correlation between income and IQ? Wikipedia says:

Taking the above two principles together, very high IQ produces very high job performance, but no greater income than slightly high IQ.

IQ scores account for about one-fourth of the social status variance and one-sixth of the income variance. Statistical controls for parental SES eliminate about a quarter of this predictive power.

To me, this indicates that there is something other than IQ (==g) that governs real-world performance.

Replies from: timtyler
comment by timtyler · 2010-04-02T17:33:56.313Z · LW(p) · GW(p)

The claim for g is that it is by far the best single intellectual performance indicator. That is not the same as the idea that it accounts for most of the variation. There could be lots of variation that is governed by many small factors - each of low significance.

From the cited page:

"Arthur Jensen claims that although the correlation between IQ and income averages a moderate 0.4" ...and... "Daniel Seligman cites an IQ income correlation of 0.5"

I am not entirely sure about the implicit idea that people's aim in life is to make lots of money either. Surely it is more reasonable to expect intelligent people - especially women - to trade off making income against making babies - in which case, this metric is not a very fair way of measuring their abilities, because it poorly represents their goals.

comment by timtyler · 2010-04-02T10:35:34.693Z · LW(p) · GW(p)

One option is to ditch "instrumental rationality" as a useless and redundant term - leaving "rationality" meaning just "epistemic rationality".

Another observation from computer-science is that we have the separate conceptions of memory and CPU. Though there isn't much of a hardware split in brains, we can still employ the functional split - and discuss and test memory and processing capabilities (somewhat) separately. Other computer-science-inspired attributes of intelligent agents are serial speed and degree of parallelism. If we are attempting to subdivide intelligence, perhaps these are promising lines along which to do it.

Replies from: torekp
comment by torekp · 2010-04-04T23:09:16.790Z · LW(p) · GW(p)

I think I can see why instrumental rationality could be regarded as just part and parcel of epistemic rationality. Once the probabilities have been rationally evaluated, what work is left for "instrumental reason" to do? Am I on the right track at all? If not, please elaborate.

comment by timtyler · 2010-04-02T10:33:54.355Z · LW(p) · GW(p)

One option is to ditch "instrumental rationality" as a useless and redundant term - leaving "rationality" meaning just "epistemic rationality".

comment by steven0461 · 2010-04-01T23:30:47.589Z · LW(p) · GW(p)

I see the distinction between intelligence and rationality as assuming a model of an agent with a part that generates logical information and a part that uses the logical information to arrive at beliefs and decisions, with "intelligence" defined as the quality of the former part and "rationality" defined as the quality of the latter part. In the latter case "quality" turns out to mean something like "closeness to expected utility maximization and probability theory".

Replies from: timtyler
comment by timtyler · 2010-04-02T10:12:05.729Z · LW(p) · GW(p)

That makes intelligence an internal sub-module, that is some distance from actions, and so can't directly be measured by tests. That is not what most scientists use the term to mean, I believe.

comment by timtyler · 2010-04-01T21:49:27.135Z · LW(p) · GW(p)
  • "rationality is the study of general methods for good decision-making"

  • "Rationality is the ability to do well on hard decision problems."

  • "Rationality is also the art of how to systematically come to know what is true."

It seems like three separate meanings! Is such overloading of meanings desirable?

Replies from: Strange7
comment by Strange7 · 2010-04-01T22:03:18.922Z · LW(p) · GW(p)

Iaijutsu includes:

  • how to not die when angry people come at you with sharp things unexpectedly

  • how to bring your sword out of it's sheath quickly

  • how to cut stuff

That might seem like three separate skills, but they're interrelated enough to be taught as a single school.

Replies from: timtyler
comment by timtyler · 2010-04-01T22:44:47.260Z · LW(p) · GW(p)

Wikipedia doesn't really seem to agree with that:

http://en.wikipedia.org/wiki/Iaijutsu

It basically defines Iaijutsu as the art of drawing your sword.

That seems like one meaning, not three.

Chemistry includes the use of bunsen burners, test tubes, and acids. However, that is not the definition of chemistry.

So: I am not convinced you have got this example straight.

comment by timtyler · 2010-04-01T21:46:24.701Z · LW(p) · GW(p)

Re: Fundamentally, rationality is the study of general methods for good decision-making [...]

It's about studying decision making - rather than being about actually making decisions?

comment by Piercello · 2010-04-03T20:20:12.611Z · LW(p) · GW(p)

I like the following pair of concise definitions:

Logic = internally consistent structure;

Reason = objectively verifiable logic.

So rationality is a subset of operations taking place within the (assumed-to-exist) structural logic of cognition/intelligence, which itself is subject to rational investigation.

An overlapping subset is the subjective emotional system, which presumably has its own logical architecture. My own interest is centered on possible ways to logically relate/integrate these two systems.

comment by RobinZ · 2010-04-01T22:48:38.217Z · LW(p) · GW(p)

I think it is worth noting some of the posts criticizing game theory on this blog - particularly Newcomb's Problem and Regret of Rationality.