Something to Protect

What was it? AI?

I get an uncomfortable feeling, Eliezer, that this work is to ultimately lead to a mechanism to attract:

• people of libertarian bent

• people interested in practically unbounded longevity of consistent, continual consciousness

and also lead to a mechanism to tar people disinclined to those two goals; tar them with the label "sentimentally irrational".

Rationality to me is simply a tool. I would have absolutely no confidence in it without the ongoing experiences of applying it iteratively, successfully to specific goals.

And of course, no matter how much you profess your love of mere usefulness, you should never actually end up deliberately believing a useful false statement.

I haven't yet needed to "deliberately believe a useful false statement" (to my knowledge), but I wouldn't be particularly disturbed if I tried to, and found it repeatedly successful. Another tool for my tool belt.

Right now I am having some success with modeling the world over the conditions I care about with:

• scientific laws (including information theory)

• mathematics

• groups of causality graphs, for the same phenomena, in competition

• specific causality graphs

• naive Bayesian

• straightforward use of Bayes' theorem

• frequentist probability and statistics

• discrete probability

• logic

(causality graphs considered can include relations defined by simulation, and all other tools listed. Whatever it is, shove it into a causality graph. I haven't found it useful to restrict the use of anything in a causality graph, particularly if they are forced to compete over the ability to be consistent with past data and predict future results.)

(The list above is somewhat ordered over more applicable to specific situations, to less applicable to specific situations. I attach the lowest confidence to any specific causality graph, more confidence with the graphs in aggregate in competition. I attach more confidence in frequentist analysis over good data, over Bayesian, but Bayesian is applicable in more circumstances.)

I have to deal with finite resource allocation in a manufacturing plant. Where else to use these tools? Possibly an the opportunity from celebrating the differences in all the people working in the plant.

I am often confused by your writing, because I don't see where you have "skin in the game". Where are you exercising your tools of rationality?

Is it all just to make the world slightly more hospitable to libertarians interested in life extension? (No negative judgment if that is the case.)

(Sorry to beg your indulgence of a long post)

The success of science was and is because it is useful, and similarly for rationalism. But one of the critiques of rationalism and of the overcoming-bias program is that it is sometimes counterproductive. The unbiased tend to be unhappy and/or insane. If someone's goals are to be happy and successful in life, he does best not to be fully rational. Irrationality is the most useful policy if these are your goals.

Your argument suggests that this is true only because this is setting the goalposts too low. For someone who merely seeks happiness, yes, irrationality is in order. But if someone's goals are much higher - if lives are at stake, perhaps even the lives of all humanity, then irrationality no longer works best. In that case, he must follow a path of strict rationality as closely as possible, because the stakes are so high.

However it could be argued that this is not always the case, that high stakes may nevertheless require a degree of irrationality. Rationality is useful for getting at the truth; but irrationality may be useful in persuading and motivating others to help. Successful leaders are notoriously irrational, and if your project is big enough, leadership will be a necessary ingredient for success.

Perhaps a solution is to split one's efforts into two pieces: a rational part, which ruthlessly seeks the truth regardless of inhibitions and discomfort; and an irrational part, which takes the core results from rational analysis, dresses them up in attractive lies, and sells them enthusiastically to the larger world. In fact I would suggest that many successful enterprises have been built on a partnership with this structure: the creative genius who works behind the scenes, and the leader who is the public face of the endeavor and who excels at presentation. You might consider a similar arrangement for your own project.

I rarely post, only read in hopes of learning. Today, I comment: I appreciate the beauty of this post.

Thank you, Eliezer.

I am often confused by your writing, because I don't see where you have "skin in the game". Where are you exercising your tools of rationality?

If I'd went ahead and said that within the post, it would've transformed a piece on rationality into overt propaganda, destroying its internal aesthetics. Read my website.

I have touched before on the idea that a rationalist must have something they value more than "rationality"

What a terrible idea... then whenever rationality comes in conflict with that thing, rationality will be discarded.

We already see lots and lots of this behavior. It's the human norm, in fact: use rationality as a tool as long as it doesn't threaten X, then discard it when it becomes incompatible with X.

Perhaps I am one of the "sentimentally irrational," but I would pick the 400 certain lives saved if it were a one-time choice, and the 500 @ 90% if it were an iterated choice I had to make over, and over again. In the long run, probabilities would take hold, and many more people would be saved. But for a single instance of an event never to be repeated? I'd save the 400 for certain.

Your 80% and 90% figures don't really add up either. You don't describe how many people in total will die, regardless of you decision. If the max death number possible from this catastrophe is 500, then your point is valid. But what if it were 100 million, or even better, all of humanity? Now, the difference in chance of saving your loved one via either strategy is vanishingly small, and you are left with a 90% chance or a 100% chance of saving humanity as a whole. It's exactly the same as the situation you describe above, but it seems the moral math reverses itself. You need to more fully specify your hypothetical situations if you wish to make a convincing point.

Caledonian, I think you're misreading him. He's not saying: the cause is the one thing you never think rationally about. He's saying: the cause is good (rationally good) and to protect/preserve it you have to pull yourself into conformance with the real world, because that's where the action is. To achieve that you have to hold up what you (perhaps mistakenly) think of as "reason" against the real world, and be prepared to re-evaluate if it doesn't work. What your re-evaluation seeks is better techniques of reason - not to throw reason away.

Caledonian, I think you're misreading him. He's not saying: the cause is the one thing you never think rationally about. He's saying: the cause is good (rationally good) and to protect/preserve it you have to pull yourself into conformance with the real world, because that's where the action is.

I think you're misreading him, substituting a reasonable argument for the rather bizarre things he says.

Rationality by its nature cannot be only a means towards an end.

"Rationality by its nature cannot be only a means towards an end."

Rationality is conformance to reality. You can conform to reality for a cause. (You're saying, you can't mold reality to your cause - I agree, but that's not what he was meaning.) He was meaning that people have thought themselves rational when applying formal, skillful, pedigreed academic techniques that DON'T WORK, such as Jesuit style casuistry. So you have to hold the technique up against reality. You won't do that if you put the technique first by saying "I serve reason", because that morphs in your mind into "I serve Jesuit casuistry" or whatever. It blithely assumes your all-too-human technology of achieving reason works - and it might not.

Julian, Caledonian is a well-known troll on OB. We've decided against censorship for now, but you might not want to waste too much time. I generally don't respond to Caledonian unless I see someone else agreeing with him.

I totally agree with "Anon", and others who made similar points in the Circular Altruism post. Context matters! Is it a one-time choice, or an iterated choice? Is there an upper limit to the number of deaths, or no limit? Are the 500 the number of people on the sinking ship/last people on planet earth, or possible victims from a much larger pool? You can only do the math and make a rational decision when you have ALL the numbers from the relevant context.

The first steps of rationality lie not in separating problems from their context, but in determining what context is relevant.

You won't do that if you put the technique first by saying "I serve reason", because that morphs in your mind into "I serve Jesuit casuistry" or whatever.

I agree with this point, but that's exactly what Eliezer stated he was in favor of: serving something else and merely using rationality as a means toward that end while it's convenient to do so.

It doesn't do any good to avoid making an implicit error by explicitly making that error instead. Certainly we need to compare our thinking to a fundamental basis, but the goal we're seeking can't be that basis. Rationality is about always checking our thinking against reality directly, and using that to evaluate not only our methods of reaching our goals but the nature of our goals themselves.

If you adopt rationality merely because you want to use it to attain your ends, what happens if you discover that your ends aren't compatible with it? (And if that's really what you're doing, how did you know to adopt rationality in the first place? Just keep trying random stuff until you happen to stumble into the correct meta-strategy by chance? I think rationality has to be the starting point, not something picked up along the way.)

I don't have anything desperately important to me, and you say I'm not allowed to just pick something. Given this, what am I supposed to do, to become more rational? Am I just doomed? I really desperately want to believe true things and not false things, but you say that's not good enough.

Good question, Nominull. Unfortunately I lack the ability to answer your question from personal experience. Mine just fell into my lap.

But is believing true things what you most desperately want, in all the world?

Caledonian, I gather Eliezer put "rationality" in quotes because people may believe they are committed to rationality when in fact they are not. If they have a goal which is contingent on rationality that will help them from straying from the path.

What he said immediately after the part you mention was: "The Art must have a purpose other than itself, or it collapses into infinite recursion"

He wasn't talking about pseudo-rationality. When he talks about "The Art", he's talking about rationality.

And he's wrong: truth points to itself.

Anon: do you suggest that others follow your policy as well? Then when many people have individual made isolated choices like that, far fewer lives will have been saved. And in the whole history of the world, choices like that must have been made many times. Why does it matter whether it is you who are repeating the choice or other people?

The question about whether the 500 are that last people in the world is adding other utilities into the issue, such as preserving the human race, and so on. In that case you have a different comparison; naturally, you may have to consider other factors besides the utility of the lives. But as long as you consider only the lives, Eliezer is right.

Caledonian: "I think rationality has to be the starting point."

Can you expand on this? A rationalistic moral relativist might say that actions require goals, ultimate goals are arbitrary, and so rationality cannot be the starting point there. In the real world, by the time one is able to entertain ideas like 'choosing to be more rational', you're already going to have goals, preferences, ideas about how you should live your life. So it could be countered that 'rationality' never has to supply everything; its purpose will largely be to critique existing purposes, order them by significance, or evaluate new possibilities. Say something more about what you think the role of rationality should be in developing a morality, and about the particular powers it has to fulfil that role.

Anon, Wendy:

Certainly finding out all of the facts that you can is good. But rationality has to work no matter how many facts you have. If the only thing you know is that you have two options:

1. Save 400 lives, with certainty
2. Save 500 lives, 90% probability; save no lives, 10% probability. then you should take option 2. Yes, more information might change your choice. Obviously. And not interesting. The point is that given this information, rationality picks choice 2.

1. Save 400 lives, with certainty
2. Save 500 lives, 90% probability; save no lives, 10% probability.

i.e.

1. Save 4 lives, with certainty
2. Save 5 billion lives,0.00000009% probability; save no lives, 99.99999991% probability.

Any takes for #2? I seem to remember Ben Jones saying he would choose #1 in a case similar to the second case.

Formerly, I think I would have chosen #2 in the first case and #1 in the second. But Eliezer has converted me. Now I choose #2 in both cases. But would he do that himself? Consider:

"Perhaps I am one of the 'sentimentally irrational,' but I would pick the 400 certain lives saved if it were a one-time choice, and the 500 @ 90% if it were an iterated choice I had to make over, and over again. In the long run, probabilities would take hold, and many more people would be saved. But for a single instance of an event never to be repeated? I'd save the 400 for certain." (Anon, above)

"If the probabilities of various scenarios considered did not exactly cancel out, the AI's action in the case of Pascal's Mugging would be overwhelmingly dominated by whatever tiny differentials existed in the various tiny probabilities under which 3^^^^3 units of expected utility were actually at stake.

You or I would probably wave off the whole matter with a laugh, planning according to the dominant mainline probability: Pascal's Mugger is just a philosopher out for a fast buck.

But a silicon chip does not look over the code fed to it, assess it for reasonableness, and correct it if not. An AI is not given its code like a human servant given instructions. An AI is its code. What if a philosopher tries Pascal's Mugging on the AI for a joke, and the tiny probabilities of 3^^^^3 lives being at stake, override everything else in the AI's calculations? What is the mere Earth at stake, compared to a tiny probability of 3^^^^3 lives?

How do I know to be worried by this line of reasoning? How do I know to rationalize reasons a Bayesian shouldn't work that way?" (Eliezer Yudkowsky, Pascal's Mugging)

Who sees the similarity? Eliezer no doubt thinks that Anon is biased toward certainty, but so is he: he simply has less of the bias.

So I hereby retract my argument against voting, Pascal's Mugging, and Pascal's Wager. In the particular Mugging we discussed, there may have been anthropic reasons to make it proportionally improbable. But without such reasons, it should be accepted.

"It takes visceral panic, channeled through cold calculation, to cut away all the distractions." - this just made it to my quotes file.

If i understand Eliezer's point correctly in terms of the map/territory analogy, what he says is that having somewhere to go and actually needing to put your map to use will motivate you to make that map as accurate as possible, if you care about your destination more than you 'believe in' the current iteration of your map and/or the techniques used to derrive it.

A rationalistic moral relativist might say that actions require goals, ultimate goals are arbitrary, and so rationality cannot be the starting point there.

Lots of things act without having any sort of goals. Does fire have a goal of reducing high-energy compounds into oxidized components and free energy? No, but it does it anyway.

You can limit 'action' to intentional events only, I suppose.

However, how does declaring that goals are arbitrary rule out assertions about necessary starting points?

So it could be countered that 'rationality' never has to supply everything; its purpose will largely be to critique existing purposes, order them by significance, or evaluate new possibilities.

If the goals already developed are incompatible with each other, rationality isn't going to help much. If they're incompatible with rationality, it really isn't going to help. But no helping is possible.

Say something more about what you think the role of rationality should be in developing a morality, and about the particular powers it has to fulfil that role.

Rationality is required to form a coherent model (however incomplete or imperfect) of the world. To take an action with the intention of bringing about a specific result requires a coherent model. Ergo...

An incoherent actor can't be said to have any goals at all.

Formerly, I think I would have chosen #2 in the first case and #1 in the second. But Eliezer has converted me. Now I choose #2 in both cases. But would he do that himself?

Isn't that implicitly what he does for a living? Eliezer could become a firefighter or emergency medical technician, or work for clean drinking water in rural Africa, with a near-certainty of preventing several deaths in the next year. Meanwhile, there is a very small chance of someone creating an non-Friendly AI in the next year. We can argue about the probabilities (of the problem arising, of successfully presenting it), but Eliezer has already chosen the existential threat.

"So I hereby retract my argument against voting, Pascal's Mugging, and Pascal's Wager. In the particular Mugging we discussed, there may have been anthropic reasons to make it proportionally improbable. But without such reasons, it should be accepted."

I'm certainly glad you think so, Unknown, because I was just contacted by the Dark Lords of the Matrix. It turns out that we are living in a simulation. I have no idea what the physics of the world outside are like, but they're claiming that unless you personally send $100 to SIAI right now, they're going to put one dust speck in the eye of each of BusyBeaver(BusyBeaver(BusyBeaver(3^^^^^^^^^^^^^^^^^^^3))!!)! people.

Get out your checkbook, quickly, before it's too late!

(same anon from above who asked about the context of the 400/500 problem being an issue)

In response to GreedyAlgorithm who said:

Certainly finding out all of the facts that you can is good. But rationality has to work no matter how many facts you have. If the only thing you know is that you have two options:

1. Save 400 lives, with certainty
2. Save 500 lives, 90% probability; save no lives, 10% probability. then you should take option 2. Yes, more information might change your choice. Obviously. And not interesting. The point is that given this information, rationality picks choice 2.

While I agree with your constrained view of the problem and its analysis, you are trying to have your cake and eat it too. In such a freed-from-context view, this is (to use your own words) "not interesting". It's like asserting that "4.5 is greater than 4" and that since we wish to pick the greater number, the rationalist picks 4.5. True as far as it goes, but trivial and of no consequence.

Eliezer brought in the idea of something more valuable than your own life, say that of your child. By stepping outside the cold, hard calculus of mere arithmetic comparisons he made a good point (we are still discussing it), but he opened the door for me to do the same. I see your child, and raise you "all of humanity".

Either we are discussing a tautological, uninteresting, degenerate case which reduces down to "4.5 is greater than 4, so to be rational you should always pick 4.5" (which, I agree with, but is rather pointless) or we are discussing the more interesting question of the intersection between morality and rationality. In that case, I assert bringing "extra" conditions into the problem matters very much.

If "rationality has to work no matter how many facts you have" [Greedy's words] (which I agree with) then you must grant me that it should provide consistent results. To make the problem "interesting" Eliezer brought in the "extra" personal stake of a loved family member, and came to his rationalist conclusion, pointing out why you'd want to "take a chance" given that you don't know if your daughter is in the certain group or might be saved as one of the "chance" group. I merely followed his example. His daughter may still be in the certain group or not (same situation) but I've just added everyone else's daughter into the pot. I don't see how these are fundamentally different cases, so rationality should produce the same answer, no?

Z. M. Davis, given the existence of that many people, and given that threat, the probability that I personally would be the one threatened in that must be multiplied by one over the number of people, since it could have been anyone else. So the expected disutility from your mugging is one dust speck multiplied by the probability that the Matrix scenario is actually true. This probability is very low, and even if it were unity, the disutility of one dust speck isn't going to get me to pay $100.

So again, I said "without such reasons, it should be accepted." But I have the reasons in this case.

Zubon, of course, Eliezer might well take my second gamble. I was only speaking in preparation for what came ahead: Anon doesn't want to gamble with human lives if there's a small chance of failure. Eliezer may be willing to do this, but once the chance of success becomes extremely small (much smaller than 1 in a billion), as in the Mugging case, he ignores the expected utility, thus falling into exactly the same sort of irrationality as Anon. In relation to this, it is significant that he admitted that he would reject the Mugging even if he had no reason to think that expected utility of rejecting it was greater than the expected utility of accepting it.

Or in other words: expected utility must equal utility times probability, no matter how small the probability is.

It's probably just that I'm stupid, but I don't understand the anthropic solution to Pascal's Mugging. Why does it matter that other people could have been asked? What if it were stipulated that the mugger threatens everyone?

Maybe I should actually study Kolmogorov complexity before trying to grapple with such matters.

Viz. the dilemma posed in Circular Altruism, what should we do? When forced to "Shup up and multiply", we have forgone our intuitions and picked the choice based upon our mathematics. However, we are not just overcoming our own intuitions, but also the intuitions of everyone who does not simply "Shut up and multiply". We are held accountable not by those who knows the math, but by those who have intuitions like ourselves.

If we save everyone, we are heroes. If we do not, we are held accountable not for the math, but for the very intuition we had to overcome multiplied by as many people who are now trying to hold us accountable.

So at the very least, we should add ourselves as the 501st person who might die in the second case. The price we pay for the burden of rationality!

RS, if that really bothers you, you haven't found your something to protect yet.

So, is your point that we need a cause against which to evaluate the success of our mathematics? That perhaps this sort of feedback that, persumably, you encounter on a daily basis, is something that does not come through rationality itself, but through the very real feedback of what you have chosen to protect?

I guess my previous post was a reflection that I am just a budding rationalist, and also that my skills have not been sharpened against the proper stone.

So, is your point that we need a cause against which to evaluate the success of our mathematics? That perhaps this sort of feedback that, persumably, you encounter on a daily basis

I'm not going to get feedback on my final success or failure for, oh, probably at least another 10 years.

My point, rather, was that your post illustrated very clearly why rationality comes from having something to protect - you thought of doing something rational, but worried about the other people whose intuitions differed from yours, and what they might think of you. So that worry is a force binding you to the old way of thinking.

But if the thing you were protecting was far more important than what anyone thought of you, that wouldn't slow you down. This isn't about iconoclasm - it's about an inertial drag exerted by all the little fears and worries, an inertial drag of the way that you or others previously did things; the motivating force has to be more powerful than that, or you won't move.

"The point is that given this information, rationality picks choice 2." - Posted by: GreedyAlgorithm

Sorry, no. Given this information, rationality says that there is not enough information to make an appropriate decision, and demands to know the context. If contextual information isn't available, rationality will say that either option 1 or 2 may be right, depending on circumstances.

Rationality never dismisses context as irrelevant just because it isn't known. If unknown factors make the right answer uncertain, then you must accept that it is uncertain.

Context can change what you're trying to achieve. Many people seem to assume that the point (re Circular Altruism problems) is to save as many lives as possible, but this might have to be balanced with other goals - e.g. setting a limit to acceptable risk (as in not risking destruction of the entire human population, whatever their number), or spreading risk instead of marking certain people for death (as in putting the last few people from a sinking ship in the lifeboat, not leaving them behind to make a crowded lifeboat safer).

Making assumptions is one of the danger pitfalls for rational thinkers. So is a reluctance to say "I don't know the answer" when appropriate.

For some reason this post reminds me of the Buddhist parable asceticsim now, nymphs later.

I don't think it's all that uncommon to begin cultivating an art for some specific purpose, proceed to cultivate it largely for its own sake, and eventually to abandon the original purpose.

Under Multiple Worlds, aren't you condemned, whatever you do or don't do, to there being a number tending to infinity of worlds where what you want to protect is protected, and a number tending to infinity where it is not ?

Caledonian: Let's distinguish between the aesthetics of rationality and the pragmatics of rationality. Is my model of the world consistent, do my goals make sense - that's pragmatics. Aesthetics is by comparison nebulous and subtle, but perhaps it encompasses both admiration for the lawlike nature of reality and self-admiration for one's own relationship to it. :-)

It seems to me that you are taking issue with the idea that the pragmatics of rationality should be trumped by a higher cause. This essay says nothing about that. It says, first, that it's a psychological fact that people don't adopt rationality as a conscious value until some other, already existing value is threatened by irrationality, and second, that you won't keep developing as a rationalist without such pressure.

As for whether reason by itself can supply supreme values, I had to ask because so many people do think you can get an ought from an is. (I still don't know what you meant by "truth points to itself".)

You are not alone, Z. M. Davis: I disagree with Eliezer over whether Robin's anthropic solution is a satisfactory solution to Pascal's Mugging. (Eliezer repeated his endorsement of Robin's anthropic solution here a few weeks ago.) Since I started reading Eliezer 6 years ago, this is the first time I can recall disagreeing with him on a question of fact. (As I have pointed out many times in the comments here, I disagree with him significantly on terminal values.) If anyone wants to reply to this, I humbly suggest doing so by clicking on my name below.

Let's distinguish between the aesthetics of rationality and the pragmatics of rationality.

Extraordinary - I don't believe I've ever heard anyone speak of the aesthetic aspects of rational thought before.

I'm not sure I agree with the concept, but it's something to think about.

It says, first, that it's a psychological fact that people don't adopt rationality as a conscious value until some other, already existing value is threatened by irrationality,

And when that value is threatened by the rationality? What then?

and second, that you won't keep developing as a rationalist without such pressure.

I suspect relatively few people have a deep desire for knowledge and understanding. They're usually the only ones I see developing as rationalists at all. If you don't have a need for the world to make sense, you tend to develop ad hoc methods for getting what you want. The need for systematic understanding is either present, or not.

For those saying they have nothing to protect or still need to find something to protect, remember that you are human and, unless you have no natural family or reproductive ties, you always have the people you love to protect. It may seem counterintuitive if you've bought into Hollywood rationality, but love is a powerful motivational force. If you think that, in theory, being more rational is good, but don't see how you can effect greater rationality in your mind, consider the many benefits of your increased rationality (again, not Hollywood rationality, but rationality of the type Eliezer describes above).

In my case, I know I'm trying harder than ever to become a better person because of my wife. And when I do something that hurts her, my first thought is to figure out what is wrong with my thinking that led to this. My second is to find a better way to express my love, through increasing her happiness and enjoyment of life. And, realizing that the best thing I can do is shut up an multiply, I figure out how to change myself to be a better multiplier.

Excellent point by Worley. Since I have assumed the role on this blog of pointing out that happiness is not the meaning of life, let me hasten to add that happiness is a very useful barometer. Whether you are happier on average now than you were 10 years ago is for example probably a more reliable barometer of whether your life is on a better track than it was 10 years ago than change in financial net worth over those 10 years (though net worth is an important barometer too). And the one situation in which happiness is least likely to steer you wrong is when you use your wife's happiness as a barometer for how good a job you are doing as a husband.

The object of the game of life is not just to become more rational but rather to become more rational, more ethical and more loving. "Being loving" is defined as helping those close to you to become more rational, ethical and loving. This is the way we maximize the ethics and the rationality of every intelligent agent in our reach, which, if it is not the purpose of life, is a sufficiently good approximation for most people. (Singularity scientists however will probably need a more sophisticated definition of the purpose of life.)

By "ethics" I mean simply the sincere desire to do good and to avoid doing evil. (I freely admit I do not have a formula or algorithm that allows a person to tell good from evil in any situation). I bring the concept of ethics into this little exposition because I want to suggest that it unethical to increase the rationality of an unethical person. By doing so, you are increasing his capacity to do evil. That suggestion goes against the egalitarianism that is such a central part of our ethical culture: the conventional ethical wisdom is that every human is equally deserving of loving treatment. I want to suggest that that is wrong and that although the majority of us could stand to become much more loving, it is also true that we should direct our love as much as possible towards ethical people and away from unethical people.

I end with a warning. Rationality only becomes powerful when it is combined with knowledge. If you wish to be rational about physics or space travel, it is easy to find accurate knowledge to help you, but it is much more difficult to find accurate knowledge about how to become more loving: in that domain, the accurate knowledge is mixed with a much larger amount of false information. And as has been said here before, most psychologists are idiots.

Hollerith, if 'most psychologists are idiots', I wonder how they discovered all the cognitive biases ?

if 'most psychologists are idiots', I wonder how they discovered all the cognitive biases

He said 'most', not 'all'. And just because someone is an idiot doesn't mean everything they do is wrong. Even Freud managed to do some good descriptive work before descending into madness and delusion.

I mentioned psychologists in a particular context, namely, how to apply the skills of rationality to the project of nuturing and supporting your friends, lovers and family. Worley and I think rationality can be applied to that project. But I thought just leaving it at that would mislead some of the readers who have not had a lot of practical experience in life: unlike many of the other projects rationality is typically applied to, this project is different in that you cannot just travel to your nearest bookstore and by browsing the shelves expect to find accurate knowledge to help you in this project (again because the true information is mixed with a much larger amount of false and misleading information and it is impractical to decide which is which). This remains true even if your nearest bookstore is in an elite university and full of textbooks.

If Eliezer mentions a book or article in psychology in a positive light, that is strong evidence that that book or article is worth reading. In 2001 I took his the advice on his web site and read Robert Wright's Moral Animal and "The Psychological Foundations of Culture", and I am extremely pleased with the outcome.

Caledonian: "I don't believe I've ever heard anyone speak of the aesthetic aspects of rational thought before."

It's funny - the phrase "aesthetics of rationality" appears in the final paragraph of Eliezer's post; apparently it's what the whole thing was about. But I didn't notice it either, until I was seriously casting about for some way to show that Caledonian person why their criticism was off the mark. I think Eliezer's point may be something like this: the aesthetics of rationality are all that could truly make it an end in itself; this necessarily involves attachment to a particular notion of rationality; and this attachment will hinder genuine progress in rationality, which may require adoption of a different but superior notion of rationality.

Along the way, I think I belatedly noticed a subtext to your own first comment too - you think Eliezer, and other promethean transhumanists like him, are themselves examples of limited rationality, their goals or expectations being unrealistic and therefore irrational. I've seen you say as much here, but I hadn't figured out that this was probably on your mind as you wrote your comment.

Anyway, I should get back to thinking about specks versus torture.

Aesthetics are rarely a topic when rationality is discussed. Mostly because they're only relevant to ancient-Greek-style thought.

On the list of things likely to cause unreasonable attachment, it's pretty far down. Love of familiarity, wanting to appear intelligent to others, wanting to appear intelligent to oneself, unwillingness to face conclusions that one finds unpalatable, general inflexibility... these are all plausible causes of failure. But aesthetics?

I think you're pretty close to the core of this one. You identified that having something to protect gives you strength. And having a worthy cause to work for, for the same reason.

But what is that reason? What is it that gives you strength? What is the underlying cause of us gaining strength from certain causes?

I'm not certain I understand the topic well enough myself, but I think I have something that you might find insightful here.

Moral Idealism. That's where your power comes from. Whether you're fighting to protect a loved one, or you're fighting to promote a worthy cause, you have the power to dedicate yourself with every fiber of your being because you believe your actions are righteous!

You see it all the time. When people are completely confident in the righteousness of their cause, they will put their all into it. You see it when someone protects a loved one, you see it when someone works for a worthy cause, and you see it particularly with religions; their unwavering faith in their belief instills them with a sense of righteousness.

I think your mention of people being more afraid of the crowd disagreeing with them than dying highlights a very dangerous philosophical flaw people hold today. They don't believe that protecting their lives is a righteous cause!! Having grown up in an altruistic society, they've probably been hammered with the message that other peoples' lives are more important than their own. So they lack the moral justification to protect themselves and they have a flawed moral premise that works to enslave them to the whim of the crowd.

You're worried about people not having a good reason to be rational? Here's the answer. Your own life must be your ultimate value. It must be an end in itself, and not the means to anyone else's ends. You must judge value with your life as the standard of judgment. Don't think in terms of good and evil, think in terms of good for you and bad for you. Not only is logic and reasoning a tool to promote your life, you depend on it for survival. I can't imagine any way to throw away reason and promote your life at the same time.

(If you can instill people with the power of moral idealism to promote their own lives, you might also have a higher turnout of people buying into cryogenics life insurance policies. :P)

Personally, I find aesthetic purity to be a very strong source of attachment for me. It's certainly caused 'unreasonable attachments', like being stuck on being "right" and ascribing a purity to it (eg. I am right about this and you are wrong, therefore I will absolutely refuse to do this small nitpicky thing and I don't care if I jam up the whole process because it's MORALLY WRONG not to do so. I am the lone voice of dissent!). Oh, school..

I came across the same hack, or coping trick. Just remap the definition of what you're being pure about to "winning" or "rationality".

Pretty sure I'm displaying that I missed the point somehow.

The proper choice between (1) certainly save 400 lives and (2) 90% probability of saving 500 lives with 10% probability of saving no lives, depends on your utility function, which depends on the circumstances. If your utility is proportional to the number of lives saved, then sure, go with (2).

On the other hand, suppose that some cataclysm has occurred, those 500 lives are all that remains of the human race, and extinction of the human race has such an extremely negative utility for you that all other considerations amount to rounding error in the utility function. Then, to a close approximation, you want the choice C that maximizes P(S | C), where S="human race survives".

We have

P(S | C=1) = P(S | N=400)

P(S | C=2) = 0.9 * P(S | N=500)

where N is the size of the current population. Therefore, you should choose (1) if

P(S | N=400) / P(S | N=500) > 0.9.

Holden made a similar point.

All right, I'd like to attempt a summary to make sure that I am understanding this post, if anyone see's some mistake in my interpretation, I'd appreciate it if they let me know.

Virtually everyone wants their beliefs to be true, this amounts to practically everyone wants to be epistimically rational. Rationality is a rare trait, so obviously that desire is not enough to make you epistimically rational. But that desire mixed with the rare desire to have all of your beliefs make useful predictions about whatever they talk about, is enough, provided that you never subordinate the mere predictive power of a belief to its truth. If you allow yourself to believe something because you thought it was true, even after you notice that some other belief makes reliably better predictions about whatever target of inquiry, at that point you fail as a rationalist.

Is there anything I'm missing?

What if what I want isn't true beliefs or to be rational, but to have the best method for finding truths in general, and I use the predictive power of a belief as the best guide to its truth value? In that case, if I find a method that works better than mine, i.e., leads to beliefs with higher predictive power more often than my old method, I'll switch to that method. I don't pride myself on having the best method, I pride myself on doing everything I can to find the best method. And, let's say that finding the best method for finding truth in general is far more important to me than my own life. Is that enough ya think?

It seems a lot like trying to be rational for its own sake, and I know that EY say's that that leads to an infinite recursion, but I don't know why the person I described above must be using circular justification.

please help if you can

1. Save 400 lives, with certainty.
2. Save 500 lives, 90% probability; save no lives, 10% probability.

I think it ought to be made explicit in the first scenario that 100 lives are being lost with certainty, because it's not necessarily implied by the proposition. I know a lot of people inferred it, but the hypothetical situation never stated it was 400/500, so it could just as easily be 400/400, in which case choosing it would certianly be preferable to the second option. I think it's important you make your hypothetical situations clear and unambiguous. Besides, a 100% probability of 100 deaths explicitly stated will influence the way people perceive the question. If you leave out writing out that 100 people are dying, you're also subtly encouraging your readers to forget about those people as well, so it comes as little surprise that some would prefer option 1.

You will never master rationality well enough to win with any weapon, if you do not appreciate the beauty for its own sake.

I have a low prior for this statement, but I don't have any data. I wonder why Eliezer thinks this is the case.

Here I have a question that is slightly unrelated, but I'm looking for a good cognitive science science fair project and I'm having trouble thinking of one that would be not completely impractical for a high-schooler to do, won't take more than a few months, and would be interesting enough to hold people's attention for at least a few minutes before they head off to the physics and medical research projects. No one ever does decent cognitive science projects and I really want to show them that this branch of science can be just as rigorous and awesome as the other ones. Does anyone have any ideas?

I want to read the X saga but I can't seem to find it. Can anyone point my way?

I've been coming back to this post for 7 years or so, and the whole time it's obvious that I don't have something to protect, and haven't found one, and haven't yet found a way to find something to protect. It seems pretty cool though - and accurate that people who really care about things are able to go to great lengths to improve the way they think about the thing and their ability to to solve it.

I can say that once I realized I cared about wanting to care about something, that helped me quite a bit and I started improving my life.

Very interesting. I can't help feeling that "trying to be a better rationalist" is somehow a paradoxical aim.

Roughly speaking I would say that we have preferences, and their is no rational way of picking preferences. If you prefer pizza to icecream, or pleasure to pain, or living to dying, then that is that. Rationality is a mechanism for effectively seeking your preferences, ordering pizza, not putting your had in a fire etc. You can't pick rational preferences (goals), you can pick a rational route towards those goals.

If you adopt "I want to be more rational" as a preference/goal in-itself it feels like the snake is eating its own tail. 

Maybe "meta goals" like this do arise elsewhere, eg. "I don't currently have any interest in being strong/rich/powerful/skilled for its own sake, and nor are these things worth pursuing based on my current preferences (which are more efficiently achieved else-ways).  However, these are things that might be generically useful for achieving preferences I may or may not have in the future, so I should acquire them as tools for later".

But if we take rationality to mean "taking the best actions with the available information to meet your goals", then, at least by this definition, pursuing the meta-goals appears to be definitionally irrational. This extends to the meta-goal of "being a better rationalist".

I savor the succulent choleric chaos of declaring that I value mere phlegm above yellow bile. That is almost a contradiction, but not quite; and the resulting blend has a choleric quality as well: a delicious humor.

Lessons from experimenting prove to be more valuable than from Authority? I think that Adam and Eve would beg to differ. I know, mentioning them probably disqualifies me as fertile ground for rationalist seeding, huh? Oh well, can't win them all.

But anyway, thanks for the well done Harry Potter fanfic. Truly, I am going to reread it several times, I'm sure.