No Safe Defense, Not Even Science

I can't help but remember you talking about a teacher who always had an error in his lectures, and over the course of the semester made them harder and harder to find. The last lecture, which was the most complex, didn't have a flaw.

I was more or less surrounded by people of average sanity when I grew up, but they still seemed pretty nuts to me. (Completely off-topic, but I really wonder why people tell children known-fantasies such as Santa Clause and the Easter Bunny)

I don't think its really accurate to say most people are insane. Clearly they need to be sane for the world to keep on running. IMO, they are insane when they can afford to be - which is pretty common in politics, religion and untestable hypothesizes, but a LOT less common in the workplace. Most people just aren't interested in truth because truth doesn't pay out in a lot of circumstances. I wonder if science might change in your direction (and how quickly?) if betting markets were more commonly accepted?

You're thinking of Kai Chang's My Favorite Liar. It's linked in the Post Scriptum.

When they taught me about the scientific method in high school, the last step was "go back to the beginning and repeat." There was also a lot about theories replacing other theories and then being replaced later, new technologies leading to new measurements, and new ideas leading to big debates.

I don't remember if they explicitly said, "You can do science right and still get the wrong answer," but it was very strongly (and logically) implied.

I don't know what you were taught, but I expect it was something similar.

All this "emotional understanding" stuff sounds like your personal problem. I don't mean that it isn't important or that I don't have sympathy for any pain you suffered. I just think it's an emotion issue, not a science issue.

That is why I am a rationalist and a libertarian. Everyone is totally and completely responsible for every choice they make and everything they do. Every one, from toddler to parent, no one can protect you from your responsibility. That is the difference between a child and a real adult - the adult knows and accepts their responsibility, the less than mature tries to deny or hide from it.

@Brian: Emotion is the driver of everything, even rationality.

Why do you lack social curiosity? Do you think it's a neuro-quirk, or just a normal quirk?

Eliezer,

this article is GOLD!

People who've had their trust broken in the sanity of the people around them, seem to be able to evaluate strange ideas on their merits.

I'd say instead that this prod produces a high variance response. Some rise to the challenge and become more rational, while others fall even deeper into delusion. Yes the most rational people tend to have experienced this, but so also for the most irrational people.

Science is captitalised, suggesting an abnormal definition of the term. Can this definition be found somewhere? What is "Science" - if it it different from science - and if it is not different, then why captialise it?

I started to seriously think about rationality only when I started to think about AI, trying to understand grounding. When I saw that meaning, communication, correctness and understanding are just particular ways to characterize probabilistic relations between "representation" and "represented", it all started to come together, and later was transferred to human reasoning and beyond. So, it was the enigma of AI that acted as a catalyst in my case, not a particular delusion (or misplaced trust). Most of the things I read on the subject were outright confused or in the state of paralyzed curiosity, not deluded in a particular technical way. But so is "Science". The problem is in settling for status quo, walking along the trodden track where it's possible to do better.

Thus, I see this post as a demonstration by example of how important it is to break the trust in all of your cherished curiosity stoppers.

I tried, but didn't find a flaw, anyone else?

The idea that flaws need to be added - and that the final lecture will be flawless - is both silly and presumptuous. There will almost certainly be flaws, whether they are added or not, and our judgment is not adequate to determine whether our own work has them or not.

Eliezer, all of your problems with "Science" seem to stem not from any problems with the method itself, but from your personal tendency to treat the method as a revelation that people have an emotional investment in: in other words, a religion.

There are a variety of ways people can fail to put science into practice. One of the most pernicious is failing to apply it in situations where it is clearly called for, because we have an emotional investment in holding positions that we don't want to disturb. One especially dangerous subtype of this error is when the important subject is our 'scientific' reasoning and the conclusions we derived from it. It is even more dangerous than the general case because it doesn't just involve a corruption of our ability to deal with one specific set of problems, but a corruption of the general method we must use to rationally investigate the world. Instead of merely having a blind spot, we lose our sight completely, while at the same time losing our ability to detect that we're blind.

You are guilty of this error. That doesn't mean that you've gained a unique insight that must be shared with the world at all costs. This is a very old and trivial insight that most people worth listening to have already produced independently.

I agree with your general view, but I came to the same view by a more conventional route: I got a PhD in philosophy of science. If you study philosophy of science, you soon find that nobody really knows what science is. The "Science" you describe is essentially Popper's view of science, which has been extensively criticized and revised by later philosophers. For example, how can you falsify a theory? You need a fact (an "observation") that conflicts with the theory. But what is a fact, if not a true mini-theory? And how can you know that it is true, if theories can be falsified, but not proven? I studied philosophy because I was looking for a rational foundation for understanding the world; something like what Descartes promised with "cogito ergo sum". I soon learned that there is no such foundation. Making a rational model of the world is not like making a home, where the first step is to build a solid foundation. It is more like trying to patch a hole in a sinking ship, where you don't have the luxury of starting from scratch. I view science as an evolutionary process. Changes must be made in small increments: "Natura non facit saltus".

One flaw I see in your post is that the rule "You cannot trust any rule" applies recursively to itself. (Anything you can do, I can do meta.) I would say "Doubt everything, but one at a time, not all at once."

@Caledonian: If it is an old and trivial insight, why do most scientists and near all non-scientists ignore it?

As Eli said in his post, there is a difference between saying the words and knowing, on a gut level, what it means - only then have you truly incorporated the knowledge and it will aid you in your quest to understand the world.

Also, you say: Caledonian: but from your personal tendency to treat the method as a revelation that people have an emotional investment in

Of course people have an emotional investment in this stuff!! Do not make the old mistake of confusing rationality with not being emotional (I guess Star Trek with Mr. Spock is guilty of that, at least for our generation)

And what could be more emotional than dumping the legends of your tribe/parents/priests/elders?

For rationality and emotion in science, read for instance here: The Passionate Scientist: Emotion in Scientific Cognition Paul Thagard http://cogsci.uwaterloo.ca/Articles/Pages/passionate.html

You will have to study [...] and social psychology [...]

Please could you recommend some social psychology material?

As you explain so clearly here, the point is to think for ourselves instead of trusting in any person or system. This valuable insight can be reached by many idiosyncratic paths through life. Your personal path to it, trusting too much in Science itself, is an ironically interesting one, unlikely to be trod by most. That's why your line "Science Isn't Strict Enough" fails to resonate with some readers.

Jared, why should you trust yourself more than someone else? And if there is someone more worthy of trust than you, wouldn't it be a more rational strategy to let him think for you instead of thinking for yourself?

If my own judgment is so faulty that I choose to let somebody else do all my thinking for me, then how can I even trust the thinking behind my choice?

If you think that Science rewards coming up with stupid theories and disproving them just as much as more productive results, I can hardly even understand what you mean by Science beyond the "observe, hypothesize, test, repeat" overview given to small children as an introduction to the scientific method. Was Eliezer-18 blind to anything beyond such simple rote forumulas?

Negative results are forgiven but hardly ever rewarded (unless the theory disproven is widely believed).

If you'd put aside the rather bizarre bitterness and just say: "Bayesian rationality is a good way to pick which theories to test. Here's some non-toy examples worked through to demonstrate how" that would be much more useful than these weird parables and goofy "I am an outcast" rants.

"@Caledonian: If it is an old and trivial insight, why do most scientists and near all non-scientists ignore it?"

They don't. The mismatch between you and them is that they're busy thinking about something else at the moment. I like the rule Turney gave above: "Doubt everything, but one at a time, not all at once." Of course, a single person can't follow that rule completely (there's not enough time in a lifespan to doubt EVERYTHING), and most people pick the wrong things to doubt or are lazy in applying the rule.

Of course, that rule's going to get in the way of reaching truth in some cases (some falsehoods come in self-reinforcing pairs both of which must be doubted in order to falsify either, and some things can't profitably be denied even for the sake of argument), but that's the case with any process, and this is something we've known since Goedel.

This kind of confuses me about this series... If all he was telling us was that Science is a powerful set of rules, and that therefore it can't eliminate all contradictions nor state all facts, I'd simply agree with him. But he seems to be saying that Bayesianism is different from Science, that somehow applying it instead of Science will have better results. It seems to me that both are processes, and both have blind spots.

I find it difficult to be sympathetic towards someone who complains he wasn't warned that the rule "do not take things on faith" wasn't supposed to be taken on faith.

We could provide a warning, of course. But how would we then ensure that people understood and applied the warning? Warn them about the warning, perhaps? And then give them a warning about the warning warning?

We could talk until we're blue in the face, but the simple truth is that you cannot force people to apply a method consistently, rigorously, or intelligently. No amount of adding onto the lesson will make people apply it properly, it merely offers them more things to misunderstand, ignore, and apply inconsistently.

We could provide a warning, of course. But how would we then ensure that people understood and applied the warning? Warn them about the warning, perhaps? And then give them a warning about the warning warning?

That's the problem with discrete reasoning. When you have probabilities, this problem disappears. See http://www.ditext.com/carroll/tortoise.html

@billswift: Emotion might drive every human action (or not). That's beside the point. If an emotion drives you into a dead end, there's something wrong with that emotion.

My point was that if someone tells you the truth and you don't believe them, it's not fair to say they've led you astray. Eliezer said he didn't "emotionally believe" a truth he was told, even though he knew it was true. I'm not sure what that means, but it sounds like a problem with Eliezer, involving his emotions, not a problem with what he was told.

Jared, it is possible to see that someone is more intelligent and trustworthy than you, without therefore being yourself more intelligent and trustworthy than him.

Eliezer didn't trust science too much. He didn't trust it enough. Instead of taking the duties and requirements of skepticism seriously, he treated the scientific method as another faith system.

I'm sure that was a very comforting and familiar approach to take, but it was still wrong. Completely, fundamentally wrong. It's utterly incompatible with the skepticism, open-mindedness, and radical doubt that is essential to the scientific method. And it seems to have had long-lasting implications for the sorts of positions Eliezer takes.

One suggestion for the flaw:

Conclusions from this article: a) you are never safe b) you must understand a) on a emotional basis c) the only way to achieve b) is through an experience of failure after following the rules you trusted

The flaw is that the article actually does the opposite of what it wants to accomplish: by giving the warning(a) it makes people feel safer. In order to convey the necessary emotion of "not feeling safe"(b) Eliezer had to make the PS regarding the flaw.

In a certain sense this also negates c). I think Eliezer doesn't really want us to fail(c) in order to recognize a), the whole point of overcomingbias.com is to prevent humans from failing. So if Eliezer did a good job in conveying the necessary insecurity through his PS then hopefully c) won't happen to you.

Roland

Roland, agreed.

Does anyone disagree that science does not have nearly as strict quantitative constraints as Bayescraft on what you may believe?

@Vladimir Nesov:

why do you say that the problem disappears when you have probabilities?

I guess you still have the same basic problem which is: what are your priors? You cannot bootstrap from nothing and I think that is what the tortoise was hinting at, that there are hidden assumptions in our reasoning that we are not aware of and that you can't think without using those hidden assumptions.

Roland,

Probabilities allow grades of beliefs, and just as Achilles's pursuit of tortoise can be considered as consisting of infinite number of steps, if you note that steps actually get infinitely short, you can sum them up to a finite quantity. Likewise, you can join infinitely many infinitely unlikely events into a compound event of finite probability. It is a way to avoid regress Caledonian was talking about. Evidence can shift probabilities on all metalevels, even if in some hapless formalism there are infinitely many of them, and still lead to reasonable finite conclusions (decisions).

Likewise, you can join infinitely many infinitely unlikely events into a compound event of finite probability. It is a way to avoid regress Caledonian was talking about.

No, Mr. Nesov, it is not. You and I are talking at cross purposes.

Caledonian, you are not helping by disagreeing without clarification. You don't need to be certain about anything, including estimation of how much you are uncertain about something and estimation of how much you are uncertain about the estimation, etc.

"The experimental method shall decide which hypothesis wins"?

When there are experiments that can reasonably be done, or have already been done, then this works, right?

"Do you think it's a neuro-quirk, or just a normal quirk?"

Wait, there's another kind?

"Doubt everything, but one at a time, not all at once."

Interestingly, Robin Hanson has an existing post on this subject:

To have the best chance of succeeding in a radical project, you should instead choose just a few related dimensions on which to make radical choices, and then make conservative conventional choices on all the other dimensions. This strategy minimizes the chance that some other project dimension will go badly wrong and take down your central radical idea with it.

About flaws in the post: the idea that environmentalists shouldn't oppose a scaling up of nuclear power is a flaw. This paper: http://www.stormsmith.nl/ was lucky enough to be written (I was lucky enough to find it since every nuclear utility on the planet ignored unearthing the basic economic analysis contained within it). Basically, scaling up of nuclear fails to cost the complete life-cycle of decommissioning a powerplant. And, additionally, going to all nukes ensures the nuclear lobby (the process of lobbying is something Libertarians don't understand) becomes a nearly permanent chunk of the world's economy. My point is that Bayesian reasoning here only unearths the nuclear facts and lies the nuclear industry forwards. You need reliable information sources in addition to Bayes to make correct judgement calls on eoconmics/energy here.

Cryonics is another flaw. IDK if it works or not. But the expensive process certainly shouldn't be a part of Universal Healthcare coverage at present. The best research (not 1970's deductive reasoning)about brains I've read to date, suggests thought is a substrate specific (IDK how much, semiconductors no way but maybe more inclusive than CNS proteins) process that functions as temperature-dependant solitons. Whatever temperature the brain goes down to in the cold water of ice-slip hypothermia survivors, does not necessarily mean brain processes will survive liquid nitrogen or helium temperatures. Under the mathematical model of reality most transhumanists have, temperature (requires physics) doesn't even exist!! I hope cryonics works, and if I were rich enough I might sign up or fund suspension research, but to suggest those who don't believe in cryonics are fools is to suggest brains work identically at 25C and -273C. Then the "rationalist" rebuttal is always to invoke "uploading for immortality" (where Transhumanism dies to me, despite all its progressive memes). If rationalists can't understand mathematics isn't physics, I don't want to be labelled a rationalist and I will pepper posts such as this to avoid unsuspecting readers from mindlessly believing a mindless belief system; I am trying to prevent H+ from functioning as a cult.

Intrade free money?! Surely you must know it takes $5000-$15000/yr in basic costs alone to live in most of the Western world. Surely you must know the average savings rate is very low. Please qualify statements like this with "rich/middle classes can make investments on liquid (can Insite really handle trillions of $$ as suggested and wouldn't it then be subject to manipulation: I'll bet the 65 cents in my pocket Phillip will splash coffee on himself). Bayes may be important, but if it misses very basic facts (like 9/10ths of the world can't presently afford to live off investment income), why would the world want to incorporate more Bayes, a small subset of probability theory already in math, logic and computer science curriculums (I think)? Sweet. I did splash coffee on myself and double up to $1.30. Now I can donate to the political party most likely to teach (for the purpose here of reasoning skills, not ethics) probability theory instead of religious dogma, in public schools. That is the conclusion of the post: better public education at the expense of pop-culture. Or more funding for public education commercials?

I don't know about you guys, but being wrong scares the crap out of me. Or to say it another way, I'll do whatever it takes to get it right. It's a recursive sort of doubt.

Hi,

Once again interesting post, but it doesn't apply to my personal case. I've access to no statistics on the issue so I can't claim how exceptional I am, but my own parents are "more rational than average" (they are atheist maths teachers), and I don't think they are insane. Or not more than anyone else, at least.

I did realize that my parents were not perfect, and that if I could trust them in loving me and caring for me and wanting the best for me, I couldn't blindly trust anything they would say. But that didn't require, for me, such a massive emotional break.

Maybe you'll tell I'm not "unusually rational"... but at least, I'm trying to better myself, or I wouldn't spend hours reading LW ;)

Before reading LW, I was not really doubting Science, but neither was I considering it to be an utter sacrilege to claim it was flawed (or that it could be superseded by something better). I knew Bayes' Theorem, without realizing it was much more important than just understanding a mammography result (which was the way I was using it). That's a shortcoming of my part, sure.

I can say that even before reading LW, I had a gut feeling that the Cophenaguen interpretation was just... not right. I was in doubt, torn between that gut feeling, and my own humility saying "well, you're not a physicists, how can you be more right than they are ?" MWI seems much cleaner to me (even if there are a few things that still bother me), but that gut feeling drove me into reading more about QM and rationality. That, and having something to protect.

Well, that's just my own personal experience, hoping it can help understand things better, but not claiming any generalization from it.

Now that we have once again established that 1 and 0 are not probabilities, we have to remember that probabilities are still a strictly ordered set. How do we make it less dangerous?

For me, the discovery that science is too slow was bound up with the realization that science is not safe. My private discovery of the slowness of science didn't come from looking at the process of scientific discovery and reflecting on the time it took - rather, it arose from realizing that the things I learned or discovered via science were slower more painful than those I learned from other methods. "Other methods" encompasses everything from pure mathematics to That Magical Click, the first inescapable and the second, initially, unsupported. Realizing that science was a fairly low-quality set of tools carried with it the realization that its inefficiency was a function of its precautions. Not trusting science as the ideal method for discovery, I ceased to trust it as ideal for reliability.

New to this site, Bayescraft, and rationalism as a whole, I still have a mentor left to distrust. Consciously, I know that these techniques are imperfect, but I have yet to understand them well enough to be failed by them.

Welcome to the Earth where ethanol is made from corn and environmentalists oppose nuclear power.

I find this to be a very attention grabbing comparison, so much so that I had to re-read this post 5+ times before I could see the forest through the trees (or tree as the case may be).

The reason these two examples strike me so is that I once held both of the underlying beliefs (ie that corn ethanol is bad and so is nuclear power). While I reversed both of these beliefs many years ago (prior to discovering HPMOR and lesswrong) I now see them as "belief as attire" (tree huggers think nuclear is bad, I'm a tree hugger, therefore I think nuclear is bad) and "password guessing" (why is corn ethanol a bad idea?... thermodynamics....Gold Star!)

After gathering more information about these two "controversies" than can be gathered from Mother Jones or Popular Mechanics, I firmly support nuclear power expansion and think it is quite insane that we don't make more ethanol from corn. I would be happy to support my positions, the former would be rather concise, the later would be considerably longer, so I'll save it until asked.

Perhaps this would have been less distracting:

Welcome to the Earth where 46% of Americans believe in creationist origins of humans and only 15% believe in evolutionary origins of humans.

No one begins to truly search for the Way until their parents have failed them, their gods are dead, and their tools have shattered in their hand.

Nihil supernum.

That post scriptum.. It's just so amusing to have someone write out your exact thoughts and worries like that. It is very rarely that i get tears in my eyes from giggling, and I can't stop smirking about that. It is quite bothersome indeed that I am so unskilled in this art of reasoning that the best i can do is follow your words and hope they lead me to somewhere where i can eventually realize said flaws. I feel my journey will be riddled with such flaws no matter how hard i try to to avoid it. It's the nagging feeling I have almost constantly, and of which i have trouble explaining

What's trust?

Forget it. It's an honest question but it'll just appear as attention grabbing.

This is probably my favorite entry ever.

What's wrong with ethanol made from corn, anyway?

The only flaw I can see it's that this reasoning seems to put a lot of weight on the probability of meeting disaster by sticking with a well thought, well tested but imperfect set of rules, and not enough weight on the probability of meeting disaster by trying to be clever and do better than the rules, either by genuine fear of their limitations, or to follow a course of actions that you favour for reasons that aren't really part of your goals but that you are then free to endorse by using the fear of the rules' limits.

I get that the main point of the post is that you can't just relax, get comfortable and think you finally got the perfect way of thinking if you want to actually get the right answers often enough, and I also agree with it. But I still feel that, when considering if to depart from the rules, even those of the lowly common sense, the possibility that you are just about to shoot yourself in the foot and meet some interesting new disaster for reasons you'll see only after and that could have been avoided by sticking to it, has a significant probability compared to the possibility that you are actually in a situation where the rules aren't enough and you'll have to improve them to succeed, that this probability should be carefully weighted before choosing how to act, and that this warning should have been included in the reasoning above, as it has usually been included in other posts.

A practical example would be making this comment as a newcomer, ignoring the common sense consideration that as a flaw it feels kinda obvious and that someone else would have pointed it out by now if I weren't simply missing the point of the post, but I still think that "don't trust the rules" is a rule that requires an awful lot of caution before being mistrusted too much.

Does your experience support the claim that rationalists had their trust in their milieu (e.g. parents, cult, etc.) broken?

P.s. that was my personal experience but I'd really like to hear from others.

This wasn't precisely the reason I got here, but I think the biggest reason that I was open to the idea of rationality when I finally stumbled upon LessWrong and the whole Effective Altruism idea was my experience of becoming gay, after diverging from my christian upbringing. 

During my time going to bible school (yes, I was in that deep), there was a lot of theology where I was confused and felt like I didn't understand why something was true, but just accepted that "I'm sure some high-level theologian out there has a reasonable answer for this, I just don't know it." However, once I began living on my own, naturally drifted away from the church, and then discovered that gay stuff wasn't as bad as I'd been told, I realized (this is how I think of it now) that there was a very large black mark on the source of knowledge internally labelled as "religious authority." That was the first frayed thread in that anachronistic tapestry I had properly seen unraveled for myself, and I suspect that there are many more of such threads that I overlooked previously.