Epistemic Luck

Try this: Choose a book that you expect to disagree with and read it from start to finish over several weeks. See what impact it has on you. I tried this and felt my beliefs changing despite none of the arguments being convincing. It seemed to peter out a few weeks after I finished the book. I hypothesize that in an extended experiment we could actually brainwash ourselves to the point of holding some radically different views.

"Isn't it lucky that we happened to be born into the one true faith?"

In my case, I got a lot of my atheism from my father; I haven't had the kind of "crisis of faith" that many other aspiring rationalists said they had. (My mom is a non-practicing Catholic.) I was practically born into this culture, so I get worried about this on occasion.

Surely the truth about knowledge and justification isn't correlated with which school you went to

It seems pretty likely that there is some correlation. (Suppose, without loss of generality, that some kind of epistemic externalism is true. Then some schools -- the ones where externalism is popular -- correlate with the production of true beliefs about epistemology.) The problem is just that we don't know in advance which schools actually have it right.

Perhaps what you mean to suggest is that going to school X isn't a sensitive method of belief formation. Even if what it taught was false (like school Y), you would still end up believing it (just as the students of Y do).

Then again, one could say much the same thing about being born into a religious cult (or a different historical epoch). I do consider myself lucky to have avoided such an epistemically stunted upbringing. But this kind of luck does not in any way undermine my present beliefs.

In Meta-Coherence vs. Humble Convictions, I suggest that what matters is how you assess the alternative epistemic position. If you really think it is just as well-informed and well-supported as your own, then this should undermine your present beliefs. Otherwise, it need not. To consider oneself the beneficiary of epistemic luck is not necessarily irresponsible. (Indeed, it's arguably necessary to avoid radical skepticism!)

This reminds me of Where Recursive Justification Hits Bottom, where Eliezer explains that an agent that uses Occamian priors and Bayes' rule, when evaluating whether this is effective, would assign an Occamian prior to the theory that it is effective and update using Bayes' rule to account for its success and failures.

If you learn and adopt a mode of epistemology at one school, that is what you will use to evaluate a competing mode advocated by another school.

Signaling is a factor, but there is another issue. That is the choice of what to work on. Academics and academic departments, no less than individuals, have to make choices of where to focus their time and resources - no one has unlimited time and attention and choices have to be made. So one person, or school, may focus on cognitive biases and skimp on memory biases, or the other way around. This choice of what is important is actually very strongly affected by signaling issues - much more so than signaling affects particular beliefs, I think.

I see some problems here but it doesn't seem quite as intractable as Alicorn suggests.

If your beliefs are highly correlated with those of your teachers then you need to immerse yourself in the best arguments of the opposing side. If you notice that you are not changing your mind very often then you have a deeper problem.

To give a few related examples. One of the things that gives me confidence in my major belief structure is that I am an Atheist Capitalist. But, as I child I was raised and immersed in Atheist Communism. I rejected the communism but not the Atheism. At least in the small set my parents/early teachers were only 50% right in their basic belief structure and that doesn't sound too unlikely.

On the other hand I have been troubled by the extent to which I have become more sensitive to liberal arguments over the past 2 years. My social and professional circle is overwhelmingly liberal. It is unlikely that this does not have an effect on my beliefs.

To compensate I am attempting immerse myself in more conservative blogs.

Now of course there is no way to be sure that the balancing act is working. However, if we take as a starting point that errors among well informed people are randomly distributed then as a rough approximation your adherence to the beliefs of your community should be proportional to the number of intellectuals who hold those same beliefs.

Having been sincerely religious as a young adult has its drawbacks for my present self (e.g. lost time and effort), but one positive effect is that I'm not as worried about this, because I've felt what it's like to find that my entire worldview fails on its own terms. (Basically, I eventually came to realize that it was completely out of character for the deity I knew— or any deity derived from the scriptures and theologies I professed— to create a universe and humanity completely un-optimized along any recognizable moral axis. But I digress.)

I was lucky enough to have a starting point with an intellectual and philosophical tradition (rather than, say, the extremes of evangelical Christianity), so I can't credit myself too much. But unlike people who haven't ever had to fundamentally change their minds about the world, I can point to a set of evidence that would make me switch to another worldview, and I can be reasonably confident that I'd do so if given that evidence. (It would generally take more evidence for me than for a pure Bayesian, but working inefficiently is better than refusing to work at all.)

Can you think of another behavior pattern that is more accurate than this?

Assuming that someone isn't going to hold a belief they know to be false, they are teaching you perceived truth. Why wouldn't you adapt those beliefs? If you, the student, possessed the ability to denounce those beliefs in a manner fitting rational discussion it seems likely that the master would have been able to do so as well.

This isn't to say all masters are all right all the time, but what else do we have to go on until we go into the territory ourselves? These people went ahead and brought back a map. Different trailblazers saw the territory differently and now they get to bicker about which has a more accurate map.

When learning from people, I assume we copy portions from their map. They are lined up next to my discoveries and observations and are assimilated as best as possible. At some point, when I wander through that territory, I get to compare my map to what I see.

Am I wrong or missing something? Or is that the whole point of the post? Or... ?

Lots of biases still live in your head doing their thing even when you know about them. This one, though, puts you in an awfully weird epistemic situation. It's almost like the opposite of belief in belief - disbelief in belief. "This is true, but my situation made me more prone than I should have been to believe it and my belief is therefore suspect. But dang, that argument my teacher explained to me sure was sound-looking! I must just be lucky - those poor saps with other teachers have it wrong! But of course I would think that..."

Emphasis mine. I don't see how the emphasized part is a bias issue. I don't see how thinking about how a certain area of your map was colored causes the weird epistemic situation. I think the problem is this:

I must just be lucky - those poor saps with other teachers have it wrong!

A predictor is right or wrong. Your teachers are irrelevant once the belief is in your map. Treating beliefs transported from a teacher as a lucky event has a few problems:

• It implies that your teacher is your teacher because you were lucky. Choose your teacher better.
• It implies that teacher A and teacher B have the same map quality of a particular territory but they disagree. This is irrelevant until you get to look at the competing maps or the territory involved. Your teacher is still the best guess you have. Why would your map look like anything else?
• It implies that you cannot change you (or your unlucky friend's) beliefs as new evidence or more maps become available.

But all of this seems obvious and you address these points elsewhere in your post.

So perhaps you are saying that once you believe something, it becomes entrenched and now has an unfair advantage? I certainly understand this. As far as I can tell, this is one of the problems that rationality was supposed to address. Is the teacher scenario merely a special case?

These have a decided enough effect that I've heard "X was a student of Y" used to mean "X holds views basically like Y's".

It still does. The question is open to what extent this is because students tend to choose advisers whose views they already agree with.

Hmm... This might actually be a major breakthrough of sorts. When I have time, I'll delve into the literature to see what folks have to say about it. On the surface, this seems related to the notion that we just need a good-enough theory to make progress towards a better theory.

It might be the case that folks in different departments are climbing the same mountain from different sides; if people in department A are advocating Big Theory p, and people in department B are advocating Big Theory q, and p and q are both approximately correct, then we shouldn't be surprised that they both have proponents.

Are you sure you have the causality of it right? I always thought of graduate schools as selectors/filters for certain kinds of intelligences and pov, rather than causators.

"But dang, that argument my teacher explained to me sure was sound-looking! I must just be lucky - those poor saps with other teachers have it wrong!"

This is actually something I've been wondering about regarding the disproportionate overlap between libertarianism and anthropogenic global warming skepticism. I'd like to think that this disproportionate overlap is because both views stem from a rational and objective assessment of the available data, but at the same time, I can't deny that anthropogenic global warming would throw a monkey wrench into libertarian philosophy if it was real, so being skeptical of it saves us from doing an awful lot of mental gymnastics...

Learning the arguments of opposing viewpoints is helpful, but if you're being truly rational, it shouldn't be necessary to remove bias.

I figure that if you know there are other groups with different beliefs, they are likely to have heard arguments you haven't. You should assume there are arguments, and use them to adjust your beliefs. If you later hear such arguments, and they make more sense then you thought, then that should make you believe them more. If they make less sense, even if they're still good arguments and would only help convince you if you hadn't already assumed their existence, you should believe them less.

On the other hand, if you've studied it significantly, you've probably heard pretty much all the best arguments on each side, and the problems are just due to being irrational. I guess you should try to fight that by trying to remain skeptical of the opinions of people near you, regardless of whether they're more or less normal than what you believe.

Thinking out loud: So if you're french,trying to figure out what true english way to say something would be, you don't want to... you want to specifically avoid the kinds of mistake french speakers (you) tend to make because they are french. What a native english speaker might call frenglish. You have access to native spanish and japanese speakers, you know they are making mistakes, and you know that some of their mistakes are made because of their native language.

One source of information is more fluent french speakers, they have overcome some obstacles unique to francophones, and they can point them out in less fluent speakers. This may give a pattern of errors indicating the kind of errors francophones tend to make in learning english. (especially if you can compare to known early errors in spanish/japanese speakers)

So, first look at what the near-fluent speakers of all nationalities agree on as english. Then look at the mistakes made by learners in various nationalities. This will give you a fairly empirical picture of what a strong french accent looks like. That should give you some hints about what a weak[slight] french accent would look like. Correct for that, and you'll be closer to native english.

Widening the spread of your mentors should reduce this bias, as long as you didn't choose mentors that agree with each other. Obviously, there isn't really enough time to be taught from a wide enough sample of perspectives to properly eliminate it.

It's almost like the opposite of belief in belief - disbelief in belief.

More like belief in disbelief.

During one of my epistemology classes, my professor admitted (I can't recall the context) that his opinions on the topic would probably be different had he attended a different graduate school.

I read this as an admission that modern academic philosophy has nothing whatever to do with the search for truth and everything to do with status-seeking, signalling, and affiliation games. But at least he was being sort of honest.

This is good. It seems (to me) to mean that the LessWrong community is starting to "get the hang" of the importance of explanation...

By that I mean that a person who found themselves in the state of being "very" "intelligent" might, at the exact same time that they realized their state of intelligence had been the result of what we call "insights"--a working out of the problem on a level-independent way from the presuppositions inherent in the overwhelming bias of the problem as stated...

that that agent would also concurrently understand that the simple act of "explaining" how it might look to another person trapped further down the scale would cause them to agree, and hence look upwards, seeking the thing we had described for their benefit.

tl;dr in this case is: explanation IS intelligence, as much as insight if not more. "Misunderstood geniuses" is an oxymoron, no matter what genius-level you find yourself in,

There's no obvious way to compensate.

Think for yourself.

I know it sounds trite, but really, any time I might spend expouding on the theme of socialized learning would only yield fancier ways of saying pretty much that.