Expecting Short Inferential Distances

The explanation from ancestral environment seems likely. However, there is also a rational argument for refusing to accept a claim unless all the steps have been laid out from your own knowledge to the claim. While there are genuine truth seekers who have genuinely found truth and who we therefore should, ideally, believe, nevertheless a blanket policy of simply taking these people at their word has the unfortunate side-effect of also rendering us vulnerable to humbug, because we are not equipped to tell apart the humbug from the true statements many steps removed from our knowledge.

At the same time, people do not universally reject claims that are many steps removed from their own experience. After all, scientists have made headway with the public. And unfortunately, humbug also regularly makes headway. There have always been niches in society for people claiming esoteric knowledge.

Eliezer, this is a great insightful observation.

The young seem especially vulnerable to accepting whatever they are told. Santa Claus and all that, but also any nonsense fed to them by their schools. Schools for the young are particularly effective instruments for indoctrinating a population. In contrast, the old tend to be quite a bit more resistant to new claims - for better and for worse.

An evolutionary explanation for this is fairly easy to come up with, I think. Children have a survival need to learn as much as they can as quickly as they can, and adults have a vital role as their teachers. In their respective roles, it is best for adults to be unreceptive to new claims, so that their store of knowledge remains a reliable archive of lessons from the past, and it is best for the young to accept whatever they are told without wasting a lot of time questioning it.

As long as we're on the subject of evolutionary-psychology/sociobiology/whatever if someone tries to argue against it by saying it's just a bunch of reactionaries trying to justify inequity you can point to the data which says it ain't so. Another soldier sent against the army of reductionism defeated, surely a signal from Providence that all will be assimilated.

For example, talking as if you think "simpler explanation" is a knockdown argument for evolution (which it is)

I don't quite agree - by itself, X being "simpler" is a reason to increase my subjective belief that X is true (assuming that X is in a field where simplicity generally works) but it's not enough to prove e.g. creationism false. Rather, it is the total lack of evidence for anything supernatural that is the knockdown argument - if I had reason to believe that even one instance of say, ghosts or the effects of prayer were true, then I'd have to think that creationism was possible as well.

This is certainly an insightful post. I'm not sure the example is that compelling though.

If you argue with a young earth creationist, they could full well understand what you mean, but simply disagree and claim "God did it," is a simpler explanation still. In fact, if we were to presuppose an intelligent being of infinite power existed and created things, it seems it would actually be a simpler explanation.

Most people, though perhaps not all, who have no belief in an omnipotent designer will pretty quickly accept evolution. So that might not be the cognitive problem in that situation.

I'm sure there are examples out there (opposition to free trade, perhaps?), but the rejection of evolution in favor of creationism is rather more complex and deep rooted.

When you say "A clear argument has to lay out an inferential pathway, starting from what the audience already knows or accepts. If you don't recurse far enough, you're just talking to yourself."

this strongly reminds me of what it is like to try talking, as an atheist, with a christian about any religious issue. I have concluded years ago that I just shouldn't try anymore, that reasonable verbal exchange is not possible...

I suppose that I should recurse... but how and how far where I am not sure.

This reminds me of teaching. I think good teachers understand short inferential distances at least intuitively if not explicitly. The 'shortness' of inference is why good teaching must be interactive.

Psychohistorian: it depends on what you mean by "simple" and "explanation". The sense in which "it's the simplest explanation" is a powerful argument for something is not one in which "God did it" is the simplest explanation for anything.

Eliezer_Yudkowsky: I've seen the kinds of failures of explanation you refer to, and there's also the possibility that the explainer just isn't capable of explaining all of the inferential steps because he doesn't know them. In that case, the explainer is basically "manipulating symbols without understanding them". This is why I've formulated that principle (sort of a corollary to what you've argued here) that:

"If you can't explain your idea/job/research to a layman, given enough time, and starting from reference to things he already understands, you don't understand it yourself."

I have experienced this problem before-- the teacher assumes you have prior knowledge that you just do not have, and all of what he says afterwards assumes you've made the logical leap. I wonder to what extent thoughtful people will reconstruct the gaps in their knowledge assuming the end conclusion is correct and working backwards to what they know in order to give themselves a useful (but possibly incorrect) bridge from B to A. For example, I recently heard a horrible biochem lecture about using various types of protein sequence and domain homology to predict function and cellular localization. Now, the idea that homology could be used to partially predict these things just seemed logical, and I think my brain just ran with the idea and thought about how I would go about using the technique, and placed everything he said piece-wise into that schema. When I actually started to question specifics at the end of the lecture, it became clear that I didn't understand anything the man was saying at all outside of the words "homology" and "prediction", and I had just filled in what seemed logical to me. How dangerous is it to try to "catch up" when people take huge inferential leaps?

Yes, this is good stuff, I wish I could identify the inferential gaps when I communicate!

Silas, aren't there some things it is simply impossible for some people to understand?

TGGP: Yes for people below some IQ threshold. No for someone of the same IQ as the explainer.

(I probably should have added the intelligence criterion the first time around, I guess, but I was simplifying a bit.)

This is an excellent post, Eliezer!

Taking this phenomenon into consideration not only gives me cause to go back over my own teaching technique (of a rather specialized trade) and make sure I am not leaving out any steps that seem obvious to me (the specialist), but, like Laura, it helps me to understand times when I was baffled by a speaker or writer whose tone implied I'd be an idiot not to follow along easily.

When I write for a very bright "puzzle-solving-type" audience, I do the mental equivalent of deleting every fourth sentence or at least the tail part of every fourth sentence to prevent the reader from getting bored. I believe that practice helps my writings to compete with the writings around it for the critical resource of attention. There are of course many ways of competing for attention, and this is one of the least prejudicial to rational thought. I recommend this practice only in forums in which the reader can easily ask followup questions. Nothing about this practice is incompatible with the practices Eliezer is advocating. This week I am experimenting with adding three dots to the end of a sentence to signal to the reader the need mentally to complete the sentence.

So, what sentence did I delete from the above? A sentence to the effect that I only do this for writing that resembles mathematical proof fairly closely: "Suppose A. Because B, C. Therefore D, from which follows E, which is a contradiction, so our original assumption A must be false."

After writing a first draft, I go back and add a lot more words than I had saved with the "do not bore the reader" practice. E.g. I add sentences explicitly to contradict interpretations that would lead to my being dismissed as hopelessly socially inept, eccentric or evil. Of course because I advocate outlandish positions here, I still get dismissed a lot.

Richard, you may or may not care that having read the above my willingness to read anything you write in future has somewhat decreased.

I would add, Richard, that writing "dear reader" on a medium like this comes off as patronizing.

Some of your claims about the EEA are counterintuitive to me. Basically, it's not obvious that all information not strictly private would have been public. I'm thinking, for example, of present-day isolated cultures in which shamans are trained for several years: surely not all of their knowledge can be produced in a publicly comprehensible form. There must be a certain amount of "Eat this herb -- I could tell you why, but it would take too long to explain". Or so I imagine.

So how much of your description of knowledge in the EEA is your guessimation, and how much is the consensus view? And where can I find papers on the consensus view? My Google-fu fails me.

I present to you Exhibit A from the field of computer programming.

I find an easy way to get some of the complicated inferential jumps for free is to find a similar set of inferential jumps they have made in a similar subject. It is much easier to correct a "close" inferential jump than it is to create a new one out of thin air.

Example: When discussing the concept of programming you can use the concept of an assembly line to get their head into a procedural mode of thinking. Once they think about an object visiting a bunch of stations in a factory you can replace "object" with "program" and "station" with "line of code." They still have no idea how programming works, but they can suddenly create a bunch of inferential jumps based on assembly lines.

In my experience, they now start asking questions about programming as related to assembly lines and you can fill in the gaps as you find them.

"So what happens at the end of the line?"
"Well, the program generally loops back around and starts over."
"Oh. So it follows the same line forever?"
"Not necessarily. Sometimes the line takes a detour and heads off into a new area of the plant for awhile. But it generally will come back to the main assembly line."
"But what's the point? Like, how does that make my computer run?"
"Think of the computer like the company. The company owns a whole bunch of assembly lines all over the place and, periodically, it will ask a certain plant to start up and keep running. One of the stations in the assembly line is something like, 'Give report to company'. The company looks at the report, realizes it is a visual report, and hands it to the assembly line that processes visual reports. That assembly line takes the report, breaks it down into RGB pixels, and puts it on the monitor for you to see. For that to happen, all of these programs have to keep spinning on their assembly lines and doing the work at each station and keep sending reports back to the company. You don't have to worry about all of the lines because the computer is doing it for you."
"Wow, that's complicated."
"Yeah, it can get pretty crazy. As a programmer, I design the assembly lines."

Or whatever.

As someone who has done (some) teaching, I think this is absolutely correct. In fact, the most difficult thing I find about teaching is trying to find the student's starting knowledge, and then working from there. If the teacher does not goes back enough 'inferential steps', the student won't learn anything - or worse, they might think they know when they don't.

Excellent stuff.

Now I think of it, this reminds of something Richard Dawkins used to say at some talks: that we (the modern audience) could give Aristotle a tutorial. Being a fantasist myself, I've sometimes wondered how that could be possible. Leaving aside the complications of building a time machine (I leave that to other people), I wondered how would it be to actually meet Aristotle and explain to him some of the things we now know about life, the universe & everything.

First of all, I'd have to learn ancient greek, of course, or no communication would be possible. That would be the easy (and the only easy) part. More complicated would be that, to teach anything modern to Aristotle, one would have to teach an incredible amount of previous stuff. That is, one would have to step quite a large number of inferential steps. If I wanted to explain, for example, the theory of evolution, that would require a lot of anatomy, geography, zoology, botany, and even mathematics and philosophy. One would have to be a true polymath to achieve the feat. It's not that we don't know more about the universe than Aristotle, it is that to cross the inferential 'gap' between Aristotle and us would require an inordinate amount of knowledge.

Maybe a good metaphor is based on Dennett's crane idea: we develop ideas that help us reach higher levels of understanding, but as soon as we reach those upper levels we discard them to build new ones for higher levels. To help someone on the floor, one has to 'rebuild' these old cranes no longer in use.

See also: www.justfuckinggoogleit.com, www.lmgtfy.com, Reddit anti-"repost" rage and the comments like this that appear in practically every online community.

This is the reason it's a Bad Thing that so many of the deeper concepts of Mormonism have become public knowledge. The first question I get asked, upon revealing that I'm a Mormon, is often, "So, you believe that if you're good in this life, you'll get your own planet after you die?" There are at least three huge problems with this question, and buried deep beneath them, a tiny seed of truth. But I can't just say "The inferential distance is too great for me to immediately explain the answer to that question. Let me tell you about the Plan of Salvation, and we'll move from there," because that sounds like I'm Trying To Convert You, which is a Scary and a Bad Thing, because... out of explanations come brainwashing. Or something.

2 Nephi 28:30:

30 For behold, thus saith the Lord God: I will give unto the children of men line upon line, precept upon precept, here a little and there a little; and blessed are those who hearken unto my precepts, and lend an ear unto my counsel, for they shall learn wisdom; for unto him that receiveth I will give more...

This is one of those things that I realize is so obvious once I thought about it, but until it was pointed out to me, I would have never seen it.

To Mazur’s consternation, the simple test of conceptual understanding showed that his students had not grasped the basic ideas of his physics course: two-thirds of them were modern Aristotelians...“I said, ‘Why don’t you discuss it with each other?’” Immediately, the lecture hall was abuzz as 150 students started talking to each other in one-on-one conversations about the puzzling question. “It was complete chaos,” says Mazur. “But within three minutes, they had figured it out. That was very surprising to me—I had just spent 10 minutes trying to explain this. But the class said, ‘OK, We’ve got it, let’s move on.’...More important, a fellow student is more likely to reach them than Professor Mazur—and this is the crux of the method. You’re a student and you’ve only recently learned this, so you still know where you got hung up, because it’s not that long ago that you were hung up on that very same thing. Whereas Professor Mazur got hung up on this point when he was 17, and he no longer remembers how difficult it was back then. He has lost the ability to understand what a beginning learner faces.”

http://harvardmagazine.com/2012/03/twilight-of-the-lecture

If there is a probability of faulty inference, then longer inferences are exponentially less likely to be valid, with the length of the valid inference being proportional to logarithm of the process fidelity. Long handwaved inferences can have unbelievably low probability of correctness, and thus be incredibly weak as evidence.

Furthermore, informal arguments very often rely on 'i can't imagine an alternative' in multiple of their steps, and this itself has proven unreliable. It is also too easy to introduce, deliberately or otherwise, a huge number of implicit assumptions all of which must be true for the argument to be valid.

With logarithmic dependence to the fidelity of argument, even dramatically more reliable informal arguments do not lead to dramatically longer inference chains that can be done. One has to use formal methods to produce long inference chains.

The lack of this knowledge got me a nice big "most condescending statement of the day award" in lab a year ago.

I don't think this is quite right, but taking up the challenge may be helpful when writing:

I have always figured that if I can't explain something I'm doing t oa group of bright undergraduates, I don't really understand it myself, and that challenge has shaped everything I have written.

Daniel Dennett

And if you think you can explain the concept of "systematically underestimated inferential distances" briefly, in just a few words, I've got some sad news for you...

"I know [evolution] sounds crazy -- it didn't make sense to me at first either. I can explain how it works if you're curious, but it will take me a long time, because it's a complicated idea with lots of moving parts that you probably haven't seen before. Sometimes even simple questions like 'where did the first humans come from?' turn out to have complicated answers."

And if you think you can explain the concept of "systematically underestimated inferential distances" briefly, in just a few words, I've got some sad news for you...

"This is going to take a while to explain."

Did I do it? Did I win rationalism?!

Expecting short Inferential Distances wouldn't that be a case of rational thought producing beliefs which are themselves evidence? :P Manifested in over-explaining to the point of cognitive-dissonance? How about applying Occam's Razor and going the shorter distance: improve the clarity of the source by means of symbolism though a reflective correction (as if to compensate for the distortion in the other lens). To me it means to steel-man the opponent's argument to the point where it becomes not-falisifible. See, the fact that science works by falsification and pseudoscience by verification puts them in different paradigms that will only be reconciled by verification alone. Meaning also, science will have value because it can predict, so who cares about its inner workings of reason! This to me makes sense, because right now we seem to rank our intelligence superior to that of a virus, which is a problem of underestimating your enemy :). We are Neurotribes, autistic kids for ex. think in pictures; a different type intelligence may be emerging, may be without beliefs :)

"It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them. Operations of thought are like cavalry charges in a battle — they are strictly limited in number, they require fresh horses, and must only be made at decisive moments." Alfred North Whitehead

that last sentence ha

    I think this concept may be fundamental in explaining the mecanics of cohesion and division in society. This could help understand why politics tend to get more and more divided. Especially on the internet, but also IRL, people tend to confirm their ideas rather than confront them to different ones, as first observed by C. Wason (Peter C. Wason, « On the failure to eliminate hypotheses in a conceptual task », in Quarterly Journal of Experimental Psychology, 1960), and confirmed since. Or, one could argue, that reinforcing one's ideas (the ideas of a person or a community), building a shield of protective arguments around them, rather these arguments are solid and rationnal or just every means possible to deter ennemy attacks, is comparable to building evermore steps further away from people attacking these ideas, trying to change or oppose them.

    When it comes to people who have radicalized themselves to the point that they refuse to accept reality, in the sense of something we all agree to build from (global warming is real and manmade, there no such thing as "races" in humanity, if it's raining it is not not raining etc.), one could say they have build not inferential steps but an infertential wall. Conspiracy theorists for instance, or at least some of them, share some mecanims with people suffering from schizophrenia, in the sense that they will actually take arguments against their position as reinforcing this very position that is being contradicted, thinking that counterarguments are just trying to prevent them from discovering the truth, or that all people disagreeing with them are part of the conspiracy. It is as if they are building new steps when one tries to cross the already existing ones.

    Nevertheless, this grim presentation of mine shouldn't undermine the one great thing about this concept of inferential distance, that is, if this distance can be divided into steps, we can all reach for each other, even people that are staircases away. 

It's been nearly a century since relativity and quantum physics took the stage, and we have gathered considerably more data, and filled in a great many areas since then, yet we are still waiting for the next big breakthrough.

The problem may be in the way we approach scientific discovery.

Currently, when a new hypothesis is advanced, it is carefully considered as to how well it stands on its own and whether, eventually, it is worthy of being adopted as a recognized theory and becoming a part of our overall model of how everything works.

This might be the problem. Suppose the next advance cannot be made in this fashion? By this I am proposing that the next breakthrough may involve not a single new concept, that can be tested independently for worthiness, but several that cannot be tested individually.

For example, when you build a stone wall, you can test its strength and stability with each new stone placed. When you build a stone arch, attempting to test its strength and stability after each piece is going to get you labeled as an incompetent as the uppermost pieces will always fall if you merely try to set them in place, one at a time. Insisting that there are several pieces that must be placed at once before it can be tested is necessary, yet in theoretical physics, it well get you labeled as a crackpot.

For example, suppose one were to approach a transportation company with a radical new idea on how to improve airplanes, and even air travel in general. The company would want to test it and validate the concept before adoption. But suppose you told them that the idea could not even be evaluated unless it simultaneously includes the research and development of radical new ideas in such seemingly unrelated fields as personnel management, inventory control, and submarine transports. Chances are they would politely (or perhaps not politely) decline any further involvement.

Yet that is precisely the problem with the Standard Model.

We have conundrums in explaining consciousness, the dual-slit experiment, Schrodinger's cat, the number of spatial dimensions required, the expansion of the universe, the acceleration of the expansion of the universe, dark matter, dark energy, quantum uncertainty, the speed of light, singularities, the Big Bang, the heat-death (or Big Chill) of the universe, the Big Rip, gravity, entropy, and the list continues. Anyone that attempts to address more than one or two of these things at one time is likely to be dismissed at once as a crackpot. 

Yet is fairly well believed that Einstein's classic papers, submitted in today's climate, would go straight to the crackpot slush pile.

There is also the problem that, for proposed idea in physics to be given a hearing of any sort almost invariably requires advanced degree work in physics, with appropriately degreed instructors and sponsors. A paid position in the field is very nearly a prerequisite as well. Further, given the preoccupation with string theory that has consumed so many of them, and so restricted the opportunities of those who are not adherents . . . I've heard there may be less than 200 individuals in the world employed as theoretical physicists that are not dedicated to string theory (which doesn't seem to yield useful results in terms of advancing or redefining the Standard Model).

Additionally, given that they all go through a similar process to become recognized theoretical physicists, it almost certainly channels and colors their thinking on the subject, which is the same thing as saying that it limits them.