Take heed, for it is a trap

To everyone who just read this and is about to argue with the specific details of the bullet points or the mock argument:

Don't bother, they're (hopefully) not really the point of this.

Focus on the conclusion and the point that LW beliefs have a large inferential distance. The summary of this post which is interesting to talk about is "some (maybe most) LW beliefs will appear to be crackpot beliefs to the general public" and "you can't actually explain them in a short conversation in person because the inferential distance is too large". Therefore, we should be very careful to not get into situations where we might need to explain things in short conversations in person.

The way to have these conversations is to try to keep them as narrow as possible. You're not trying to explain your worldview, you're just trying to take the other person one step forward in inferential distance. There should be one point that you're trying to make that you want the other person to take away from the conversation, and you should try to make that point as clearly and simply as possible, in a way that will be understandable to the other person. Maybe you can give them a glimpse that there's more to your thinking than just this one point, but only if it doesn't distract from that point.

Bob doesn't do this. He feels that he needs to explain the nature of evidence, he uses an example which is controversial to Rian (and thus is a distraction from the point that Bob is trying to establish with the example), and he responds to every issue that Rian brings up instead of trying to bring the conversation back to the original point. Bob's problem is not that he has particularly unusual or crazy beliefs, it's that he has various views that are different from Rian's and he lets the conversation bounce from one to another without ever sticking with one point of disagreement long enough to get a clear explanation of his views across.

you are likely to agree with the majority of these statements

If you hadn't amplified the oddness of the beliefs on the list, this would be true. The trouble is, the way you amplified oddness is mostly by changing the substance of what was communicated, not just the style. Like by using over-general words so that people will hear one connotation when you might have been trying to say another. And so, why should we agree with statements that say the wrong thing?

Starting out with an incorrect guess about the reader is really bad for the rest of the post. You should start with your message instead, maybe even use personal experience - "I've had conversations where I brought up beliefs spread on LW, and people thought I was a crackpot."

But I also disagree with the thesis that the solution is to try to "hide the crazy." Bayesian Bob doesn't break things down into small enough parts and tries to use too many "impressive" statements that are actually harmful to communication. So a first action might be to stop digging himself deeper, but ultimately I think Bob should try to get better at explaining.

If you fail to conceal your low-status beliefs you'll be punished for it socially.

This shows a lack of understanding of signaling theory.

A poor kid wears middle class clothes so that people will think they're middle class and not poor. A middle class person wears rich clothes so that people will think they're rich and not middle class. A rich person wears whatever they want, because middle class people are already wearing 'rich' clothes and nobody's going to confuse them for being poor while they're matching ripped jeans with Rolex watches. If you and your beliefs are already low status, then having 'crackpot' beliefs will push your status lower. If you are already high status, then eccentric beliefs will increase your status. At the highest levels of status, people will automatically and unconsciously update their beliefs toward yours.

Your story sounds like Ryan is much higher status than Bob. Ryan's got kung-fu master level rationality skills versus low level Bayesian judo. Ryan also sounds more articulate and intelligent than Bob, although that might be the halo effect talking since we already established he's higher status. Bob is outgunned on every level and isn't smart enough to extricate himself, so of course he's going to be punished for it socially. It could have been an argument between any two ideological positions and Bob would have lost.

It says nothing about how most of us on Less Wrong should display our beliefs.

I classed this as a 'why commonly held LW beliefs are wrong' post when I first saw the list, then skipped to the conclusion (which made a really useful point, for which I upvoted the post.) I'm mentioning this because I think that the post would communicate better if you revealed your main point earlier.

The conversation between Rational Rian and Bayesian Bob is uncannily reminiscent of several conversations I had when I first grew infatuated with some of EY's writings and Lesswrong overall. This later led me to very quickly start wondering if the community would be willing to dedicated some intellectual effort and apply rationality to hiding bad signalling.

I think the OP is worth posting in the main section. But someone should write up something, about how to raise the sanity waterline without damaging your own reputation after that. Now I know when people call on someone to do something, this more or less means no one especially not me. This is why I've been doing my own thinking on the matter, but I'd first like to know if people on LW are interested at all in this line of thought.

For an example: A basic stratagem seems to be to successfully diagnose, perhaps even affirm, some of your acquaintances beliefs then over time present some simple and powerful, if perhaps by now obsolete or superseded arguments that first started several of LW's more prominent writers (or yourself) on a path to the current set of beliefs. This naturally isn't rationality building (though it might happen in the process), just spreading beliefs, but the objective here is to change the in group norms of your social circle.

Then, you can start individually building rationality skills.

It is not nearly as bad as you make it out. Bayesian Bob just seems really bad at explaining.

Rian seems to not consider detectives investigating a crime to be gathering evidence, but Bob does not seem to notice this. We can come up with examples of socially categorized types of evidence and explain why the categories are socially useful.

Absence of Evidence is Evidence of Absence can be explained in scientific terms. If a scientific experiment looking for evidence of a theory produces no results, that is evidence against the theory. This is easier to deal with in a scientific experiment because its controlled nature allows you to know how hard it was looking for evidence, to calculate how likely it would be to find the evidence if the theory were correct. Outside the context, the principle is harder to apply because the conditional probability is harder to calculate, but it is still valid.

Not once did Bob bring up such concepts as likelihood ratios or conditional probability.

And plenty of other comments have noted the problem with "starts out with a 50% probability".

As has also been pointed out already, most of the bullet point statements are either not actually controversial, or distorted from the idea they refer to. In particular, though probability theory does not perfectly align with the scientific method, it does explain how the scientific method has been as successful as it is.

I myself have discussed LW ideas with people from skeptics and atheist groups, and not come off as a crackpot.

A statement, any statement, starts out with a 50% probability of being true, and then you adjust that percentage based on the evidence you come into contact with.

Zed, you have earned an upvote (and several more mental ones) from me for this display of understanding on a level of abstraction even beyond what some LW readers are comfortable with, as witnessed by other comments. How prescient indeed was Bayesian Bob's remark:

(I shouldn't have said that 50% part. There's no way that's going to go over well. I'm such an idiot.)

You can be assured that poor Rational Rian has no chance when even Less Wrong has trouble!

But yes, this is of course completely correct. 50% is the probability of total ignorance -- including ignorance of how many possibilities are in the hypothesis space. Probability measures how much information you have, and 50% represents a "score" of zero. (How do you calculate the "score", you ask? It's the logarithm of the odds ratio. Why should that be chosen as the score? Because it makes updating additive: when you see evidence, you update your score by adding to it the number of bits of evidence you see.)

Of course, we almost never reach this level of ignorance in practice, which makes this the type of abstract academic point that people all-too-characteristically have trouble with. The step of calculating the complexity of a hypothesis seems "automatic", so much so that it's easy to forget that there is a step there.

S -> statements
P -> propositions
N -> non-propositional statements
T -> true propositions
F -> false propositions

I would be much happier with that survey if it used the standard five-degrees-of-belief format rather than a flat agree/disagree. Especially later on, it includes many statements which I believe or disbelieve with low confidence, or which I consider irrelevant or so malformed as to be essentially meaningless.

Some of those statements in the list are sufficiently unclear that I can't really agree or disagree with them. Others have multiple different claims in them and I agree with some parts and disagree with others. And some are just false.

Most scientists disagree with this but that's just because it sounds counter-intuitive and scientists are biased against counterintuitive explanations.

This one is false, as some other comments have pointed out.

Besides, the scientific method is wrong because it is in conflict with probability theory. Oh, and probability is created by humans, it doesn't exist in the universe.

(Bayesian) Probability theory doesn't say that the scientific method is wrong. It provides a formal specification of why the scientific method (of changing beliefs based on evidence) is correct and how to apply it. The second sentence refers to the true beliefs explained in Probability is in the Mind and Probability is Subjectively Objective, but it mangles them.

Every fraction of a second you split into thousands of copies of yourself. Of course you cannot detect these copies scientifically, but that because science is wrong and stupid.

"Science is wrong and stupid" is just false. It's more like, you can't detect these copies directly but they are implied by math that has been supported by experiment. Unless you want to claim that theoretical physics is unscientific, you have to accept that using math to find out facts about the physical world is possible.

And finally, the truth of all these statements is completely obvious to those who take the time to study the underlying arguments. People who disagree are just dumb, irrational, miseducated or a combination thereof.

This exaggerates the simple (tautological?) true statement that "I have enough evidence to convince a rational person of all the above" until it is not true.

I learned this all from this website by these guys who want us to give them our money.

This is also a misrepresentation. Some of the guys on the website work for a nonprofit and want you to give money to fund their research, which they believe will save many lives. Or if you don't want to do that, they want you give money to some other charity that you believe will save the most possible lives. A majority of the content producers don't ask for money but many of them do give it.

argument ad populum

I think a more correct term in this context would be argumentum ad verecundiam. It's about arguing based on the opinion of a small number of authoritative people, not the general public.

If you have worked your way through most of the sequences you are likely to agree with the majority of these statements

I realize this is not the main point of the post, but this statement made me curious: what fraction of Less Wrong readers become convinced of these less mainstream beliefs?

To this end I made a Google survey! If you have some spare time, please fill it out. (Obviously, we should overlook the deliberately provocative phrasing when answering).

I'll come back two weeks from now and post a new comment with the results.

It is possible to run a person on Conways Game of Life. This would be a person as real as you or me, and wouldn't be able to tell he's in a virtual world because it looks exactly like ours.

I don't see why you call this a "crackpot belief". The ( extended ) Church-Turing Thesis has near-universal acceptance and implies that humans can be simulated by turing machines. Similarly, it is widely accepted that Conways Game of Life can run turing machines . Physicists who don't believe this are widely regarded as controversial.

I think you should have introduced your point much earlier on (perhaps at the beginning).

This reminds me of some old OB posts, I think, on non-conformity - the upshot being that you can't get away with being public on all the ways you are a maverick and to do so is self-sabotaging.

I think the Sequences paint a picture of scientists on Many Worlds that is just wrong. Sure, if you count all scientists. But if you just look at the ones whose opinions matter: 58 percent think its true. Eighteen percent disagree.

If you have worked your way through most of the sequences you are likely to agree with the majority of these statements:

I have, but I don't. A couple I agree with and there are some others about which I can at least see how they could be used as a straw man. Then there are some which are just way off.

Then there is:

It is possible to run a person on Conways Game of Life. This would be a person as real as you or me, and wouldn't be able to tell he's in a virtual world because it looks exactly like ours.

That I agree with. And it doesn't qualify as a 'crackpot belief'.

The moral here is that crackpot beliefs are low status

You probably should have opened with that. It's true and basically universally accepted here already.

Concealing unconventional beliefs with high inferential distance to those you are speaking with makes sense. Dismissing those beliefs with the absurdity heuristic does not.

Also, I think you underestimate the utility of rhetorical strategies. For example, you could:

• Talk about these weird beliefs in a hypothetical, facetious manner (or claim you had been).
• Close the inferential difference gradually using the Socratic method.
• Introduce them to the belief indirectly. For example, you could link them to a more conventional LessWrong sequence post and let them investigate the others on their own.
• Ask them for help finding what is objectively and specifically wrong with the weird belief.

Most of the statements you make are false in their connotations, but there's one statement you make (and attribute to "Bayesian Bob") that seems false no matter what way you look at it, and it's this one: "A statement, any statement, starts out with a 50% probability of being true" Even the rephrasing "in a vacuum we should believe it with 50% certainty" still seems simply wrong. Where in the world did you see that in Bayesian theory?

For saying that, I label you a Level-0 Rationalist. Unless someone's talking about binary digits of Pi, they should generally remove the concept of "50% probability" from their minds altogether.

A statement, any statement, starts out with a probability that's based on its complexity, NOT with a 50/50 probability. "Alice is a banker" is a simpler statement than "Alice is a feminist banker who plays the piano.". That's why the former must be assigned greater probability than the latter.

I think Bayesian Bob should just get better at arguing. It's the same thing I tell students when they complain that they can't explain their paper properly in a 4 sentence abstract: The number of possible sentences you might write is very very large. You're going to have to work a lot harder before I'm convinced that no sequence of 4 sentences will suffice.

My experience has been that if I'm arguing about something I know well and I'm very confident about, it never feels like I'm in danger of "losing status".

"Bayesian Bob: ... I meant that in a vacuum we should believe it with 50% certainty..."

No we shouldn't: http://lesswrong.com/lw/jp/occams_razor/

As for proving a negative, I've got two words: Modus Tollens.

Bob does need to go back to math class! ;)

A statement, any statement, starts out with a 50% probability of being true, and then you adjust that percentage based on the evidence you come into contact with.

Suppose we have a statement X, and the only thing we know about X is that it was randomly selected from the set S of statements of 100 characters, with the alphabet consisting of the digits 0 to 9 and the symbols + and =. If

• A = 'X is true'
• B = 'X was randomly selected from the set S of statements of 100 characters, with the alphabet consisting of the digits 0 to 9 and the symbols + and ='

then P(A|B) can be straightforwardly computed by enumerating the set S and checking how many true statements it contains (or some cleverer variation of this). The above quote, on the other hand, suggests that we start with P(A)=0.5, and then... do what? By Bayes' Theorem, P(A|B) = P(A)*P(B|A)/P(B), but it's hard to see how that helps.

(In case I've chosen a pathological example, what is a good example of starting with .5 probability of a statement being true, and then adjusting that?)

A statement, any statement, starts out with a 50% probability of being true, and then you adjust that percentage based on the evidence you come into contact with.

That's wildly wrong. "50% probability" is what you assign if someone tells you, "One and only one of the statements X or Y is true, but I'm not going to give you the slightest hint as to what they mean" and it's questionable whether you can even call that a statement, since you can't say anything about its truth-conditions.

Any statement for which you have the faintest idea of its truth conditions will be specified in sufficient detail that you can count the bits, or count the symbols, and that's where the rough measure of prior probability starts - not at 50%. 50% is where you start if you start with 1 bit. If you start with 0 bits the problem is just underspecified.

Update a bit in this direction: That part where Rational Rian said "What the hell do you mean, it starts with 50% probability", he was perfectly right. If you're not confident of your ability to wield the math, don't be so quick to distrust your intuitive side!

I don't think it's that bad. Anything at an inferential distance sounds ridiculous is you just matter-of-factly assert it, but that just means that if you want to tell someone about something at an inferential distance don't just matter-of-factly assert it. The framing probably matters at least as much as the content.

science is wrong

No. Something like "Bayesian reasoning is better than science" would work.

Every fraction of a second you split into thousands of copies of yourself.

Not "thousands". "Astronomically many" would work.

Computers will soon become so fast that AI researchers will be able to create an artificial intelligence that's smarter than any human

That's the accelerating change, not the intelligence explosion school of singularity. Only the latter is popular around here.

Also, we sometimes prefer torture to dust-specs.

Add "for sufficiently many dust-specks".

I also agree with lessdazed's first three criticisms.

--

Other than these, it's not a half-bad summary!

the scientific method is wrong

Is your refrigerator running?

Time isn't real.

Rainbows are real.

Computers will soon become so fast that AI researchers will be able to create an artificial intelligence that's smarter than any human.

Not Kurzweil...anything but that

When this happens humanity will probably be wiped out.

Marxist-style fatalism? Worse than Kurzweil.

truth of all these statements is completely obvious

Obviousness isn't a property of statements or facts. This idiom of thought embodies the mind projection fallacy.

Just because I read the sequences doesn't mean I'm particularly likely to agree with any of them. Some, yes, but not all. Many of the statements you listed are controversial even on LW. If they were unanimously accepted here without further discussion, it would be a worrying sign.

The first part of this post reminded me of Stranger Than History.

I wonder what should Friendly AI do, when it discovers something that at first sight seems like a "crackpot belief" to its human operators. Let's assume that the AI is far smarter than humans (and the "crackpot belief" requires many logical steps), but is still in a testing phase and humans don't believe in its correctness.

If AI tells the discovery openly to humans, they will probably turn it off quickly, assuming there was something wrong in a program.

On the other hand, if the AI predicts that humans are not ready for this information, and tries to hide it, a security subroutine will detect that "AI wants to cheat its masters" and will force a shutdown. Even worse, if the AI decides that the right thing is telling the information to humans, but not right now, and instead give them first some "sequences" that will prepare them to accept the new information, and only give them the new information when they have changed their thinking... the security subroutine might still evaluate this as "AI wants to manipulate its masters" and force a shutdown.

Next question is, what would humans do. I am afraid that after receiving an unbelievable information X, they might simply add "not X" into AI axioms or values. It might seem like the rational thing to do; they will not think: "we don't like X", but rather: "X is a cognitive error, possibly one that AIs are prone to, so we should protect our AI against this cognitive error".

As an example, imagine a world before the quantum physics was discovered; and imagine that AI discovered quantum physics and multiple universes -- and gave this all info together to the unprepared humans. Now imagine some new discovery in future, possibly one hundred times less understandable to humans, with even more shocking consequences.

Think back for a second to your pre-bayesian days. Think back to the time before your exposure to the sequences. Now the question is, what estimate would you have given that any chain of arguments could persuade you the statements above are true? In my case, it would be near zero.

p(convincing argument exists) >= (Number of Bayesians post-sequence - Number of Bayesians pre-sequence) / Number of people reading sequence.

Or, in simpler terms, "the sequences are a convincing chain of argument, and they are effective." I'll admit I'm working with a smart group of people, but none of my friends have had trouble with any of the inferential steps in the sequences (if I jump in to the advanced stuff, I'll still lose them just fine thanks to the miracle of inferential distances, obviously, and I haven't convinced all my friends of all those points yet :))

Are you Zed Shaw?

I can't handle this site's scrutiny right now.