Belief in Belief vs. Internalization

Is there an underlying problem

I don't know either, but I do know that an internalised, correct understanding of pressure and one of its measures 'pounds per square inch' would be a sufficient condition to save his life. The parable of the pressure vessel seems to be a case of an incorrect belief (one pound isn't much), whereas the parable of the invisible dragon seems to be a case of a correct belief (invisible dragon in my garage) that hasn't been internalised, and so has not produced beliefs it ought to (invisible DRAGON IN MY GARAGE!)

Is there an underlying problem of crying wolf; too many warning messages obscure the ones that are really matters of life and death?

This is certainly an enormous problem for interface design in general for many systems where there is some element of danger. The classic "needle tipping into the red" is an old and brilliant solution for some kinds of gauges - an analogue meter where you can see the reading tipping toward a brightly marked "danger zone", usually with a 'safe' zone and an intermediate zone also marked, has surely prevented many accidents. If the pressure gauge on the door had such a meter where green meant "safe to open hatches" and red meant "terribly dangerous", that might have been a better design than just raw numbers.

I haven't worked with pressure doors but I have worked with large vacuum systems, cryogenic systems, labs with lasers that could blind you or x-ray machines that can be dangerously intense, and so on. I can attest that the designers of physics lab equipment do indeed put a good deal of thought and effort into various displays that indicate when the equipment is in a dangerous state.

However, when there are /many/ things that can go dangerously wrong, it becomes very difficult to avoid cluttering the sensorium of the operator with various warnings. The classic example are the control panels for vehicles like airplanes or space ships; you can see a beautiful illustration of the 'indicator clutter problem' in the movie "Airplane!":

I'm not seeing how the parable helps one be less wrong in real life. In the parable the victim has seen a dog taken by the dragon.

I must admit I up-voted the post mostly because I thought the parable was funny.

The gauge says one pound per square inch. One pound doesn't sound like much so the man undoes the latches. [...] I'm not seeing how the parable helps one be less wrong in real life.

Easy - not using metric system will kill you.

The post was inspired by a comment which I felt confused lack of internalization with belief in belief. On reflection, I probably didn't establish the connection sufficiently.

This is because of spill-over from 'religious tolerance'. Most people will feel uncomfortable mocking a ridiculous belief; they have the "everyone is entitled to their own opinion" meme in mind. This makes people disagree with other beliefs much less than they ought to.

People are much more comfortable mocking ridiculous actions (because everyone is not entitled to their own facts) - which is why evangelists are scorned where the average religious person wouldn't be, despite evangelists acting consistently and the average religious person acting inconsistently on beliefs that the mocker doesn't disagree with.

I think the people who wrote the entry probably do disagree with Eliezer's beliefs in this regard. They seem to be mocking his beliefs, not just the actions he takes based on them.

That's not to say that there's any shortage of people who do take issue with, or even outright mock people, for acting on beliefs they do not disagree with.

I suspect (*) the principle is: sincere stupidity is no less stupid than insincere stupidity.

It is important here to note that it's a silly wiki of no importance that doesn't pretend to be of any importance. (Many readers aren't happy with its obnoxiousness.) It just happens to be one of the few places on the Internet with a detailed article on LessWrong.

If this is considered a problem, the solution would be a publicity push for LessWrong to get it to sufficient third-party notability for a Wikipedia article or something. The question then is whether that would be good for the mission: "refining the art of human rationality." I'd suggest seeing how the influx of HP:MoR readers affects things. Small September before large one.

(*) "I suspect" as I'm not going to be so foolish as to claim the powers of a spokesman.

I always felt like this was unfair, because it amounts to attacking him for actually believing the things he talks about. That is, it's okay to talk about an immensely-powerful AI happening in the future, it's just not okay to act on that belief.

"That isn't" the belief being mocked.

The dragon didn't eat him before because it wasn't hungry. If there's a tiger in your garage, it probably won't attack you as soon as it sees you, but the longer you spend in its vicinity, the greater your chances of being mauled.

The footprints were dragon-shaped, and it preferentially targeted the types of dogs that dragons most like to eat.

Would they be safe? Probably. Would they enjoy the experience? Probably not...

A common illustration of this distinction (often told by preachers) is that Blondin the tightrope walker asked the crowd if they believed he could safely carry someone across the Niagra falls on a tightrope, and almost the whole crowd shouted 'yes'. Then he asked for a volunteer to become the first man ever so carried, at which point the crowd shut up. In the end the only person he could find to accept was his manager.

Which is, of course, followed by handing out buckets of stones and pointing out suitable targets of righteous retribution. Adulterers, people who eat beetles, anyone who missed the sermon...

I'm quite prepared to admit that there are many cases where my actions do not make sense in light of a gut level acceptance of my beliefs. I may not think donating money to a charity to support starving children is the highest utility use of my money, but even in spite of my own experiences with starvation, starving children are very much an invisible dragon to me.

There's a big gap between being a strong enough rationalist to acknowledge that one's actions don't make sense, and being a strong enough rationalist to reverse the situation, but at least the understanding can't hurt in making a start.

Somewhat related: at least one person has internalized their belief about the Singularity in a way that appears at least as weird as our hypothetical neighbor boarding up his garage and moving house.

I'm not so sure. Retirement plans are far, boarding up the garage is near.

Now, I wonder if anyone would take part in such a raffle if all the entrants who didn't win were killed on the spot. That would mean that everyone would win in some universe, and cease to exist in the other universes where they entered. Could that be a kind of intellectual assent vs belief test for Many Worlds?

No. No. No!

Quantum Sour-Grapes (ie. what you described) could be the result of a technically coherent value system but not a sane one. Unless there is some kind of physical or emotional torture involved dying doesn't make things better regardless of QM!

Could that be a kind of intellectual assent vs belief test for Many Worlds?

No, because it assumes you're indifferent to any effects you have on worlds that you don't personally get to experience.