Making your explicit reasoning trustworthy

Reading your post felt very weird to me, as if you were deliberately avoiding the obvious conclusion from your own examples! Do you really believe that people follow kosher or die in religious wars due to using abnormally explicit reasoning? The common thing about your examples is putting ideals over personal gain, not reasoning over instinct. Too much acting on explicitly stated values, not explicitly stated beliefs. In truth, using rationality for personal gain isn't nearly as dangerous as idealism/altruism and doesn't seem to require the precautions you go on to describe. If any of the crazy things I do failed to help me, I'd just stop doing them.

Which prompts a question to everyone: what crazy things do you do that help you? (Rather than help save the light cone or something.)

A quote from the linked-to "cautions for Christians against clever arguments”, to save others the pain of wading through it to figure out what it's talking about:

It always begins the same way. They swallow first the rather subtle line that it is necessary for each to think for himself, to judge everything by the light of whether it appears reasonable to him. There is never any examination of that basic premise, though what it is really saying is that the mind of man becomes the ultimate test, the ultimate authority of all life. It is necessary for man to reason and it is necessary for him to think for himself and to examine things. But we are creatures under God, and we never can examine accurately or rightly until we begin with the basic recognition that all of man's thinking, blinded and shadowed as it is with the confusion of sin, must be measured by the Word of God. There is the ultimate authority.

Thanks for a ton of great tips Anna, just wanted to nit pick on one:

Remember that if reading X-ist books will predictably move your beliefs toward X, and you know there are X-ist books out there, you should move your beliefs toward X already. Remember the Conservation of Expected Evidence more generally.

I suspect that reading enough X-ist books will affect my beliefs for any X (well, nearly any). The key word is enough -- I suspect that fully immersing myself in just about any subject, and surround myself entirely by people who advocate it, would significantly alter my beliefs, regardless of the validity of X.

Anna - I'm favorably impressed by this posting! Thanks for making it. It makes me feel a lot better about what SIAI staff mean by rationality.

In the past I've had concerns that SIAI's focus on a future intelligence explosion may be born of explicit reasoning that's nuts (in the sense of your article), and the present posting does a fair amount to assuage my concerns - I see it as a strong indicator that you and some of the other SIAI staff are vigilant against the dangers of untrustworthy explicit reasoning.

Give Michael my regards.

If you can predict what you'll believe a few years from now, consider believing that already.

I've been thinking about this lately. Specifically, I've been considering the following question:

If you were somehow obliged to pick which of your current beliefs you'd disagree with in eight years time, with real and serious consequences for picking correctly or incorrectly, what criteria would you use to pick them?

I'm pretty sure that difficulty in answering this question is a good sign.

I think it is a bit unfair to frame arguments to trust outside views or established experts as arguments to not think about things. Rather, they are arguments about how much one should trust inside views or your own thoughts relative to other sources.

Thanks for posting this, it's awesome.

I particularly endorse trying to build things out of your abstract reasoning, as a way of moving knowledge from "head-knowledge" to "fingers-knowledge".

Regarding this sentence: "Remember that if reading X-ist books will predictably move your beliefs toward X, and you know there are X-ist books out there, you should move your beliefs toward X already."

Since I'm irrational (memetic insecure) and persuasive deceptions (memetic rootkits) exist, the sentence needs some qualifier. Maybe: "If you believe that the balance of the unknown arguments favor believing X, then you have reason to believe X."

Make every link in a chain of argument explicit. Most of the weirder conclusions I have seen in my own and others' beliefs have come about because they had conflated several different lines of reasoning or have jumped over several steps that appeared "obvious" but that included a mistaken assumption, but were never noticed because they weren't spelled out explicitly.

Also, be very careful about not confusing different meanings of a word, sometimes these can be very subtle so you need to be watchful.

For actually reasoning with an argument, keep it schematic. One of the reasons reading philosophy is so hard, is that it is written in prose. For any but the simplest arguments, though, you need to convert it to schematic form before you can actually reason about it effectively. Like trying to do mathematics or play music from a written description (though not quite that extreme), it just doesn't work well.

There is a much simpler way of winning than carefully building up your abstract-reasoning ability to the point where it produces usefully accurate, unbiased, well-calibrated probability distributions over relevant outcome spaces.

The simpler way is just to recognize that, as a human in a western society, you won't lose much more or win much more than the other humans around you. So you may as well dump the abstract reasoning and rationality, and pick some humans who seem to live relatively non-awful lives (e.g. your colleagues/classmates) and take whatever actions they take. Believe what they believe, even if it seems irrational. Do what they do.

Careful probability estimation and actions taken based upon anticipations of consequences is the kind of cognitive algorithm befitting a lone agent who actually reaps what (s)he sows. For a human, herd-mentality seems to be the more elegant solution: elegant in the sense that the epistemology is hard to get right, but there is a robust argument about consequences and utilities: almost all of the relatively-average-strategy humans in the herd will get roughly the same deal out of life.

Research from hedonic psychology on the "Hedonic Treadmill" effect backs this up further: even if you make more (or less) money than average, you probably won't actually be happier or better (worse) off.

Of course there are details and complications: which subgroup of humans do you join? How do you make the tradeoff between different subcultures etc. But still, you don't even need a general solution to that problem, you only need to decide which of the handful of specific subcultures available to you seems best for you.

And, of course, it goes without saying that this strategy is useless for someone who is determined to invest emotionally in a nonstandard life-narrative, like utilitarian charity or life-extension. From this point of view, one might object that joining the herd is selfish in the sense that it isn't the action which maximizes utility across the herd; but then again most people don't have a utilitarian concept of selfishness and don't count benefit to random strangers as part of their actual near-mode, actionable goal set, so from their axiological point of view, herding is an acceptable solution.

The main problem I see with this post is that it assumes that it's always advantageous to find out the truth and update one's beliefs towards greater factual and logical accuracy. Supposedly, the only danger of questioning things too much is that attempts to do so might malfunction and instead move one towards potentially dangerous false beliefs (which I assume is meant by the epithets such as "nutty" and "crazy").

Yet I find this assumption entirely unwarranted. The benefits of holding false beliefs can be greater than the costs. This typically happens when certain false beliefs have high positive signaling value, but don't imply any highly costly or dangerous behavior. Questioning and correcting such beliefs can incur far more cost than benefit; one can try to continue feigning them, but for most people it will be at least somewhat difficult and unpleasant. There are also many situations where the discovery of truth can make one's life miserable for purely personal reasons, and it's in the best interest of one's happiness to avoid snooping and questioning things too much.

It seems to me that the problem for uncompromising truth-seekers is not just how to avoid invalid reasoning leading to crazy false beliefs, but also how to avoid forming true beliefs that will have negative signaling consequences or drastically reduce one's happiness. Now, maybe you would argue that one should always strive for truth no matter what, but this requires a separate argument in addition to what's presented in the above post -- which is by itself insufficient to address the reasons for why people are "afraid to think fully about certain subjects."

Propositional calculus is brittle. A contradiction implies everything.

In Set theory, logic and their limitations Machover calls this the Inconsistency Effect. I'm surprised to find that this doesn't work well as a search term. Hunting I find:

One page

In classical logic, a contradiction is always absurd: a contradiction implies everything.

Another page

Another trouble is that the logical conditional is such that P AND ¬P ⇒ Q, regardless of what Q is taken to mean. That is, a contradiction implies that absolutely everything is true.

Any false fact that you believe acts as a logic bomb. Once you come across the true fact, the combination permits you to construct a logical argument to reach any conclusion. This is just what the monsters from the Id have been waiting for.

The inconsistency effect implies that fallible creatures dare not rely on pure logic.

I wouldn't say that this is a fear of an "inaccurate conclusion," as you say. Instead, it's a fear of losing control and becoming disoriented: "losing your bearings" as you said . You're afraid that your most trustworthy asset - your ability to reason through a problem and come out safe on the other side; an asset that should never fail you - will fail you and lead you down a path you don't want to go. In fact, it could lead to Game Over if you let that lead you to kill or be killed, as you highlight in your examples of the Unabomber, Mitchell Heisman and zealot soldiers.

I especially like the orientation metaphor here. And I think that your piece addresses this. First, you need to know where you are. Recognize when you are in far mode and thinking abstractly and when you are in near mode and thinking concretely. Then you can think about where you should be, near or far. Learn to recognize which one is better for your current situation and be able to switch between them. This is also part of being oriented. Finally, have a kill switch if you feel yourself losing control.

For example, a Christian might give away all their possessions, rejoice at the death of their children in circumstances where they seem likely to have gone to heaven, and generally treat their chances of Heaven vs Hell as their top priority.

Steven Landsburg used this reasoning, combined with the fact that Christians don't generally do this, to conclude not that Christians don't act on their beliefs, but that Christians don't generally believe what they claim to believe. I think the different conclusion is reached because he assigns a lot more rationality to people than you do. But certainly there are, for some people, very strong incentives against admitting that you've stopped believing in God.

I, too, really appreciated this post.

Unfortunately, though, I think that you missed one of the most important skills for safer reasoning -- recognizing and acknowledging assumptions (and double-checking that they are still valid). Many of the most dangerous failures of reasoning occur when a normally safe assumption is carried over to conditions where it is incorrect. Diving three feet into water that is unobstructed and at least five feet deep won't lead to a broken neck -- unless the temperature is below zero centigrade.

I like this; it seems practical and realistic. As a point of housekeeping, double-check the spaces around your links--some of them got lost somewhere. :)

Posted on behalf of someone else who had the following comment:

I would have liked for [this post] to contain details about how to actually do this:

If you're uncomfortable not knowing, so that you find yourself grasping for one framework after another, build your tolerance for ambiguity, complexity, and unknowns.

People fear questioning their “don't walk alone at night” safety strategy, lest they venture into danger.

I routinely walk (and run) alone at night. Indeed, I plan on going for a 40k run/walk alone tonight. Yet I observe that walking alone at night does really seem like it involves danger - particularly if you are an attractive female.

I actually know people (ok, so I am using my sisters as anecdotes) who are more likely to fear considering a "don't walk alone at night" strategy because it may mean they would have to sacrifice their exercise routine. Fortunately Melbourne is a relatively safe city as far as 'cities in the world' go.

I love this post, but I think "we can walk into rationality without fear" is too strong.

I'd just like to point out that 5 looks like a specific application of 1. Recognizing that your "goal" is just what you think is your goal, and you can be mistaken about it in many ways.

Minor typo -"denotationally honest, including refusing to jobs that required a government loyalty oath" - no need for "to" before "jobs".