Connecting Your Beliefs (a call for help)

Also keep in mind that it's more important to make your beliefs as correct as possible then to make them as consistent as possible. Of course the ultimate truth is both correct and consistent; however, it's perfectly possible to make your beliefs less correct by trying to make them more consistent. If you have two beliefs that do a decent job of modeling separate aspects of reality, it's probably a good idea to keep both around, even if they seem to contradict each other. For example, both General Relativity and Quantum Mechanics do a good job modeling (parts of) reality despite being inconsistent and we want to keep both of them. Now think about what happens when a similar situation arises in a field, e.g., biology, psychology, your personal life, where evidence is messier then it is in physics.

Something bothers me about this post. Querying my mind for "what bothers me about this post", the items that come up are:

• the post starts with a pretty picture of a "web of beliefs", but is itself a long string of words
• the picture is decorative, but certainly not illustrative; it is not a picture of your beliefs
• the post offers some beliefs that you claim are connected, but none of the reasoning behind the claims
• the post offers anecdotes, e.g. "successful businessmen may have this trait more often than others", but does not represent them as beliefs in a web of beliefs

The general theme appears to be "this post doesn't practice what it preaches".

If you did represent the beliefs you mention in this post in the form of a connected web, what would that look like?

(On a very few prior occasions I've tried such explicit representations, and was not motivated to keep using them.)

I hadn’t noticed that my worldview already implied intelligence explosion.

I'd like to see a post on that worldview. The possibility of an intelligence explosion seems to be an extraordinary belief. What evidence justified a prior strong enough as to be updated on a single paragraph, written in natural language, to the extent that you would afterwards devote your whole life to that possibility?

I’m not talking about the problem of free-floating beliefs that don’t control your anticipations. No, I’m talking about “proper” beliefs that require observation, can be updated by evidence, and pay rent in anticipated experiences.

How do you anticipate your beliefs to pay rent? What kind of evidence could possible convince you that an intelligence explosion is unlikely, how could your beliefs be surprised by data?

This is interesting to me in a sort of tangential way. It seems like studying philosophy exercises this tendency to propagate your beliefs in order to make them coherent. In fact logical belief propagation seems to embody a large aspect of traditional philosophy, so I would expect that on average someone who studies philosophy would have this tendency to a greater degree than someone who doesn't.

It would be interesting to me if anyone has seen any data related to this, because it feels intuitively true that studying philosophy changed my way of thinking, but it's of course difficult to pinpoint exactly how. This seems like a big part of it.

This post made me realize just how important it is to completely integrate the new things you learn.

I have been reading a lot of books and blogs on the subject of students that finish school with honors, but don't seem to work very hard while doing so. I also met one of those people in person (he finished an entire 4 year curriculum with honors in just 3 months and is now a professor of that content)

It all boils to the same thing: Whatever the textbook is trying to tell you, make sure you integrate that in your life. Only then will you see if you really understood what it was saying and if you are missing any extra information, or if the information in the book is wrong. Once integrated you do not need any extra studying to get an A/10 for the exam.(because you will have recursively updated all your beliefs to include the thing you where supposed to learn)

Some of these books and blogs go into detail on how to how to do this. One of the methods i read was making a doodle of the idea in your notebook. This doodle borrows heavily from your current state of knowledge. An example of what I did: To model the process of taking a raw resource and making it into a profitable end product i drew a mine with rocks coming out, then a table with a chisel on the rock and finally a diamond with a price-tag. I know how diamonds are made so i could use that to represent this process.

There are many more methods, another that i have not yet tried to use is basically making a flashcard.: Question/Evidence/Conclusion http://calnewport.com/blog/2009/04/06/4-weeks-to-a-40-streamline-your-notes/

EDIT: I'm having a hard time explaining what i am trying to say, i will post a new comment or top level post if i manage to figure it out. Basically I'm trying to say that there already well working and documented methods for connecting and updating beliefs in the world of outlier student research.

I spent a week looking for counterarguments, to check whether I was missing something

What did you find? Had you missed anything?

Part of my day job involves doing design reviews for alterations to relatively complex software systems, often systems that I don't actually understand all that well to begin with. Mostly, what I catch are the integration failures; places where an assumption in one module doesn't line up quite right with the end-to-end system flow, or where an expected capability isn't actually being supported by the system design, etc.

Which isn't quite the same thing as what you're talking about, but has some similarities; being able to push through from an understanding of each piece of the system to an understanding of the expected implications of that idea across the system as a whole.

But thinking about it now, I'm not really sure what that involves, certainly not at the 5-second level.

A few not-fully-formed thoughts: Don't get distracted by details. Look at each piece, figure out its center, draw lines from center to center. Develop a schematic understanding before I try to understand the system in its entirety. If it's too complicated to understand that way, back out and come up with a different way of decomposing the system into "pieces" that results in fewer pieces. Repeat until I have a big-picture skeleton view. Then go back to the beginning, and look at every detail, and for each detail explain how it connects to that skeleton. Not just understand, but explain: write it down, draw a picture, talk it through with someone. That includes initial requirements: explain what each requirement means in terms of that skeleton, and for each such requirement-thread find a matching design-thread.

So, of course someone's going to ask for a concrete example, and I can't think of how to be concrete without actually working through a design review in tedious detail, which I really don't feel like doing. So I recognize that the above isn't really all that useful in and of itself, but maybe it's a place to start.

This reminds me of a blog post by Jula Galef I read a couple weeks ago: The Penrose Triangle of Belief .

That's actually a good question. Let me rephrase it to something hopefully clearer:

Compartmentalization is an essential safety mechanism in the human mind; it prevents erroneous far mode beliefs (which we all adopt from time to time) from having disastrous consequences. A man believes he'll go to heaven when he dies. Suicide is prohibited in a patch for the obvious problem, but there's no requirement to make an all-out proactive effort to stay alive. Yet when he gets pneumonia, he gets a prescription for penicillin. Compartmentalization literally saves his life. In some cases many other lives, as we saw when it failed on 9/11.

Here we have a case study where a man of intelligence and goodwill redirected his entire life down a path of negative utility on the basis of reading a single paragraph of sloppy wishful thinking backed up by no evidence whatsoever. (The most straightforward refutation of that paragraph is that creating a machine with even a noteworthy fraction of human intelligence is far beyond the capacity of any human mind; the relevant comparison of such a machine if built would be with that which created it, which would have to be a symbiosis of humanity and its technology as a whole - with that symbiosis necessarily being much more advanced than anything we have today.) What went wrong?

The most obvious part of the answer is that this is an error to which we geeks are particularly prone. (Supporting data: terrorists are disproportionately likely to be trained in some branch of engineering.) Why? Well, we are used to dealing in domains where we can actually apply long chains of logic with success; particularly in the age range when we are old enough to have forgotten how fallible were our first attempts at such logic, yet young enough to be still optimists, it's an obvious trap to fall into.

Yet most geeks do actually manage to stay out of the trap. What else goes wrong?

It seems to me that there must be a parameter in the human mind for grasping the inertia of the world, for understanding at a gut level how much easier is concept than reality, that we can think in five minutes of ideas that the labor of a million people for a thousand years cannot realize. I suppose in some individuals this parameter must be turned up too high, and they fall too easily into the trap of learned helplessness. And in some it must be turned too low, and those of us for whom this is the case undertake wild projects with little chance of success; and if ninety-nine fail for every one who succeeds, that can yet drive the ratchet of progress.

But we easily forget that progress is not really a ratchet, and the more advanced our communications, the more lethal bad ideas become, for just as our transport networks spread disease like the 1918 flu epidemic which killed more people in a single year than the First World War killed in four years, so our communication networks spread parasite memes deadlier still. And we can't shut down the networks. We need them too badly.

I've seen the Singularity mutate from a harmless, even inspiring fantasy, to a parasite meme that I suspect could well snuff out the entire future of intelligent life. It's proving itself in many cases immune to any weight of evidence against it; perhaps worst of all, it bypasses ethical defenses, for it can be spread by people of honest goodwill.

Compartmentalization seems to be the primary remaining defense. When that fails, what have we left? This is not a rhetorical question; it may be one of the most important in the world right now.

The trouble is that even proper beliefs can be inadequately connected to other proper beliefs inside the human mind.

Proper beliefs can be too independent; if you have a belief network A -> B and the probabilities of 'B given A' and 'B given not-A' are similar, A doesn't have much value when you care about B. It doesn't change your belief much, because it isn't connected very much.

But my guess is most human brains have "A -> B" and don't have "B given A" and "B given not A". So they don't check the difference, so they don't see A isn't connected much to B.

So the general skill is noticing when B doesn't depend much on A.

I'm not sure what the "making sure your beliefs are actually connected in the first place" skill looks like when broken down to the 5-second level.

• Find a belief connection.
• Flip the truth value of A and see if B changes much.
• If it doesn't, delete the connection.
• If it does, remember the connection.

"Deleting the connection" and "remembering the connection" using our human brains are other 5-second level skills.

I don't know if this is an answer or a rephrasing of the problem, but "making sure your beliefs are propagated to the rest of your knowledge" is what I classified as the Level 2 Understanding.

Somewhat off-topic, but what struck me about your blog post is the apparent contradiction between "And when you’re that smart, you can do almost anything." and presuming that we can actually program a "super-smart" AI with a certain set of values that it would not instantly override based on the considerations we cannot even imagine.

Just to give an example, it might decide that humans are bad for the universe, because they might some day evolve to destroy it. Would we be able to prevent it from wiping/neutralizing humanity for the sake of the universe and potentially other intelligent species that may exist there? Would we even want to? Or there might be a universal law of AI physics that leads it to destroy its ancestors. Or maybe some calculations, Asimov's Foundation-style, would require it to subject humanity to untold millennia of suffering in order to prevent its total destruction by excessive fun.

No point arguing with these examples, since a super-smart AI would think in ways we cannot fathom. My point, again, is that believing that we can give an AI a set of goals or behaviors it would care about once it is smarter than us seems no smarter than believing in an invisible man in the sky who has a list of 10 things we are not supposed to do (thanks, George Carlin).