I think this paragraph reflects a very serious confusion that is seen on LW regularly:

How strongly should I believe P? How should I adjust my probability for P in the face of new evidence X? There is a single, exactly correct answer to each such question, and it is provided by Bayes’ Theorem. We may never know the correct answer, but we can plug estimated numbers into the equation and update our beliefs accordingly.

Most of your beliefs are not produced by some process that you can break into its component parts and analyze mathematically so as to assign a numerical probability. Rather, they are produced by opaque black-box circuits in your brain, about whose internal functioning you know little or nothing. Often these circuits function very well and let you form very reliable judgments, but without the ability to reverse-engineer and analyze them in detail, which you presently don't have, you cannot know what would be the correct probability (by any definition) assigned to their outputs, except for the vague feeling of certainty that they typically produce along with their results.

If instead of relying on your brain's internal specialized black-box circuits you use some formal calculation procedure to produce probability estimates, then yes, these numbers can make sense. However, the important points are that: (1) the numbers produced this way do not pertain to the outputs of your brain's opaque circuits, but only to the output of the formal procedure itself, and (2) these opaque circuits, as little as we know about how they actually work, very often produce much more reliable judgments than any formal models we have. Assigning probability numbers produced by explicit formal procedures to beliefs produced by opaque procedures in one's head is a total fallacy, and discarding the latter in favor of the former makes it impossible to grapple with the real world at all.

I'll post this here and put a copy on my desktop so I remember to check it in a few months. My beliefs tend to change very often and very quickly.

Beliefs that I think are most likely to change:

• Existence is very tied up with relative causal significance. Classical subjective anticipation makes little sense. Quantum immortality should be replaced with causal immortality.
• Reflective consistency is slippery due to impermanence of agency and may be significantly less compelling than I had previously thought.
• Something like human 'morality' (possibly more relevant to actions pre-Singularity than Eliezer's conception of 'good' as humanity-CEV) might be important for reasons having to do with acausal control and, to a lesser extent, smarter-than-human intelligences in the multiverse looking at humanity and other alien civilizations as evidence of what patterns of action could be recognized as morally justified.
• Building a seed AI that doesn't converge on a design something like the one outlined in Creating Friendly AI may be impossible due to convergent decision theoretic reflective self re-engineering (and the grounding problem). Of course, for all purposes this intuition doesn't matter, as we still have to prove something like Friendliness.
• Solving Friendliness (minus the AGI part (which would be rather integrated so this is kind of vague)) is somewhat easier than cleanly engineered seed AI, independent of the previous bullet point.
• Death the way it is normally conceptualized is a confusion. The Buddhist conception of rebirth is more accurate. (And it doesn't mean the transparently stupid thing that most Westerners imagine.)
• Most of Less Wrong's intuitions about how the world works are based on an advanced form of naive realism that just doesn't work. "It all adds up to normality" is either tautologous or just plain wrong. What you thought of as normality normally isn't.
• Ensemble universe theories are almost certainly correct.
• Suicide with the intent of ending subjective experience is downright impossible. (The idea of a 'self' that is suffering is also a confusion, but anyway...) The only form of liberation from suffering is Enlightenment in the Buddhist sense.
• I am the main character to the extent that 'I' 'am'. At the very least I should act as if I am for instrumentally rational reasons.
• People that are good at 'doing things' will never have the epistemic rationality necessary to build FAI or seed AI, due to limits of human psychology. Whoever does build AI will probably have some form of schizoid personality disorder or, ironically somewhat oppositely, autistic spectrum disorder.
• My intuition is totally awesome and can be used reliably to see important themes in scientific, philosophical, and spiritual fields.

"But philosophers still argue about ... theism ... as if these weren’t settled questions."

If this is really what you think, then why do you continue with your blog?

My Web of Opinions, FWIW:

Ontology and Epistemology: Philosophy is very much a matter of opinion. But that doesn't mean that one philosophical position is as good as another. Some are better than others - but not because they are better supported by reasons. "Support" has nothing whatever to do with the goodness of a philosophical position. Instead, positions should be judged by their merits as a vantage point. Philosophers shouldn't seek positions based on truth, they should seek positions based on fruitfulness.

Language and Logic: Bertrand Russell and W.V.O. Quine have much to answer for. Michael Dummett, Saul Kripke, and David Lewis have repaired some of the damage. Eliezer's naive realism and over-enthusiasm for reductionism set my teeth on edge. We need a more pluralist account of theories, models, and science. Type theory and constructivism are the wave of the future in foundational mathematics.

Ethics: Ethics answers the question "What actions deserve approval or disapproval?" (Not the question "What ought I to do?") The question is answered by Nash(1953) in the two-person case. Actions that do not at least conform to the (unique, correct) bargain deserve disapproval and punishment. Notice that the question presumes rational agents with perfect information - that is why what you ought (ethically) to do may sometimes differ from what you ought (practically) to do. Future utilities should be discounted at a rate of at least 1% per year (50% per lifetime).

Rationality and decision theory. I suspect that TDT/UDT are going in the wrong direction, but perhaps I just don't understand them yet. The most fruitful issues for research here seem to be in modeling agents as coalitions of subagents, creating models in which it can be rational to change one's own utility function, biology-inspired modeling using Price equations and Hamilton's rule, and the question of coalition formation and structure in multi-agent Nash bargaining. Oh yeah: And rational creation/destruction of one agent by another.

Futurism: The biggest existential risk facing mankind is uFAI. Trying to build a FOOMing AI which has the fixed goal of advancing extrapolated human values seems incredibly dangerous to me. Instead, I think we should try to avoid a singleton and do all we can to prevent the creation of AIs with long-term goals. But at this stage, that is just a guess. There is a 50%+ probability, though, that there is no big AI risk at all, and that super-intelligences will not be all that much more powerful than smart people and organizations.

Note: posting this feels very self-indulgent. Though I do see value in setting it down as a milestone for later comparison.

If I told you I might get stuck in it, like a fly.

Nice post!

By the way, "superdictionary" made me think of OmegaWiki, a sort-of-fork from Wiktionary to build a multilingual translating dictionary. (No, it's not about Omega ;-)

"The normativity of logic is: “If you want to be speaking the same language as everyone else, don’t say things like ‘The ball is all green and all blue at the same time in the same way.’”"

You surely don't mean this: everyone one else is logical, why not me?

For a start, is everyone else logical? And even if they are, is that the best justification we have for logic?

I can't imagine how anything short enough to be bearable by others could possibly qualify as a summary of my beliefs. There are too many things to have beliefs about. After reading Luke's list, I don't feel that I know what Luke thinks about anything much, other than a tiny set of highly-meta things (which are, yes, interesting and important).

I like the idea of publicly summarizing one's beliefs, but if I were going to do that I'd need something much more comprehensive before I'd feel at all like calling it "my web of beliefs" or "a summary of my beliefs".