Simplified Humanism, Positive Futurism & How to Prevent the Universe From Being Turned Into Paper Clips

Would you trade those base emotions for a paperclip?

Sorry, I just copied the title from Michael Anissimov. I agree that the title was chosen to be inflammatory.

He likely means a formal statement of the claim about decision systems that would take the form something like "Under the following formal definition of a decision system, as long as the following pathological/stupid conditions don't hold, a decision system will not seek to modify its goals." There are a fair number of mathematical theorems that have forms close to this where we can prove something for some large set of things but there are edge cases where we can't. That's the sort of thing Eliezer is talking about here (although we don't even have a really satisfactory definition of decision system at this point so what Eliezer wants is very optimistic here.)

Heresy alert: Eliezer seems to be better at writing than he is at AI theory. Maybe he should write a big piece of SF about unfriendly and friendly AI to make these concepts as popular as Skynet or the Matrix. A textbook on rationality won't have as much impact.

I have to dissent here: I actually stopped reading the sequences with several more to go because many of them have a very high words-to-content ratio (especially because they were written as separate blog posts over multiple days, and often take the time to summarize points from previous posts). I was really hoping that Eliezer's book would be a concise summary of the rationality content here, not only for my own benefit, but because let's face it: telling LW newcomers that they should probably get started reading the several hundred posts that make up the sequences is a pretty large barrier to entry.

Although, now that I think about it, I'm likely atypical. Even though I very much enjoyed (parts of) GEB, I thought it was very wordy and actually never finished it (quit around page 400).

That's the length of the first draft-- the finished version might be a good bit longer or shorter.

I can't believe you just put those two books in the same sentence.

A bit of calculating: It's typical to have about 250 words per page in a published book (unless it's really wide, or has fine print, or something), so that would come out to about 1200 pages. Of course, if it's printed with larger pages, it'll be around the same weight as Harry Potter and the Order of the Phoenix, which had almost that many words.

Well, we rationalists have computed that to be the optimal cached response to questions of our spontanaity.

The stakes are so high in the FAI problem that it's worth it to get very close to 0 risk. I'm not even sure the FAI program can get us comfortably close to 0 risk: An AI won't start acting Friendly until CEV has been partially computed, so we'd probably want to first handcraft an approximation to CEV without the use of AGI; there are a number of ways that could go wrong.

In contrast, AGI containment seems almost completely worthless as an existential-risk reducer. If mere humans can break out of a crude AI box, it stands to reason that a self-improving AGI that is capable of outwitting us could break out of any human-designed box.

The goal of development of FAI is reduction of its existential threat to near 0%, by mathematically proving stability and desirability of its preferences. It's OK, but it reminds me of zero-risk bias.

Excellent point. The goal of FAI should be to increase expected value, not to minimize risk.