Singularity FAQ

This needs more work, my nitpick detector trips off on every other sentence. If you will be willing to heavily revise, I'll compile a more detailed list (or revise some sections myself).

Examples (starting from the end, skipping some):

• "but that stable eventual goal may be very difficult to predict in advance" - you don't predict goals, you make them a certain way.
• "We must also figure out how to build a general intelligence that satisfies a goal at all, and that stably retains that goal as it edits its own code to make itself smarter. This task is perhaps the primary difficulty in designing friendly AI." - last sentence unwarranted.
• "Eliezer Yudkowsky has proposed[57] Coherent Extrapolated Volition as a solution to two problems facing Friendly AI design:" - not just these two, bad wording
• "We have already seen how simple rule-based and utilitarian designs for Friendly AI will fail. " - should be a link/reference, a FAQ can be entered at any question.
• "The second problem is that a superintelligence may generalize the wrong principles due to coincidental patterns in the training data" - a textbook error in machine learning methodology is a bad match for a fundamental problem, unless argued as being such in this particular case.
• "But even if humans could be made to agree on all the training cases, two problems remain." - just two? Bad wording.
• "The first problem is that training on cases from our present reality may not result in a machine that will make correct ethical decisions in a world radically reshaped by superintelligence." - the same can be said of humans (correctly, but as a result it doesn't work as a simple distinguishing argument).
• "Let's consider the likely consequences of some utilitarian designs for Friendly AI." - "utilitarian": a potentially new term without any introduction, even with a link, is better to be avoided.
• "An AI designed to minimize human suffering would simply kill all humans" - could/might would be better.
• "caters to the complex and demanding wants of humanity" - this statement is repeated about 5 times in close forms, should change the wording somehow.
• "by wiring humans into Nozick’s experience machines. " - an even more opaque term without explanation.
• "Either option would be easier for the AI to achieve than maintaining a utopian society catering to the complexity of human (and animal) desires." - not actually clear (from my point of view, not simulated naive point of view). The notion of "default route" in foreign minds can be quite strange, and you don't need much complexity in generating principle for a fractal to appear diverse. (There are clearly third alternatives that shelve both considered options, which also makes the comparison not terribly well-defined.)
• "It's not just a problem of specifying goals, either. It is hard to predict how goals will change in a self-modifying agent. No current mathematical decision theory can predict the decisions of a self-modifying agent." - again, these things are there to be decided upon, not "predicted"
• etc.

(Originally sent as a PM, but I think it's worth saying in public.)

Good work, first of all. I think you might still be a few inferential leaps past many plausible readers, though. For instance, many people don't actually know that it's physically possible to run a WBE a million times faster than a brain, nor that there's a lot we know is possible to program but can't do yet.

You need to point out that nerve impulses are much slower than semiconductor logic gates, and that most of the reason the brain is better at many tasks is because 50 years of programming hasn't yet caught up to several hundred million years of evolution on things like vision processing. Concepts like those might be helpful for the non-LW readers.

But I don't mean to criticize too much, because what you've done is pretty excellent!

In general, a pretty good FAQ, but I do have some criticism. First of all, I'm not sure if I'd put general transhumanist background information (2.1-2.5) under "How Likely is an Intelligence Explosion". I realize that those technologies are relevant to a potential intelligence explostion, but the placement still seems somewhat odd.

Also, you say that "singularity" refers to the intelligence explosion in the FAQ, but in "When will the Singularity happen?", you list predictions of other forms of the singularity. I would recommend adding something to indicate what exactly each source is predicting when they say "singularity". What Ray Kurzweil predicts will happen in 2045 bares little resemblance to what Eliezer Yudkowsky expects to happen by 2060.

An AI designed to minimize human suffering would simply kill all humans: no humans, no human suffering

This seems too strong, I'd suggest changing "would" to "might" or "could".

Also, at two different points in the FAQ you use almost identical language about sticking humans in jars. You may want to change up the wording slightly or make it clear that you recognize the redundancy ("as discussed in question blah..." might do it).

"Humans can intelligently adapt to radically new problems in the urban jungle or outer space for which evolution could not have prepared them."

This here trips my 'then what did?' sensors. I understand the intent, but evolution -did- prepare us, by giving us general intelligence. Perhaps you meant that evolution did not -specifically- adapt us to those challenges?

You misspelt avocado.

Sorry, but when else am I going to get to say that? =P

ETA: Great FAQ!

This is an excellent singularity FAQ which handles a very difficult task surprisingly well.

"If 'harm' is defined in terms of thwarting human desires, it could rewire human desires. And so on."

...not if humans desire for their desires to be preserved, which they do.

Thank you so much for writing this!

Thanks for writing this!

Thanks for writing this!

Bookmarked!

Edit - please disregard this post