In chapter two I survey the philosophical problems involved in designing the goal system of a singular superintelligence, which I call the “singleton.”

That sounds a bit like you invented the term "singleton". I suggest to clarify that with a footnote.

You say you'll present some objections to CEV. Can you describe a concrete failure scenario of CEV, and state a computational procedure that does better?

Luke, as an intermediate step before writing a book you should write a book chapter for Springer's upcoming edited volume on the Singularity Hypothesis. http://singularityhypothesis.blogspot.com/p/about-singularity-hypothesis.html I'm not sure how biased they are against non-academics... probably depends on how many submissions they get.

Maybe email Louie and me and we can brainstorm about topics; meta-ethics might not be the best thing compared to something like making an argument about how we need to solve all of philosophy in order to safely build AI.

The first sentence is the most important of any book because if a reader doesn't like it he will stop. Your first sentence contains four numbers, none of which are relevant to your core thesis. Forgive me for being cruel but a publisher reading this sentence would conclude that you lack the ability to write a book people would want to read.

Look at successful non-fiction books to see how they get started.

Bibliographic references are provided here.

I notice some of the references you suggest are available as online resources. It would be a courtesy if you provided links.

"This extinction would result from a “technological singularity” in which an artificial intelligence (AI) . . . "

By this point, you've talked about airplanes, Apollo, science good, philosophy bad. Then you introduce the concepts of existential risk, claim we are at the cusp of an extinction level event, and the end of the world is going to come from . . . Skynet.

And we're only to paragraph four.

These are complex ideas. Your readers need time to digest them. Slow down.

You may also want to think about coming at this from another direction. If the goal is to convince your readers AI is dangerous, maybe you should introduce the concept of AI first. Then explain why their dangerous. Use an example that everyone knows about and build on that. You need to establish rapport with your readers before you try to get them to accept strange ideas. (For example, it is common knowledge computers are better at chess than humans.)

Finally, is your goal to get published? Nonfiction is usually written on spec. Some (many, all?) publishers are wary of buying anything that has already appeared on the internet. Just a few things to keep in mind.

I'm glad you're writing a book!

I tried reading this through the eyes of someone who wasn't familiar with the singularity & LW ideas, and you lost me with the fourth paragraph ("This extinction..."). Paragraph 3 makes the extremely bold claim that humanity could face its extinction soon unless we solve some longstanding philosophical problems. When someone says something outrageous-sounding like that, they have a short window to get me to see how their claim could be plausible and is worth at least considering as a hypothesis, otherwise it gets classified as ridiculous nonsense. You missed that chance, and instead went with a dense paragraph filled with jargon (which is too inferentially distant to add plausibility) and more far-fetched claims (which further activate my bullshit detector).

What I'd like to see instead is a few paragraphs sketching out the argument in a way that's as simple, understandable, and jargon-free as possible. First why to expect an intelligence explosion (computers getting better and more domain general, what happens when they can do computer science?), then why the superintelligences could determine the fate of the planet (humans took over the planet once we got smart enough, what happens when the computers are way smarter than us?), then what this has to do with philosophy (philosophical rules about how to behave aren't essential for humans to get along with each other since we have genes, socialization, and interdependence due to limited power, but these computers won't have that so the way to behave will need to be programmed in).

There's not much to critically engage with yet, but...

I find it odd that you claim to have "laid [your] positions on the table" in the first half of this piece. As far as I can make out, the first half only describes a set of problems and possibilities arising from the "intelligence explosion". It doesn't say anything about your response or proposed solution to those problems.

I haven't read all of the recent comments. Have you made progress yet on understanding Yudkowsky's meta-ethics sequence? I hope you let us know if you do (via a top-level post). It seems a bit weird to write a book on it if you don't either understand it yet or haven't disregarded understanding it for the purpose of your book.

Anyway, I appreciate your efforts very much and think that the book will be highly valuable either way.

But even if we can solve those problems, the question of which goal system to give the superintelligence remains. It is a question of philosophy; it is a question of ethics.

Isn't it an interdisciplinary question, also involving decision theory, game theory and evolutionary psychology etc.? Maybe it is mainly a question about philosophy of ethics, but not solely?

...and a solar system in which all available matter is converted into parts for a planet-sized computer built to solve difficult mathematical problems.

This sentence isn't very clear. People who don't know about the topic will think, "to create an utopia you also have to solve difficult mathematical problems."

This project could mean the difference between a utopian solar system of unprecedented harmony and happiness, and a solar system void of human values in which all available matter is being used to to pursue a set of narrow goals.

The Wright Brothers flew their spruce-wood plane for 200 feet in 1903. Only 66 years later, Neil Armstrong walked on the moon, more than 240,000 miles from Earth.

I'm not sure if there is a real connection here? Has any research on "flight machines" converged with rocket science? They seem not to be correlated very much or the correlation is not obvious. Do you think it might be good to advance on that point or rephrase it to show that there has been some kind of intellectual or economic speedup that caused the quick development of various technologies?

I'll offer you a trade: an extensive and in-depth analysis of your book in return for an equivalent analysis of my book.

Quick note: I think explicit metadiscourse like "In Chapter 7 I argue that..." is ugly. Instead, try to fold those kinds of organizational notes into the flow of the text or argument. So write something like "But C.E.V. has some potential problems, as noted in Chapter 7, such as..." Or just throw away metadiscourse altogether.

Thanks, everyone. I agree with almost every point here and have updated my own copy accordingly. I especially look forward to your comments when I have something meaty to say.

In this chapter I argue that the technological singularity is likely to occur within the next 200 years...

If it takes 200 years it could as well take 2000. I'm skeptical that if it doesn't occur this century it will occur next century for sure. If it doesn't occur this century then that might as well mean that it won't occur any time soon afterwards either.