I'm more an outsider than a regular participant here on LW, but I have been boning up on rhetoric for work. I'm thrown by this in a lot of ways.

I notice that I'm confused.

Good for private rationality, bad for public rhetoric? What does your diagram of the argument's structure look like?

As for me, I want this as the most important conclusion in the summary.

But in fact most goals are dangerous when an AI becomes powerful

I don't get that, because the evidence for this statement comes after it and later on it is restated in a diluted form.

goals that seem safe ... can lead to extremely pathological behaviour if the AI becomes powerful

Do you want a different statement as the most important conclusion? If so, which one? If not, why do you believe the argument works best when structured this way? As opposed to, e.g., an alternative that puts the concrete evidence farther up and the abstract statement "Most goals are dangerous when an AI becomes powerful" somewhere towards the end.

Related point: I get frequent feelings of inconsistency when reading this summary.

• I'm encouraged to imagine the AI as a super committee of

  Edison, Bill Clinton, Plato, Oprah, Einstein, Caesar, Bach, Ford, Steve Jobs, Goebbels, Buddha, etc.

• then I'm told not to anthropomorphize the AI.

Or

• I'm told the AI's motivations are what "we actually programmed into it",
• then I'm asked to worry about the AI's motivation to lie.

Note I'm talking about rhetorical, a/k/a surface-level feeling of inconsistency here.

You seem like a nice guy.

Let's put on a halo. Isn't the easiest way to appear trustworthy to first appear attractive?

I was surprised this summary didn't produce emotions around this cluster of questions:

• Who are you?
• Do I like you?
• Do I respect your opinion?

Did you intend to skip over all that? If so, is it because you expect your target audience already has their answers?

Shut up and take my money!

There are so many futuristic scenarios out there. For various reasons, these didn't hit me in the gut.

The scenarios painted in the paragraph that starts with

Our society is setup to magnify the potential of such an entity, providing many routes to great power.

are very easy for me to imagine.

Unfortunately, that works against your summary for me. My imagination consistently conjures human beings.

• Wall Street banker.
• Political lobbyist for an industry that I dislike.
• (Nobody comes to mind for the "replace almost every worker in the service sector" scenario.)
• Chairman of the Federal Reserve.
• Anonymous Eastern European hacker.

The feeling that "these are problems I am familiar with, and my society is dealing with them through normal mechanisms" makes it hard for me to feel your message about novel risks demanding novel solutions. Am I unique here?

Inversely, the scenarios in the next paragraph, the one that starts with

Of course, simply because an AI could be extremely powerful

are difficult for me to seriously imagine. You acknowledge this problem later on, with

Humans don’t expect this kind of behaviour

Am I unique in feeling that as dismissive and condescending? Is there an alternative phrasing that takes into account my humanity yet still gets me afraid of this UFAI thing? I expect you have all gotten together, brainstormed scenarios of terrifying futures, trotted them out among your target audience, kept the ones that caused fear, and iterated on that a few times. Just want to check that my feelings are in the minority here.

Break any of these rules

I really enjoy Luke's post here: http://lesswrong.com/lw/86a/rhetoric_for_the_good/

It's a list of rules. Do you like using lists of rules as springboards for checking your rhetoric? I do. I find my writing improves when I try both sides of a rule that I'm currently following / breaking.

For something you give to journalists, the bullet points are the most important part, as they (the journalists) will likely get their impression from them and probably ignore the rest to spin their narrative the way they like it. What you currently call the "Executive summary" isn't. You have to fit your points into a 30-second pitch. 2 pages are 1.75 pages too long. It will be treated as optional supplementary material by the non-experts. With this in mind, I recommend skipping the first point:

The risks of artificial intelligence are strongly tied with the AI’s intelligence.

as unclear and so low-impact.

The second one:

There are reasons to suspect a true AI could become extremely smart and powerful.

is too mildly worded. "Reasons to suspect" and "could" do not call to act. "True AI" as opposed to what? False AI? Fake AI? If MIRI believes that unchecked organic AI development will almost inevitably lead to smarter-than-us artificial intelligence levels, this needs to be stated right there in the first line.

Possibly replace your first two points with something like

• By all indications, Artificial Intelligence could some day exceed human intelligence.

• History has proven that expecting to control or to even understand someone or something that is significantly smarter than you are is a hopeless task.

The rest of the bullet points look good to me. I would still recommend running your summary by a sympathetic but critical outsider before publishing it.

Not enough effort is currently being put into designing safe AIs.

Maybe rephrase this as "Not enough effort is currently being put into designing AI safeguards"? (Or perhaps "safe AIs and AI safeguards"). For example, I'm of the belief that AI boxing, while insufficient, is very necessary, and there currently aren't any tools that would make it viable.

The risks from artificial intelligence (AI) in no way resemble the popular image of the Terminator. That fictional mechanical monster is distinguished by many features – strength, armour, implacability, indestructability – but extreme intelligence isn’t one of them. And it is precisely extreme intelligence that would give an AI its power, and hence make it dangerous.

This example is weird, since it seems to me that MIRI position is exactly ripped off from the premise of the Terminator franchise.
Yes, the individual terminator robot doesn't look much smart (*), but Skynet is. Hell, it even invented time travel! :D

(* does it? How would a super-intelligent terminator try to kill Sarah/John Connor?)

Edison, Bill Clinton, Plato, Oprah, Einstein, Caesar, Bach, Ford, Steve Jobs, Goebbels, Buddha and other humans superlative superlative in their respective skill-sets

Is "superlative superlative" intended?

Typo: "The AI could make use of it unique, non-human ..." -- should be "its unique, non-human"

I think one standard method of improving the rhetorical value of your bullet points is to attempt to come up with a scenario that seems to generally agree with you, but disagrees with your bullet points, and imagine that scenario is rhetorically being presented to you by someone else.

Example Opposition Steel Man: Imagine researchers are attempting to use a very dumb piece of software to try to cycle through ways of generating Bacteria that clean up oil spills. The software starts cycling through possible bacteria, and it turns out that as a side effect, one of the generated bacteria spreads incredibly quickly and devours lipids in living cells in the controlled training setting. The researchers decide to not use that, since they don't want to devour organic lipids, but they accidentally break the vial, and A worldwide pandemic ensues. When asked why they didn't institute AI safety measures, the researchers replied that they didn't think the software was smart enough for AI safety measures to matter, since it basically just brute forced through boring parts of the research the researchers would have done anyway.

Example Opposition Steel Man (cont): This would seem to falsify the idea that A dangerous AI will be motivated to seem safe in any controlled training setting, since the AI was too dumb to have any thing resembling purposeful motivation and was still extremely dangerous, and the researchers thought of it as not even an AI, so they did not think they would have to consider the idea that not enough effort is currently being put into designing safe AIs. I would instead say not enough effort is currently being put into designing safe software.

Then, attempt to turn that Steel Man's argument into a bullet point:

Not enough effort is currently being put into designing safe software.

Then ask yourself: Do I have any reasons to not use this bullet point, as opposed to the bullet points the Example Opposition Steel Man disagreed with?

(creating a kind of “super-committee” of the AI equivalents of, say, Edison, Bill Clinton, Plato, Oprah, Einstein, Caesar, Bach, Ford, Steve Jobs, Goebbels, Buddha and other humans superlative superlative in their respective skill-sets)

I think you will want here to create a sort of positive affect spiral, so kill Goebbels and put, say, Patton or Churchill (I guess the orientation is to appeal to anglophone cultures, since outside of them Oprah is almost unkown).

But the AI might be an extremely alien mind:

I would change the charged word "alien" and put something similar, like "strange" or "different". It might trigger bad automatic associations with science fiction.

I like it. My only comment is a grammar nitpick:

...and cast it unambiguously and error-free into computer code.

"error-free" is an adjective; it sounds odd when applied to the verb "cast". Maybe:

...and cast it unambiguously and without error into computer code.

A lot of people who are unfamiliar with AI dismiss ideas inherent in the strong AGI argument. I think it's always good to include the "G" or to qualify your explanation, with something like "the AGI formulation of AI, also known as 'strong AI.'"

The risks of artificial intelligence are strongly tied with the AI’s intelligence.

AGI's intelligence. AI such as Numenta's grok can possess unbelievable neocortical intelligence, but without a reptile brain and a hippocampus and thalamus that shifts between goals, it "just follows orders." In fact, what does the term "just following orders" remind you of? I'm not sure that we want a limited-capacity AGI that follows human goal structures. What if those humans are sociopaths?

I think, as does Peter Voss, that AGI is likely to improve human morality, rather than to threaten it.

There are reasons to suspect a true AI could become extremely smart and powerful.

Agreed, and well-representing MIRI's position. MIRI is a little light on "bottom up" paths to AGI that are likely to be benevolent, such as those who are "raised as human children." I think Voss is even more right about these, given sufficient care, respect, and attention.

Most AI motivations and goals become dangerous when the AI becomes powerful.

I disagree here, for the same reasons Voss disagrees. I think "most" overstates the case for most responsible pathways forward. One pathway that does generate a lot of sociopathic (lacking mirror neurons and human connectivity) options is the "algorithmic design" or "provably friendly, top-down design" approach. This is possibly highly ironic.

Does most of MIRI agree with this point? I know Eliezer has written about reasons why this is likely the case, but there appears to be a large "biological school" or "firm takeoff" school on MIRI as well. ...And I'm not just talking about Voss's adherents, either. Some of Moravec's ideas are similar, as are some of Rodney Brooks' ideas. (And Philip K. Dick's "The Second Variety" is a more realistic version of this kind of dystopia than "the Terminator.")

It is very challenging to program an AI with safe motivations.

Agreed there. Well-worded. And this should get the journalists thinking at least at the level of Omohundro's introductory speech.

Mere intelligence is not a guarantee of safe interpretation of its goals.

Also good.

A dangerous AI will be motivated to seem safe in any controlled training setting.

I prefer "might be" or "will likely be" or "has several reasons to be" to the words "will be." I don't think LW can predict the future, but I think they can speak very intelligently about predictable risks the future might hold.

Not enough effort is currently being put into designing safe AIs.

I think everyone here agrees with this statement, but there are a few more approaches that I believe are likely to be valid, beyond the "intentionally-built-in-safety" approach. Moreover, these approaches, as noted fearfully by Yudkowsky, have less "overhead" than the "intentionally-built-in-safety" approach. However, I believe this is equally as likely to save us as it is to doom us. I think Voss agrees with this, but I don't know for sure.

I know that evolution had a tendency to weed out sociopaths that were very frequent indeed. Without that inherent biological expiration date, a big screwup could be an existential risk. I'd like a sentence that kind of summed this last point up, because I think it might get the journalists thinking at a higher level. This is Hans Moravec's primary point, when he urges us to become a "sea faring people" as the "tide of machine intelligence rises."

If the AGI is "nanoteched," it could be militarily superior to all humans, without much effort, in a few days after achieving super-intelligence.