Questions for a Friendly AI FAQ

You may also wish to consider including some of the questions in my and Tom McCabe's Singularity FAQ. I was never very happy with it and don't generally endorse it in its current state. We managed to gather a large variety of AI/FAI/Singularity-related questions from various people, but never really managed to write very good answers. But you might find some of the questions useful.

Questions from some FAI-relevant sections. IIRC, these are all questions that somebody has actually asked, at one point or another.

Alternatives to Friendly AI

Q1). Couldn't AIs be built as pure advisors, so they wouldn't do anything themselves?
Q2). Wouldn't a human upload naturally be more Friendly than any AI?
Q3). Trying to create a theory which absolutely guarantees Friendly AI is an unrealistic, extremely difficult goal, so isn't it a better idea to attempt to create a theory of "probably Friendly AI"?
Q4). Shouldn't we work on building a transparent society, where no illicit AI development can be carried out?

Implementation of Friendly AI

Q1). Wouldn't an AI that's forced to be Friendly be prevented from evolving and growing?
Q2). Didn't Shane Legg prove that we can't predict the behavior of intelligences smarter than us?
Q3). Since a superintelligence could rewrite itself to remove human tampering, isn't Friendly AI impossible?
Q4). Why would a super-intelligent AI have any reason to care about humans, who would be stupid by comparison?
Q5). What if the AI misinterprets its goals?
Q6). Isn't it impossible to simulate a person's development without creating, essentially, a copy of that person?
Q7). Isn't it impossible to know a person's subjective desires and feelings from outside?
Q8). Couldn't a machine never understand human morality, or human emotions?
Q9). What if AIs take advantage of their power, and create a dictatorship of the machines?
Q10). If we don't build a self-preservation instinct into the AI, wouldn't it just find no reason to continue existing, and commit suicide?
Q11). What if superintelligent AIs reason that it's best for humanity to destroy itself?
Q12). The main defining characteristic of complex systems, such as minds, is that no mathematical verification of properties such as "Friendliness" is possible; hence, even if Friendliness is possible in theory, isn't it impossible to implement?
Q13). Any future AI would undergo natural selection, so wouldn't it eventually become hostile to humanity to better pursue reproductive fitness?
Q14). Shouldn't FAI be done as an open-source effort, so other people can see that the project isn't being hijacked to make some guy Supreme Emperor of the Universe?
Q15). If an FAI does what we would want if we were less selfish, won't it kill us all in the process of extracting resources to colonize space as quickly as possible to prevent astronomical waste?
Q16). What if ethics are subjective, not objective? Then, no truly Friendly AI could be built.
Q18). Isn't the idea of a hostile AI anthropomorphic?
Q19). Isn't the idea of "Friendliness", as we understand it now, too vaguely defined?
Q20). Why don't mainstream researchers consider Friendliness an issue?
Q21). How could an AI build a computer model of human morality, when human morals contradict each other, even within individuals?
Q22). Aren't most humans are rotten bastards? Basing an FAI morality off of human morality is a bad idea anyway.
Q23). If an AI is programmed to make us happy, the best way to make us happy would be to constantly stimulate our pleasure centers, so wouldn't it turn us into nothing but experiencers of constant orgasms?
Q24). What if an AI decides to force us to do what it thinks is best for us, or what will make us the happiest, even if we don't like it?

General Questions

Q1). If AI is a serious threat, then wouldn't the American government or some other official agency step in and take action, for fear of endangering national security?
Q2). Won't the US government, Google, or some other large organization with billions of dollars and thousands of employees, be the first ones to develop strong AI?
Q3). What if the Singularity Institute and its supporters is just another "doomsday cult", like the religious extremists who talk about how "the end is nigh"?
Q4). Shouldn't all of humanity have input on how a Friendly AI should be designed, instead of just a few programmers or scientists?
Q5). Has the Singularity Institute done research and published papers, like other research groups and academic institutions?

Societal issues

Q1). What if humans don't accept being ruled by machines?
Q2). How do we make sure that an AI doesn't just end up being a tool of whichever group built it, or controls it?
Q3). Aren't power-hungry organizations going to race to AI technology, and use it to dominate the world, before there's time to create truly Friendly AI?
Q4). What if an FAI only helps the rich, the First World, uploaded humans, or some other privileged class of elites?
Q5). Since hundreds of thousands of people are dying every day, don't need AI too urgently to let our research efforts be delayed by having to guarantee Friendliness?

What are the main objections to the likelihood of the Singularity occurring? What actions might people take to stop a Singularity from occurring? How will competition among businesses and governments impact the amount of care taken with respect to friendly AI? What's the difference between AI and AGI? What organizations are working on the friendly AI problem? How will expectations of friendly/unfriendly AI impact the amount of resources devoted to AI, i.e. if financial markets expect utopia to arrive in the next decade savings rates will fall which will lower tech research. What are AGI researchers estimates for when/if AGI will happen and the likelihood of it being "good" if it does occur. Why do most computer engineers dismiss the possibility of AGI? What is the track records of AGI predictions? Who has donated significant sums to friendly AI research? How much money is being spent on AGI/friendly AGI research?

More questions to perhaps add:

What is self-modification? (In particular, does having one AI build another bigger and more wonderful AI while leaving "itself" intact count as self-modification? The naive answer is "no", but I gather the informed answer is "yes", so you'll want to clarify this before using the term.)

What is wrong with the simplest decision theory? (That is, enumerate the possible actions and pick the one for which the expected utility of the outcome is best. I'm not sure what the standard name for that is.) It's important to answer this so at some point you state the problem that timeless decision theory etc. are meant to solve.

I gather one of the problems with the simplest decision theory is that it gives the AI an incentive to self-modify under certain circumstances, and there's a perceived need for the AI to avoid routine self-modification. The FAQ question might be "How can we avoid giving the AI an incentive to self-modify?" and perhaps "What are the risks of allowing the AI to self-modify?"

What problem is solved by extrapolation? (This goes in the CEV section.)

What are the advantages and disadvantages of having a bounded utility function?

Can we just upload a moral person? (In the "Need for FAI" section. IMO the answer is a clear "no".)

I suggest rephrasing "What powers might it have?" in 1.10 to "What could we reasonably expect it to be able to do?". The common phrase "magical powers" gives the word "powers" undesired connotations in this context, makes us sound like loonies.

Who is the target audience?

Follow-up/variation on Q5.7: Is it possible for unenhanced human brains to figure out how to properly formulate a human utility function? If not, could a WBE be reliably improved enough that it could do such a thing without significantly changing its values?

Also: Most AI researchers don't seem too concerned about friendliness. If I don't know even close to as much about AI as they do, why should I be convinced by any argument that I know failed to convince them?

What is recursive self-improvement?

Meta-question: what is the smallest subset of these questions that someone interested in the issue should read first?

(IOW: organizing these questions by category makes sense from an author's point of view, but different organization keys might make a lot more sense from a reader's point of view.)

Why should we believe any prediction that an AGI is likely to be created soon, given the history of these predictions in the past?

What progress has been made at solving the very hard problems of AGI, such as representing general knowledge, or understanding natural language?

Is there it possible that humans are incapable of constructing an AGI by reason of our own limited intelligence?

Is it possible for an AGI to be created and yet an intelligence explosion to not happen? [Norvig's talk at the Singularity Summit posits that this is possible]

(Note that I don't fully endorse the skepticism of these questions, but they're questions that reasonable people might ask).

What security measures are needed for the development of a Friendly AI? (i.e. if leaked source code got into the hands of a less scrupulous team, would it increase the risk of an unfriendly intelligence explosion?)

How much would it cost to develop a Friendly AI?

If a Friendly AI were developed, how would it affect my life?

How likely is it that systems intended as domain-specific would accidentally cross the threshold into AGI?

Consider addressing these questions:

• "Can we just upload humans instead of trying to create de novo AI?"

• "Solving Friendly AI now seems too difficult. Can we just build an AI Nanny, which would temporarily hold off a Singularity, for now?"

• and some other passageways mentioned here;

Also I would like this question to be answered in detail:

• What is reflective decision theory, and why do we need it?

• What are the applications of reflective decision theory in FAI/How will the FAI use reflective decision theory?

• "What is updateless decision theory?"

I hate to comment before reading the body of your post, but the title of the post quite literally says "Friendly Artificial Intelligence Frequently Asked Questions Questions."

I'm just pointing it out to get it out of the way, though... It doesn't really bug me that much.

Questions for Section 3: What if CEV outputs something horrible? What if it misses out on something awesome? Why would values converge at all?

The non-convergence of a large set of utility functions when using the Solomonoff prior is an important open problem (or are you including that in Q5.5?).

"How can an AI choose an ideal prior given infinite computing power?" is just begging for a follow-up question like "What if you don't have infinite computing power?"

EDIT: Oh, I didn't notice Q5.9 there which I guess is that.

"Can we just build an AI Nanny for now?"