Q&A with Shane Legg on risks from AI

post by XiXiDu · 2011-06-17T08:58:06.099Z · LW · GW · Legacy · 25 comments

Contents

    [Click here to see a list of all interviews]
    Publications by Shane Legg:
  The Interview:
None
25 comments

[Click here to see a list of all interviews]

I am emailing experts in order to raise and estimate the academic awareness and perception of risks from AI.

Below you will find some thoughts on the topic by Shane Legg, a computer scientist and AI researcher who has been working on theoretical models of super intelligent machines (AIXI) with Prof. Marcus Hutter. His PhD thesis Machine Super Intelligence has been completed in 2008. He was awarded the $10,000 Canadian Singularity Institute for Artificial Intelligence Prize.

Publications by Shane Legg:

• Solomonoff Induction thesis
• Universal Intelligence: A Definition of Machine Intelligence paper
• Algorithmic Probability Theory article
• Tests of Machine Intelligence paper
• A Formal Measure of Machine Intelligence paper talk slides
• A Collection of Definitions of Intelligence paper
• A Formal Definition of Intelligence for Artificial Systems abstract poster
• Is there an Elegant Universal Theory of Prediction? paper slides

The full list of publications by Shane Legg can be found here.

The Interview:

Q1: Assuming no global catastrophe halts progress, by what year would you assign a 10%/50%/90% chance of the development of human-level machine intelligence?

Explanatory remark to Q1:

P(human-level AI by (year) | no wars ∧ no disasters ∧ beneficially political and economic development) = 10%/50%/90%

Shane Legg: 2018, 2028, 2050

Q2: What probability do you assign to the possibility of negative/extremely negative consequences as a result of badly done AI?

Explanatory remark to Q2:

P(negative consequences | badly done AI) = ?
P(extremely negative consequences | badly done AI) = ?

(Where 'negative' = human extinction; 'extremely negative' = humans suffer;)

Shane Legg: Depends a lot on how you define things. Eventually, I think human extinction will probably occur, and technology will likely play a part in this.  But there's a big difference between this being within a year of something like human level AI, and within a million years. As for the former meaning...I don't know.  Maybe 5%, maybe 50%. I don't think anybody has a good estimate of this.

If by suffering you mean prolonged suffering, then I think this is quite unlikely.  If a super intelligent machine (or any kind of super intelligent agent) decided to get rid of us, I think it would do so pretty efficiently. I don't think we will deliberately design super intelligent machines to maximise human suffering.

Q3: What probability do you assign to the possibility of a human level AGI to self-modify its way up to massive superhuman intelligence within a matter of hours/days/< 5 years?

Explanatory remark to Q3:

P(superhuman intelligence within hours | human-level AI running at human-level speed equipped with a 100 Gigabit Internet connection) = ?
P(superhuman intelligence within days | human-level AI running at human-level speed equipped with a 100 Gigabit Internet connection) = ?
P(superhuman intelligence within < 5 years | human-level AI running at human-level speed equipped with a 100 Gigabit Internet connection) = ?

Shane Legg: "human level" is a rather vague term. No doubt a machine will be super human at some things, and sub human at others.  What kinds of things it's good at makes a big difference.

In any case, I suspect that once we have a human level AGI, it's more likely that it will be the team of humans who understand how it works that will scale it up to something significantly super human, rather than the machine itself. Then the machine would be likely to self improve.

How fast would that then proceed? Could be very fast, could be impossible -- there could be non-linear complexity constrains meaning that even theoretically optimal algorithms experience strongly diminishing intelligence returns for additional compute power. We just don't know.

Q4: Is it important to figure out how to make AI provably friendly to us and our values (non-dangerous), before attempting to solve artificial general intelligence?

Shane Legg: I think we have a bit of a chicken and egg issue here. At the moment we don't agree on what intelligence is or how to measure it, and we certainly don't agree on how a human level AI is going to work. So, how do we make something safe when we don't properly understand what that something is or how it will work? Some theoretical issues can be usefully considered and addressed. But without a concrete and grounded understanding of AGI, I think that an abstract analysis of the issues is going to be very shaky.

Q5: How much money is currently required to mitigate possible risks from AI (to be instrumental in maximizing your personal long-term goals, e.g. surviving this century), less/no more/little more/much more/vastly more?

Shane Legg: Much more. Though, similar to many charity projects, simply throwing more money at the problem is unlikely to help all that much, and it may even make things worse. I think the biggest issue isn't really financial, but cultural. I think this is going to change as AI progresses and people start to take the idea of human level AGI within their lifetimes more seriously.  Until that happens I think that the serious study of AGI risks will remain fringe.

Q6: Do possible risks from AI outweigh other possible existential risks, e.g. risks associated with the possibility of advanced nanotechnology?

Explanatory remark to Q6:

What existential risk (human extinction type event) is currently most likely to have the greatest negative impact on your personal long-term goals, under the condition that nothing is done to mitigate the risk?

Shane Legg: It's my number 1 risk for this century, with an engineered biological pathogen coming a close second (though I know little about the latter).

Q7:  What is the current level of awareness of possible risks from AI, relative to the ideal level?

Shane Legg: Too low...but it could well be a double edged sword: by the time the mainstream research community starts to worry about this issue, we might be risking some kind of arms race if large companies and/or governments start to secretly panic. That would likely be bad.

Q8:  Can you think of any milestone such that if it were ever reached you would expect human-level machine intelligence to be developed within five years thereafter?

Shane Legg: That's a difficult question! When a machine can learn to play a really wide range of games from perceptual stream input and output, and transfer understanding across games, I think we'll be getting close.

25 comments

Comments sorted by top scores.