How to deal with a misleading conference talk about AI risk?

post by rmoehn · 2019-06-27T21:04:32.828Z · score: 21 (9 votes) · LW · GW · 13 comments

This is a question post.

Contents

  Answers
    27 Vaniver
None
12 comments

Does it make sense to give a public response? Who would be able to do it?

The conference organizer, who had asked me to evaluate the talk, offered to interview me to set things straight. However, I don't know if that is sensible, and given my level of experience, I'm afraid I would misrepresent AI risk myself.

To be concrete: the talk was Should We Fear Intelligent Machines? by Gerald Sussman of SICP fame. He touched on important research questions and presented some interesting ideas. But much of what he said was misleading and not well-reasoned.

In response to the comments I add specifics. This is the same as I sent to the conference organizer, who had asked me for an evaluation. Note that this evaluation is separate from the interview mentioned above. The evaluation was private, the interview would be public.

Answers

answer by Vaniver · 2019-06-27T22:01:52.060Z · score: 27 (11 votes) · LW · GW

I've given responses before where I go into detail about how I disagree with some public presentation on AI; the primary example is this one from January 2017, which Yvain also responded to. Generally this is done after messaging the draft to the person in question, to give them a chance to clarify or correct misunderstandings (and to be cooperative instead of blindsiding them).

I generally think it's counterproductive to 'partially engage' or to be dismissive; for example, one consequence of XiXiDu's interviews with AI experts was that some of them (that received mostly dismissive remarks in the LW comments) came away with the impression that people interested in AI risk were jerks who aren't really worth engaging with. For example, I might think someone is confused if they think climate change is more important than AI safety, but I don't think that it's useful to just tell them that they're confused or off-handedly remark that "of course AI safety is more important," since the underlying considerations (like the difference between catastrophic risks and existential risks) are actually non-obvious.

13 comments

Comments sorted by top scores.

comment by Vaniver · 2019-06-27T22:27:29.538Z · score: 15 (7 votes) · LW · GW

I've read the slides of the underlying talk, but not listened to it. I currently don't expect to write a long response to this. My thoughts about points the talk touches on:

  • Existential risk vs. catastrophic risk. Often, there's some question about whether or not existential risks are even possible. On slide 7 and 8 Sussman identifies a lot of reasons to think that humans cause catastrophic risks (ecological destruction could possibly kill 90% of people, but seems much more difficult for it to kill 100% of people), and the distinction between the two is only important if you think about the cosmic endowment. But of course if we think AI is an existential threat, and we think humans make AI, then it is true that humans present an existential threat to ourselves. I also note here that Sussman identifies synthetic biology as possibly an existential risk, which raises the question of why an AI couldn't be a source of the existential risk presented by synthetic biology. (If an AI is built that wants to kill us, and that weapon is lying around, then we should be more concerned about AI because it has an opportunity.)
  • Accident risk vs. misuse risk. This article talks about it some, but the basic question is "will advanced AI cause problems because it did something no one wanted (accidents), or something bad people wanted (misuse)?". Most technical AI safety research is focused on accident risk, for reasons that are too long to describe here, but it's not crazy to be concerned about misuse risk, which seems to be Sussman's primary focus. I also think the sort of accident risks that we're concerned about require much deeper solutions that the normal sorts of bugs or accidents that one might imagine on hearing about this; the autonomous vehicle accident that occupies much of the talk is not a good testbed for thinking about what I think of as 'accident risk' and instead one should focus on something like the 'nearest unblocked strategy' article and related things.
  • Openness vs. closure. Open software allows for verifiability; I can know that lots of people have evaluated the decision-making of my self-driving car, rather than just Tesla's internal programming team. But also open software allows for copying and modification; the software used to enable drones that deliver packages could be repurposed to enable drones that deliver hand grenades. If we think a technology is 'dual use', in that it can both be used to make things better (like printing DNA for medical treatments) and worse (like printing DNA to create new viruses), we generally don't want those technologies to be open, and instead have carefully monitored access to dissuade improper use.
  • Solving near-term problems vs. long-term problems. Many people working on technical AI safety focus on applications with immediate uses, like the underlying math for how autonomous vehicles might play nicely with human drivers, and many people working on technical AI safety focus on research that will need to be done before we can safely deploy advanced artificial intelligence. Both of these problems seem real to me, and I wouldn't dissuade someone from working on near-term safety work (especially if the alternative is that they do capabilities work!). I think that the 'long-term' here is measured in "low numbers of decades" instead of "low numbers of centuries," and so it might be a mistake to call it 'long-term,' but the question of how to do prioritization here is actually somewhat complicated, and it seems better if we end up in a world where people working on near-term and long-term issues see each other as collaborators and allies instead of competitors for a limited supply of resources or attention.
comment by Kaj_Sotala · 2019-06-27T21:48:37.126Z · score: 3 (1 votes) · LW · GW

Hi rmoehn,

I didn't look at the contents of that talk yet, but I felt uncomfortable about a specific speaker/talk being named and singled out for the target of rather hard-to-respond-to criticism (consider how you might take it if you came across a forum discussion calling your talk misleading and not well-reasoned, without going into any specifics), so I edited out those details for now.

I feel that the AI risk community should do its best to build friendly rather than hostile relationships with mainstream computer science researchers. In particular, there have been cases before where researchers looked at how their work was being discussed on LW, picked up a condescending tone, and decided that LW/AI risk people were not worth engaging with. Writing a response outlining one's disagreement to the talk (in the style of e.g. "Response to Cegłowski on superintelligence") wouldn't be a problem as it communicates engagement with the talk. But if we are referencing people's work in a manner which communicates a curt dismissal, I think we should be careful about naming specific people.

The question in general is fine, though. :)

comment by rmoehn · 2019-06-28T21:49:50.198Z · score: 21 (5 votes) · LW · GW

I've added specifics. I hope this improves things. If not, feel free to edit it out.

Thanks for pointing out the problems with my question. I see now that I was wrong to combine strong language with no specifics and a concrete target. I would amend it, but then the context for the discussion would be gone.

comment by Kaj_Sotala · 2019-06-29T05:29:06.870Z · score: 6 (3 votes) · LW · GW

Thanks, those specifics are great!

comment by habryka (habryka4) · 2019-06-27T22:06:13.239Z · score: 16 (5 votes) · LW · GW

We briefly discussed this internally. I reverted Kaj's edit since I think we should basically never touch other user's content unless it is dealing with some real information hazards, or threatens violence or doxxes a specific individual (and probably some weird edge cases that are rare and can't easily enumerate, but which "broad PR concerns" are definitely not an instance of).

(We also sometimes edit user's content if there is some broken formatting or something in that reference class, though that feels like a different kind of thing)

comment by Raemon · 2019-06-27T22:42:47.878Z · score: 17 (5 votes) · LW · GW

Probably useful to clarify so people can understand how moderation works:

The semi-official norms around moderation tend to be "moderators have discretion to take actions without waiting for consensus, but then should report actions they took to other moderators for sanity checking." (I don't think this is formal policy but I'd personally endorse it being policy. Waiting for consensus on things often makes it impossible to take action in time sensitive situations, but checking in after the fact gets you most of the benefit)


comment by Elo · 2019-06-27T22:59:47.210Z · score: 4 (2 votes) · LW · GW

This should be advertised in meta.

comment by Ben Pace (Benito) · 2019-06-27T22:12:28.031Z · score: 7 (3 votes) · LW · GW

And also we edit other users' content when they give us permission to, which happens a lot.

comment by Zack_M_Davis · 2019-06-28T06:21:39.567Z · score: 11 (3 votes) · LW · GW

(Note: posted after the parent was retracted.)

consider how you might take it if you came across a forum discussion calling your talk misleading and not well-reasoned, without going into any specifics

I would be grateful for the free marketing! (And entertainment—internet randos' distorted impressions of you are fascinating to read.) Certainly, it would be better for people to discuss the specifics of your work, but it's a competitive market for attention out there: vague discussion is better than none at all!

there have been cases before where researchers looked at how their work was being discussed on LW, picked up a condescending tone, and decided that LW/AI risk people were not worth engaging with

If I'm interpreting this correctly, this doesn't seem very consistent with the first paragraph? First, you seem to be saying that it's unfair to Sussman to make him the target of vague criticism ("consider how you might take it"). But then you seem to saying that it looks bad for "us" (you know, the "AI risk community", Yudkowski's robot cult, whatever you want to call it) to be making vague criticisms that will get us written off as cranks ("not worth engaging with"). But I mostly wouldn't expect both concerns to be operative in the same world—in the possible world where Sussman feels bad about being named and singled out, that means he's taking "us" seriously enough for our curt dismissal to hurt, but in the possible world where we're written off as cranks, then being named and singled out doesn't hurt.

(I'm not very confident in this analysis, but it seems important to practice trying to combat rationalization in social/political thinking??)

comment by Kaj_Sotala · 2019-06-28T09:45:57.361Z · score: 10 (4 votes) · LW · GW
But I mostly wouldn't expect both concerns to be operative in the same world—in the possible world where Sussman feels bad about being named and singled out, that means he's taking "us" seriously enough for our curt dismissal to hurt, but in the possible world where we're written off as cranks, then being named and singled out doesn't hurt.

The world can change as a result of one of the concerns. At first you're taking someone seriously (or might at least be open to taking them seriously), then they say something hurtful, then you write them off to make it hurt less. Sour grapes.

Also, the reactions of people who are not being directly criticized but who respect the person being criticized are also important. Even if the target of the criticism never saw it, other people in the target's peer group may also feel disrespected and react in a similar way. (This is not speculation - I've seen various computer scientists have this reaction to writings on LW, many times.)

comment by jimrandomh · 2019-06-27T21:54:54.714Z · score: 4 (2 votes) · LW · GW

Moderators are discussing this with each other now. We do not have consensus on this.