Proposal: we should start referring to the risk from unaligned AI as a type of *accident risk*

post by Christopher King (christopher-king) · 2023-05-16T15:18:55.427Z · LW · GW · 6 comments

Contents

  Current terminology: no good reference point
  New terminology: tons of reference points!
  Additional benefits
None
6 comments

In the wider political sphere, a lot of people are worried about AI misuse risk. Unaligned AI is not a type of misuse. I think the clearest way to describe this is as an accident risk, in the same sense of the word as industrial accident. In particular, AI existential risk is a type of accident from operating heavy machinery. Using this terminology can immediately help someone not familiar with AI know the category of risk we are talking about, and that in particular it isn't misuse risk.

Note that this isn't intended to replace the term existential risk. Rather, it is meant to be used in addition to that term, and in particular it should be used when contrasting with theoretical misuse risks.

Current terminology: no good reference point

Alice: I hear that you are worried about AI existential risk. So in particular, you are worried about misuse.

Bob: No, the AI kills everyone on its own.

Alice: Is there anything else like this?

Bob: Uhm, Nuclear explosions?

Alice: So a misuse risk? Bob: No, I mean last century they were worried it would set the atmosphere on fire.

Alice: I'm not familiar with that either.
Bob: It's something called instrumental convergence where the AI kills everyone to achieve a goal.

Alice: So misuse risk?

Bob: Not quite, the creators didn't intend for that result.

Alice: I still have no reference point for what you are talking about? I guess I'll need to analyze your arguments more specifically before even understanding the general category of risk you're afraid of it. The probability of me actually doing this is probably like 10%-ish.

New terminology: tons of reference points!

Alice: I hear that you are worried about AI existential risk. So in particular, you are worried about misuse.

Bob: No, I am worried about accident risk.

Alice: oh, so like a car crash or an industrial accident!

Bob: Yes! I'm worried that things will go wrong in ways the creator didn't intend.

Alice: Ah, so you do you think we need more laboratory testing?

Bob: I think even this is risky, because the impact radius will be far larger than that of the lab itself.

Alice: oh, like nuclear weapons testing or biohazards.

Bob: Yes! I think the impact radius may be even bigger than a nuclear explosion though.

Alice: although I do not quite understand how this could work, I understand enough that I want to learn more. And I know understand that you are more worried about accident risk than misuse risk, in the same way that car manufacturers are more worried about car crashes than using cars as weapons. The probability of me actually looking further into this is 30%-ish.

Additional benefits

The way society typically deals with accidents from heavy machinery is much better than the way they are currently treating AI.

In particular, a domain expert having a brilliant safety plan does not suffice for heavy machinery. Neither does the good intentions of the creators. And the possibility that the Chinese might lose limbs to heavy machines also is not sufficient. Rather, you must also loop in measures from the field of risk management.

I also think the term accident risk also avoids some of the mistakes from anthropomorphization. We often model AI as having something analogous to human motivation, due to instrumental convergence. However, from the point of view of the creators, this is still a heavy machinery accident.

I think it's better to start from this point of view, and treat the analogies to human psychology as models. The most important reason for this is so we don't project human irrationality or human morality and desires onto the machine. It is just a machine after all, and there's nothing specific to AI systems that is analogous to those two human qualities.

So in conclusion, if someone asks if you're talking about AI misuse risk, say no, you're talking about AI accident risk.

6 comments

Comments sorted by top scores.