AI Research Considerations for Human Existential Safety (ARCHES)

post by habryka (habryka4) · 2020-07-09T02:49:27.267Z · LW · GW · 7 comments

This is a link post for https://arxiv.org/pdf/2006.04948.pdf

Contents

7 comments

Andrew Critch's (Academian) and David Krueger's review of 29 AI (existential) safety research directions, each with an illustrative analogy, examples of current work and potential synergies between research directions, and discussion of ways the research approach might lower (or raise) existential risk.

7 comments

Comments sorted by top scores.

comment by Ben Pace (Benito) · 2020-07-09T04:03:33.728Z · LW(p) · GW(p)

Wow, this is long, and seems pretty detailed and interesting. I'd love to see someone write a selection of key quotes or a summary.

Replies from: rohinmshah, capybaralet
↑ comment by Rohin Shah (rohinmshah) · 2020-07-09T06:41:05.828Z · LW(p) · GW(p)

Highlighted in AN #103 [AF · GW] with a summary, though it didn't go into the research directions (because it would have become too long, and I thought the intro + categorization was more important on average).

Replies from: Benito
↑ comment by Ben Pace (Benito) · 2020-07-09T07:39:16.521Z · LW(p) · GW(p)

Thank you!

↑ comment by David Scott Krueger (formerly: capybaralet) (capybaralet) · 2020-09-19T04:54:37.962Z · LW(p) · GW(p)

There is now also an interview with Critch here: https://futureoflife.org/2020/09/15/andrew-critch-on-ai-research-considerations-for-human-existential-safety/

Replies from: Benito
↑ comment by Ben Pace (Benito) · 2020-09-19T23:31:46.074Z · LW(p) · GW(p)

I listened to this yesterday! Was quite interesting, I'm glad I listened to it.

comment by MaxRa · 2020-10-30T10:15:31.914Z · LW(p) · GW(p)

Really enjoyed reading this. The section on "AI pollution" leading to a loss of control about the development of prepotent AI really interested me.

Avoiding [the risk of uncoordinated development of Misaligned Prepotent AI] calls for well-deliberated and respected assessments of the capabilities of publicly available algorithms and hardware, accounting for whether those capabilities have the potential to be combined to yield MPAI technology. Otherwise, the world could essentially accrue “AI-pollution” that might eventually precipitate or constitute MPAI.

  • I wonder how realistic it is to predict this e.g. would you basically need the knowledge to build it to have a good sense for that potential?
  • I also thought the idea of AI orgs dropping all their work once the potential for this concentrates in another org is relevant here - are there concrete plans when this happens?
  • Are there discussion about when AI orgs might want to stop publishing things? I only know of MIRI, but would they advise others like OpenAI or DeepMind to follow their example?
comment by FactorialCode · 2020-07-09T16:59:18.789Z · LW(p) · GW(p)

Nitpick, is there a reason why the margins are so large?