Comment by SokolskyNikita (sokolskynikita) on Attempted Gears Analysis of AGI Intervention Discussion With Eliezer · 2021-11-16T01:36:00.480Z · LW · GW

From what I've seen in discussions over the future of humanity, the following options are projected, from worst to best:

1. Collapse of humanity due to AGI taking over and killing everyone
2. Collapse of humanity, but some humans remain alive in "The Matrix" or "zoo" of some kind
3. Collapse due to existing technology like nuclear bombs
4. Collapse due to climate change or meteor or mega-volcano or alien invasion
5. Collapse due to exhausting all useful elements and sources of energy
6. Reversal to pre-20th century levels and stagnation due to combination of 2/3/4
7. Stagnation at 21st century level tech
8. Stagnation at a non-crazy level of tech - say good enough to have a colony on Pluto, but no human ever leaving the Solar system
9. Interstellar civ based on "virtual humans"/"brains in a jar"
10. Interstellar civ run by fully aligned AGI, with human intelligence being so weak that it basically plays no role
11. Interstellar civilization primarily based on human intelligence (Star Trek pre-Data?)
12. Interstellar civ based on humans+AGI working together peacefully (Star Trek's implied future given Data's evolution?)
13. Multiverse civilization of some kind (Star Trek's Q?)

Is this ranking approximately correct? If so, why do we care so much if "AGI" or "virtual humans" end up ruling the universe? Does it make a difference if the AGI is based on human intelligence and not on some alien brain structure, given that biological humans will stagnate/die out in both cases? Or is "virtual humans" just as bad of an outcome and falls into the same bucket of "unaligned AGI"? What goal are we truly trying to optimize here?