What are the reasons to *not* consider reducing AI-Xrisk the highest priority cause?

infinite ethics (and maybe other fundamental ethical questions, e.g. to do with moral uncertainty)

Why does this reduce the priority of working on AI risk? Is it just that it makes the problem harder and hence less tractable?

simulation hypothesis

I've written down some thoughts on how the possibility of being in simulations affects my motivation to work on AI risk: Beyond Astronomical Waste [LW · GW], Comment 1 [LW(p) · GW(p)], Comment 2 [LW(p) · GW(p)].

There are two questions which I think are important to distinguish:

Is AI x-risk the top priority for humanity?

Is AI x-risk the top priority of some individual?

The first question is perhaps extremely important in a general sense. However, the second question is, I think, more useful since it provides actionable information to specific people. Of course, the difficulty of answering the second question is that it depends heavily on individual factors, such as

• The ethical system of the individual which they are using the evaluate the question.
• The specific talents, and time-constraints of the individual.

I also partially object to placing AI x-risk into one entire bundle. There are many ways that people can influence the development of artificial intelligence:

• Technical research
• Social research to predict and intervene on governance for AI
• AI forecasting to help predict which type of AI will end up existing and what their impact will be

Even within technical research, it is generally considered that there are different approaches:

• Machine learning research with an emphasis on creating systems that could scale to superhuman capabilities while remaining aligned. This would include, but would not be limited to
• Paul Christiano-style research, such as expanding iterated distillation and amplification
• ML transparency
• ML robustness to distributional shifts
• Fundamental mathematical research which could help dissolve confusion about AI capabilities and alignment. This includes
• Uncovering insights into decision theory
• Discovering the necessary conditions for a system to be value aligned
• Examining how systems could be stable upon reflection, such as after self-modification

S-risks

Not necessarily a reason to deprioritize AI x-risk work, given that unaligned AI could be bad from an s-risk perspective as well:

Pain seems to have evolved because it has a functional purpose in guiding behavior: evolution having found it suggests that pain might be the simplest solution for achieving its purpose. A superintelligence which was building subagents, such as worker robots or disembodied cognitive agents, might then also construct them in such a way that they were capable of feeling pain - and thus possibly suffering (Metzinger 2015) - if that was the most efficient way of making them behave in a way that achieved the superintelligence’s goals.

Humans have also evolved to experience empathy towards each other, but the evolutionary reasons which cause humans to have empathy (Singer 1981) may not be relevant for a superintelligent singleton which had no game-theoretical reason to empathize with others. In such a case, a superintelligence which had no disincentive to create suffering but did have an incentive to create whatever furthered its goals, could create vast populations of agents which sometimes suffered while carrying out the superintelligence’s goals. Because of the ruling superintelligence’s indifference towards suffering, the amount of suffering experienced by this population could be vastly higher than it would be in e.g. an advanced human civilization, where humans had an interest in helping out their fellow humans. [...]

If attempts to align the superintelligence with human values failed, it might not put any intrinsic value on avoiding suffering, so it may create large numbers of suffering subroutines.