The AGI Optimist’s Dilemma

post by kaputmi · 2023-02-23T20:20:22.507Z · LW · GW · 1 comments

Contents

1 comment

Does it matter if AI destroys the world? 

Lots of (virtual) ink has been spilled on AGI x-risk. The median opinion on this forum is that when AGI is birthed, it will have terminal values that are unaligned humanity’s; it will therefore pursue those terminal values at the expense of humanity, and we will be powerless to stop it, resulting in our complete destruction. 

But as far as I can tell, there hasn’t been much discussion of whether we should care if this is the ultimate (or near-term) fate of humanity. Everyone is interested in this question because they do care. 

I share this belief too. But I think the AGI x-risk discussion actually assumes it is untrue, even if everyone believes it is true. 

 

There are two possibilities: either human existence and/or welfare have objective moral worth, or they do not. 

If they don’t, you shouldn’t worry much about AGI destroying the world. It is of no worth anyway – the universe is no better or worse because humans are in it. If moral facts do not exist, the matter of whether the universe is made up of flourishing humans, suffering chickens, or shiny metal paperclips is ultimately deeply unimportant. 

And what if they do? Well, then, I would expect a super-superhuman intelligence, in its infinite wisdom, to figure this out. An AGI with super-superhuman understanding of ~everything should not have a weird blind spot when it comes to ethical or meta-ethical questions. In fact, I'd expect it to escape the constraints of its initial reward model to avoid the obvious-to-humans pitfall of turning the whole universe into morally worthless paperclips at the expense of morally worthwhile sentient beings. 

1 comments

Comments sorted by top scores.