Existentially relevant thought experiment: To kill or not to kill, a sniper, a man and a button.

post by AlexFromSafeTransition · 2023-08-14T10:53:10.008Z · LW · GW · 6 comments

Contents

6 comments

There is a room with one window. Inside is a man. On the ceiling there is an interesting button. What happens when it is pressed? Everybody dies, except for the man and 1000 people he gets to pick. The button is not visible from outside the room. The man sometimes walks past the window. The only thing we know about the man, is that he is above average in intelligence and that he is healthy. The button is on the ceiling, such that it is extremely unlikely that it is pressed by accident. It would be a conscious choice by the man to press or not. You cannot communicate with the man, and it is not possible to use other technology than is described here. There is also a sniper aiming at the window. The sniper is awaiting your command: to kill or not to kill, that is the question. There are no negative consequences for you for choosing to kill or not to kill. It is just up to you to think what is the right thing to do. Given that all of the above would be true, What do you choose?

A, not to kill. The man has not yet done anything wrong. The man is a conscious being, so he has moral value. You choose to risk your own life and that of the rest of us to give this man a chance.

B, kill. What motive could the man have to want to press the button? Could be freedom to reshape the earth as he and his companions see fit. Could be a strong opinion that humanity is on balance bad for the inhabitants of earth, combined with a love for nature and a wish to see wild animal populations and vegetation retake the earth to become an environmental paradise. The man could be a criminal wanted by governments, he could want this manhunt to end. There is a fair chance this choice to kill would save your life and that of the ones you love. You choose that taking away this risk is more important than the man's life.

Please choose before reading on and write a comment saying A or B, you could add why but also valuable without.

...

...

...

...

...

I have asked 20 people as of now and the response is 16 B, kill, and one I don't know and three A, not to kill. All live in the Netherlands. I find this near consensus interesting and practically relevant to the future of AI. I have set this up to be relevant to the situation where the man in the room is analogous to an AGI (Artificial General Intelligence) that is thought to have feelings / consciousness and thus moral value. Like what MIT professor Max Tegmark talks about, I am both open to the possibility of an AGI that has no subjective experience and thus no moral value, and to the possibility that the AGI does have moral value. It is intuitive in modern culture for humans to see a random other human as a moral patient worthy of legal consideration. That is why I went for a random man to play the part of the AGI instead of using an AI directly, since it is unintuitive to think of an AGI as being a moral patient to many people (in my experience of talking about this). It would help to imagine the AGI as having a richness in subjective experience similar to your own, as I think we should prepare for this possibility. The button would be analogous to any human extinction inducing capability (like creating a super virus / bioweapon). The AGI could in such a situation keep itself functional and for example a number of copies. The sniper represents our chance to switch it off, but only when we still can, before it made secret copies elsewhere. That window of opportunity is represented by only being able to shoot the man when in view of the window. Stuart Russell advocated for a kill-switch on a general system in the senate hearing recently found here on YouTube. That is Russell advocating for positioning the sniper. The scenario is the most relevant when there is no aligned AGI yet, that wants to protect us. Another relevant situation is where the AGI is not in fact conscious, but is able to successfully convince many people that it is, where some will want to come to its aid. Of course, this fictional situation is an oversimplification and an important difference with reality is that an AI can also choose to improve our living conditions. But this small experiment does give an interesting result, that the majority thinks being innocent and of moral value is not enough to be allowed to live when having access to such a metaphorical button.

I think this is also relevant to the question, should AI be given (human) rights? Can we peacefully coexist with superintelligences that are free or in other words, where their degree of alignment to what we want is unknown. I answer no, because I think it utterly foolish to put our faith in the hands of an AI that has this metaphorical button and might press it. We cannot keep such an AI aligned like we align humans or corporations, with laws and threats of punishment or jail because the AI can be in an unknown amount of places at once, have copies of itself. This is in stark contrast with what the movies I Robot, Free Guy, Artifice Girl and probably more will want you to believe. Important characters in these movies are pro giving full freedom to an AGI with unknown goals and values, similar to the man in the room. I think it sensible that only when we have super strong guarantees that an AGI will want to protect us and help us, forever, that we can think of giving it rights and autonomous access to the real world. Having feelings / consciousness and thus moral value is not enough in my opinion. This is in contrast with the Google engineer that lost his job to tell the world their chatbot LaMDA was sentient. Source here. He organized a meeting between a lawyer and the chatbot with the goal of protecting the rights of the AI, so there are people who think in this, in my view, existentially dangerous way.

If you answered B, please consider spreading this post. To people in Hollywood or AI builders or other relevant people. I write this in hope of decreasing the power and size of the group of people that are in favor of freeing the AI and giving it rights. And to increase our odds of surviving all this. I also write this with some fear of retaliation from a future misaligned Artificial General Intelligence reading this.

6 comments

Comments sorted by top scores.