The Utility of Human Atoms for the Paperclip Maximizer

post by avturchin · 2018-02-02T10:06:39.811Z · LW · GW · 21 comments

Contents

21 comments

TL;DR: use of humans’ atoms will increase AI’s expected utility by 0.00000000000000002 of total U.

The iconic example of the existential risk of superintelligence is the so-called paperclip maximizer, that is, a system which maximizes some random goal not aligned with human values. While a real paperclip maximizer is unlikely, convergent dynamic subgoals suggested by Omohundro, imply that almost any non-aligned AI will try to acquire as much computational power as possible, as well as other material resource. Thus, it will need the atoms of humans.

In this article, I will try to estimate the marginal utility of human atoms U¢ compared with total utility of all atoms’ U for the AI.

I assume that:

A) Superintelligent AI which is interested in human atoms, is limited to the Solar System. The reasons for this is that it may need human atoms to build spaceships to travel to the nearest stars, and if it reaches the stars, the value of most human atoms will be lost.

B) Superintelligent AI has powerful nanotechnology, which could be used to do whatever it wants with atoms, like build large space ships, deconstruct planets down to their final atoms etc.

C) The only resource that AI is interested in is atoms—not energy, not information etc.

D) The utility of atoms’ U is increasing linearly with increases in the available mass.

The weight of Solar System planets is 0.135% of the Sun’s mass, which is 1.98855×10E30 kg, and the mass of all planets is 2.6E1027kg, where most of this mass is Jupiter. Earth’s mass is 5.97237×10E24 kg.

The total mass of human beings on Earth is 70kg х 7.5x10E9 people = 5.25x10E11kg.

In that case, human atoms constitute marginal utility U’= 2x10E-16U. That is, use of the humans’ atoms will increase the AI’s utility by 0.00000000000000002 of total U.

But this is obviously flawed, as humans can’t survive in a vacuum, they need the whole Earth!

Imagine that AI wants to deconstruct the Earth for its atoms, but also decided to preserve human lives. It has two options:

1) Upload everybody into smaller computers. Based on various estimations of the Launder limit, and the computational capacity of human brains, the size of these computers will be different, but they could easily be 1000 times less than human bodies.

2) AI decides not to upload humans, but to build a space station, where humans can live approximately the same life as they do currently. As the typical weight of a human house is something like 10 tons, and assuming very effective nanotech, such a space station may require hardware weighing 1000 times more kg for every human kg (or perhaps even less). It will weigh 5.25x10E14kg.

Option (1) is a million times more economical than the option (2) for the AI. But even in the case of option (2), the marginal utility of human atoms U’= 2x10E-13U.

If the number of the atoms linearly translates into the speed of the start of the galactic colonization (Armstrong & Sandberg, 2013), and AI needs 1 billion seconds (30 years) to convert all the Solar System planets into space ships, the delay caused by preserving humans on a space station will be around 0.2 milliseconds.

Given Bostrom’s astronomical waste idea (Bostrom, 2003), that may be not small after all, as it will increase the sphere of the AI’s reach by 150 km, and after billions of years it will correspond to very large volume (assuming the size of the universe is like 10E21 light milliseconds, and the number of stars in it is around 10E24, times an economy of 0.2 milliseconds, could mean gain of more than 1000 stars, equal to hundreds of solar masses.)

Even Friendly AI may deconstruct humans for their atoms in the AI’s early stages, and as such sacrificy will translate in the higher total number of sentient beings in the universe at the end.

In the another work (Turchin, 2017), I suggested that “price” of human atoms for AI is so infinitely small, that it will not kill humans for their atoms, if it has any infinitely small argument to preserve humans. Here I suggested more detailed calculation.

This argument may fail if we add the changing utility of human atoms over time. For early AI, Earth’s surface is the most available source of atoms, and organic matter is the best source of carbon and energy (Freitas, 2000). Such an AI may bootstrap its nanotech infrastructure more quickly if it does not care about humans. However, AI could start uploading humans, or at least freezing their brains in some form of temporary cryostasis, even in the very early stages of its development. It that case AI may be acquire most human atoms without killing them.

Armstrong, S., & Sandberg, A. (2013). Eternity in six hours: intergalactic spreading of intelligent life and sharpening the Fermi paradox. Acta Astronautica, 89, 1–13.

Bostrom, N. (2003). Astronomical waste: The opportunity cost of delayed technological development. Utilitas, 15(3), 308–314.

Freitas, R. (2000). Some Limits to Global Ecophagy by Biovorous Nanoreplicators, with Public Policy Recommendations. Foresight Institute Technical Report.

Turchin, A. (2017). Messaging future AI. Retrieved from https://goo.gl/YArqki

21 comments

Comments sorted by top scores.