Is a paperclipper better than nothing?

I would choose the paperclipper, but not because I value its intelligence - paperclips are a human invention, and so the paperclipping AI represents a sort of memorial to humanity. A sign that humans once existed, that might last until the heat death of the universe.

My preference for this may perhaps be caused by a confusion, since this is effectively an aesthetic choice for a universe I will not be able to observe, but if other intelligences of a sort that I actually value will not be present either way, this problem doesn't matter as long as it gives me reason enough to prefer one over the other.

I think the point where I start to not prefer the paperclipper is somewhere between the trilobites and the australopithecines, closer to the australopithecine end of that.

Phrased another way: does the existence of any intelligence at all, even a paperclipper, have even the smallest amount of utility above no intelligence at all?

This is a different and cleaner question, because it avoids issues with intelligent life evolving again, and the paperclipper creating other kinds of life and intelligence for scientific or other reasons in the course of pursuing paperclip production.

I would say that if we use a weighted mixture of moral accounts (either from normative uncertainty, or trying to reflect a balance among varied impulses and intuitions), then it matters that the paperclipper could do OK on a number of theories of welfare and value:

• Desire theories of welfare
• Objective list theories of welfare
• Hedonistic welfare theories, depending on what architecture is most conducive to producing paperclips (although this can cut both ways)
• Perfectionism about scientific, technical, philosophical, and other forms of achievement

I'm tempted to choose B just because if I choose A someone will try to use the Axiom of Transitivity to "prove" that I value some very large amount of paperclippers more than some small amount of humans. And I don't.

I might also choose B because the paperclipper might destroy various beautiful nonliving parts of the universe. I'm not sure if I really value beautiful rock formations and such, even if there is no one to view them. I tend to agree that something requires both an objective and subjective component to be truly valuable.

On the other hand, maybe the value for beautiful things I will never see is some sort of "between the margins" value, something that I value, but that my values regarding eudaemonic life are lexically prior to. All other things being equal, I'd prefer a universe with even a tiny amount of eudaemonic life (that isn't suffering or anything like that) to a totally lifeless universe chock-full of unobserved beautiful stuff. But maybe a lifeless pretty universe is more valuable to me than a lifeless ugly universe, all other things being equal.

Yes, it seems like as a human I value systems/agents/whatever that tend to 'reduce entropy' or to 'bring order out of chaos' at least a tiny bit. Thus if everything else is equal I will take the paperclipper.

If the paperclipper is very, very stable, then no paperclipper is better because of higher probability of life->sentience->personhood arising again. If paperclipper is a realistic sapient system, then chances are it will evolve out of paperclipping into personhood, and then the question is whether in expectation it will evolve faster than life otherwise would. Even if by assumption personhood does not arise again, it still depends on particulars, I pick the scenario with more interesting dynamics. If by assumption even life does not arise again, paperclipper has more interesting dynamics.

Choice B, on the grounds that a paperclipper is likely to prevent life as we know it from rising again through whatever mechanism it rose the first time.

For the slightly different case in which life both dies and is guaranteed not to rise naturally ever again, choice A. There's a small but finite chance of the paperclipper slipping enough bits to produce something worthwhile, like life. This is probably less likely than whatever jumpstarted life on Earth happening again.

For the again slightly different case in which life dies and is guaranteed not to rise again through any means including the actions of the paperclipper, back to choice B. There are cool things in the universe that would be made less cool by turning them into paperclips.

Let's try a variant...

Consider two planets, both completely devoid of anything resembling life or intelligence. Anyone who looks at either one of them sees an unremarkable hunk of rock of no particular value. In one of them, the center consists of more unremarkable rock. In the other, however, hidden beneath the surface is a cache that consists of replicas of every museum and library that currently exists on Earth, but which will never be found or seen by anyone (because nobody is going to bother to look that hard at an unremarkable hunk of rock). Does the existence of the second hunk of rock have more value than the first?

If we ignore the possibility of future life arising again after human extinction, paperclipper seems (maybe, a bit) better than extinction because of the possibility of acausal trade between the paperclipper and human values (see this comment and preceding discussion).

The value of possible future life arising by chance is probably discounted by fragility of value (alien values might be not much better than paperclipper's), the risk of it not arising at all or getting squashed by its own existential risks (Fermi paradox), the risk of it also losing its values (e.g. to an UFAI), the astronomical waste of not optimizing the universe in the meantime, and possibly time discounting of the (very distant) future.

(All of this discounting might still be smaller than what takes acausal trade to work out, so it's not clear which choice is better. A cleaner question would compare a paperclipper with a sterile universe.)

• Choose A, and all life and sapience in the solar system (and presumably the universe), save for a sapient paperclipping AI, dies.

• Choose B, and all life and sapience in the solar system, including the paperclipping AI, dies.

I choose A. (OTOH, the difference between U(A) and U(B) is so small that throwing even a small probability of a different C in the mix could easily change that.)

If anyone responds positively, subsequent questions would be which would be preferred, a paperclipper or a single bacteria; a paperclipper or a self-sustaining population of trilobites and their supporting ecology; a paperclipper or a self-sustaining population of australopithecines; and so forth, until the equivalent value is determined.

I'd take the paperclipper over the bacteria. I'd probably take the paperclipper over the trilobites and the australopithecines over the paperclipper, but I'm not very confident about that.

I'd say.. no, the paperclipper probably has negative value.

DataPacRat, I like that you included subsequent questions, and I think there may also be other ways of structuring subsequent questions as well which may also make people think about different answers.

Example: Is a paperclipper better than the something for what likely duration of time?

For instance, take the Trilobites vs paperclipper scenario you mentioned. I am imagining:

A: A solar system that has trilobites for 1 billion years, until it is engulfed by it's sun and everything dies.

B: A solar system that has trilobites in a self-sustaining gaia planet for Eternity.

C: A solar system that has a paperclipping AI for 1 billion years, until it is engulfed by it's sun and all of the paperclips melt.

D: A solar system that has a paperclipping AI that keeps a planet sized mass of paperclips as paperclips for eternity.

If I prefer B>D>A>C, then it seems like I might choose the paperclipper over the trilobites if figure the paperclipper has a 99.99% chance of lasting for an eternity, and the trilobite planet has a 0.01% chance of doing so.

On the other hand, it may be that you want to make an assumption that the paperclipper and the trilobites planet are equally resilient to existential crises for the purposes of this problem.

Second Example: Does the context of the paperclipper AI and it's destruction matter?

Imagine all of humanity, and the rest of the solar system, is going to be engulfed by our sun's super nova soon. We're all going to die. There is one person who is going to be uploaded into an Hansonian EM experimental probe that will be shot away from the blast, a person with a mental disorder from a foreign country who loves making paperclips. (Unfortunately humanity only got to uploading tech 0.91 prior to the super nova- Very few people have any kind of upload capable brain right now.)

You have read several papers from an aquaintance of yours, a respected scientist, who has said multiple times "If you upload him he'll turn into a paperclipper AI, I'm sure of it." you've also read a few independent publications indicating yes, this person is going to be a paperclipper AI if uploaded under uploading tech 0.91, and here is proof.

A research assistant has snuck you a sabotage virus that will destroy the upload probe stealthily after the upload has taken place, saying "I wanted to see if I could be James Bond before I died!" and then commits suicide.

Do you run the sabotage virus? You're going to die either way, but you can either have humanity's last monument be a paperclipper AI or nothing.

At least two explicit differences in this scenario appear to be:

A: The paperclipper AI appears to have some level of popular support. Humanity wouldn't have been spending trillions of dollars making him the only shot they have if not. (If you want more explicit context, imagine that when pressed by TV interviewers, Other scientists have said they explicitly read those papers, said that they understood them, but believed that anything was better than nothing, but they did not have time to explain. Polls indicate the paperclipper was supported 51-49, and clearly with at least some strong opposition, or no one would have bothered to build a sabotage virus.)

B: You don't actually have to make a choice: there is a default choice, which will occur if you don't press the button, which is the paperclipper AI remains.

I'm not sure I actually have answers to either of the questions yet, but the fact that both of them seem like they would make it more acceptable to allow the paperclipper then other options probably indicates at least some as yet unquantified pro-paperclipper thoughts on my part.

I tend to model Paperclippers as conscious, simply because it's easier to use bits of my own brain as a black box. So naturally my instinct is to value it's existence the same as any other modified human mind (although not more than any lives it might endanger.)

However, IIRC, the original "paperclip-maximizer" was supposed to be nonsentient; probably still worth something in the absence of "life", but tricky to assign based on my intuitions (is it even possible to have a sufficiently smart being I don't value the same way I do "conscious" ones?)

In other words, I have managed to confuse my intuitions here.

[D]oes the existence of any intelligence at all, even a paperclipper, have even the smallest amount of utility above no intelligence at all?

Have utility to whom?

I presume when we are all dead, we will have no utility functions.

... Solar system, therefore universe? Does not seem plausible. For no sapient life that will ever develop in the observable universe, sapience needs to be WAY rarer. And the universe is infinite.

How would the answers to these questions affect what you would do differently here and now?

I chose A, on the off-chance that it interprets that as some kind of decision theoretical way that makes it do something I value in return for the favour.

The two scenarios have equal utility to me, as close as I can tell. The paperclipper (and the many more than one copies of itself it would make) would be minds optimized for creating and maintaining paperclips (Though maybe it would kill itself off to create more paperclips eventually?) and would not be sentient. In contrast to you, I think I care about sentience, not sapience. To the very small extent that I saw the paperclipper has a person, rather than a force of clips, I would wish it ill, but only in a half-hearted way, which wouldn't scale to disutility for every paperclip it successfully created.

Is a paperclipper better than nothing?

Nope. I choose B.

Maybe people who think that paperclips aren't boring enough can replace the paperclip maximizer with a supermassive black hole maximizer, as suggested here.

I prefer A. The paperclipping AI will need to contemplate many interesting and difficult problems in physics, logistics, etc. to maximize paperclips. In doing so it will achieve many triumphs I would like a descendant of humanity to achieve. One potential problem I see is that the paperclipper will be crueler to intelligent life in other planets that isn't powerful enough to have leverage over it.

Benatar assimetry between life and death make B the best option. But as his argument is hard to accept, A is better, whatever human values the AI implement.