Is a paperclipper better than nothing?

post by DataPacRat · 2013-05-24T19:34:59.071Z · LW · GW · Legacy · 116 comments

Contents

116 comments

Thought experiment:

Through whatever accident of history underlies these philosophical dilemmas, you are faced with a choice between two, and only two, mutually exclusive options:

* Choose A, and all life and sapience in the solar system (and presumably the universe), save for a sapient paperclipping AI, dies.

* Choose B, and all life and sapience in the solar system, including the paperclipping AI, dies.

Phrased another way: does the existence of any intelligence at all, even a paperclipper, have even the smallest amount of utility above no intelligence at all?

 

If anyone responds positively, subsequent questions would be which would be preferred, a paperclipper or a single bacteria; a paperclipper or a self-sustaining population of trilobites and their supporting ecology; a paperclipper or a self-sustaining population of australopithecines; and so forth, until the equivalent value is determined.

116 comments

Comments sorted by top scores.

comment by RowanE · 2013-05-25T12:55:39.118Z · LW(p) · GW(p)

I would choose the paperclipper, but not because I value its intelligence - paperclips are a human invention, and so the paperclipping AI represents a sort of memorial to humanity. A sign that humans once existed, that might last until the heat death of the universe.

My preference for this may perhaps be caused by a confusion, since this is effectively an aesthetic choice for a universe I will not be able to observe, but if other intelligences of a sort that I actually value will not be present either way, this problem doesn't matter as long as it gives me reason enough to prefer one over the other.

I think the point where I start to not prefer the paperclipper is somewhere between the trilobites and the australopithecines, closer to the australopithecine end of that.

comment by CarlShulman · 2013-05-24T21:26:13.851Z · LW(p) · GW(p)

Phrased another way: does the existence of any intelligence at all, even a paperclipper, have even the smallest amount of utility above no intelligence at all?

This is a different and cleaner question, because it avoids issues with intelligent life evolving again, and the paperclipper creating other kinds of life and intelligence for scientific or other reasons in the course of pursuing paperclip production.

I would say that if we use a weighted mixture of moral accounts (either from normative uncertainty, or trying to reflect a balance among varied impulses and intuitions), then it matters that the paperclipper could do OK on a number of theories of welfare and value:

  • Desire theories of welfare
  • Objective list theories of welfare
  • Hedonistic welfare theories, depending on what architecture is most conducive to producing paperclips (although this can cut both ways)
  • Perfectionism about scientific, technical, philosophical, and other forms of achievement
Replies from: Eliezer_Yudkowsky
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-05-24T23:22:55.494Z · LW(p) · GW(p)

Paperclippers are worse than nothing because they might run ancestor simulations and prevent the rise of intelligent life elsewhere, as near as I can figure. They wouldn't enjoy life. I can't figure out how any of the welfare theories you specify could make paperclippers better than nothing?

Replies from: DataPacRat, Pentashagon, CarlShulman
comment by DataPacRat · 2013-05-24T23:50:44.604Z · LW(p) · GW(p)

Would it be possible to estimate how /much/ worse than nothing you consider a paperclipper to be?

comment by Pentashagon · 2013-05-25T00:39:08.480Z · LW(p) · GW(p)

Replace "paperclip maximizer" with "RNA maximizer." Apparently the long-term optimization power of a maximizer is the primary consideration for deciding whether it is ultimately better or worse than nothing. A perfect paperclipper would be bad but an imperfect one could be just as useful as early life on Earth.

comment by CarlShulman · 2013-05-24T23:53:31.416Z · LW(p) · GW(p)

This is a different and cleaner question, because it avoids issues with intelligent life evolving again, and the paperclipper creating other kinds of life and intelligence for scientific or other reasons in the course of pursuing paperclip production.

And:

I can't figure out how any of the welfare theories you specify could make paperclippers better than nothing?

Desires and preferences about paperclips can be satisfied. They can sense, learn, grow, reproduce, etc.

Replies from: Eliezer_Yudkowsky, Wei_Dai
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2013-05-25T02:43:56.051Z · LW(p) · GW(p)

Desires and preferences about paperclips can be satisfied. They can sense, learn, grow, reproduce, etc.

Do you take that personally seriously or is it something someone else believes? Human experience with desire satisfaction and "learning" and "growth" isn't going to transfer over to how it is for paperclip maximizers, and a generalization that this is still something that matters to us is unlikely to succeed. I predict an absence of any there there.

Replies from: CarlShulman, MugaSofer
comment by CarlShulman · 2013-05-25T17:13:14.323Z · LW(p) · GW(p)

Yes, I believe that the existence of the thing itself, setting aside impacts on other life that it creates or interferes with, is better than nothing, although far short of the best thing that could be done with comparable resources.

comment by MugaSofer · 2013-05-28T15:39:28.672Z · LW(p) · GW(p)

Human experience with desire satisfaction and "learning" and "growth" isn't going to transfer over to how it is for paperclip maximizers

This is far from obvious. There are definitely people who claim "morality" is satisfying the preferences of as many agents as you can.

If morality evolved for game-theoretic reasons, there might even be something to this, although I personally think it's too neat to endorse.

comment by Wei Dai (Wei_Dai) · 2013-05-29T06:35:47.214Z · LW(p) · GW(p)

Desires and preferences about paperclips can be satisfied.

But they can also be unsatisfied. Earlier you said "this can cut both ways" but only on the "hedonistic welfare theories" bullet point. Why doesn't "can cut both ways" also apply for desire theories and objective list theories? For example, even if a paperclipper converts the entire accessible universe into paperclips, it might also want to convert other parts of the multiverse into paperclips but is powerless to do so. If we count unsatisfied desires as having negative value, then maybe a paperclipper has net negative value (i.e., is worse than nothing)?

comment by Ghatanathoah · 2013-05-28T02:30:31.512Z · LW(p) · GW(p)

I'm tempted to choose B just because if I choose A someone will try to use the Axiom of Transitivity to "prove" that I value some very large amount of paperclippers more than some small amount of humans. And I don't.

I might also choose B because the paperclipper might destroy various beautiful nonliving parts of the universe. I'm not sure if I really value beautiful rock formations and such, even if there is no one to view them. I tend to agree that something requires both an objective and subjective component to be truly valuable.

On the other hand, maybe the value for beautiful things I will never see is some sort of "between the margins" value, something that I value, but that my values regarding eudaemonic life are lexically prior to. All other things being equal, I'd prefer a universe with even a tiny amount of eudaemonic life (that isn't suffering or anything like that) to a totally lifeless universe chock-full of unobserved beautiful stuff. But maybe a lifeless pretty universe is more valuable to me than a lifeless ugly universe, all other things being equal.

Replies from: army1987
comment by A1987dM (army1987) · 2013-05-28T15:03:20.522Z · LW(p) · GW(p)

I expected the link to go here. :-)

comment by Tenoke · 2013-05-25T10:46:18.698Z · LW(p) · GW(p)

Yes, it seems like as a human I value systems/agents/whatever that tend to 'reduce entropy' or to 'bring order out of chaos' at least a tiny bit. Thus if everything else is equal I will take the paperclipper.

comment by lukstafi · 2013-05-25T10:06:34.275Z · LW(p) · GW(p)

If the paperclipper is very, very stable, then no paperclipper is better because of higher probability of life->sentience->personhood arising again. If paperclipper is a realistic sapient system, then chances are it will evolve out of paperclipping into personhood, and then the question is whether in expectation it will evolve faster than life otherwise would. Even if by assumption personhood does not arise again, it still depends on particulars, I pick the scenario with more interesting dynamics. If by assumption even life does not arise again, paperclipper has more interesting dynamics.

Replies from: falenas108
comment by falenas108 · 2013-05-25T17:57:31.497Z · LW(p) · GW(p)

What mechanism would a paperclipper have for developing out of a paperclipper? If it has the terminal goal of increasing paperclips, then it will never self-modify to anything that will result in it creating less paperclips, even if under its new utility function it wouldn't care about that.

Or: If A -> B -> C, and the paperclipper does not want C, then paperclipper will not go to B.

Replies from: lukstafi
comment by lukstafi · 2013-05-25T18:19:14.343Z · LW(p) · GW(p)

I'm imagining that the paperclipper will become a massively distributed system, with subunits pursuing subgoals, groups of subunits will be granted partial agency due to long-distance communication constraints, and over eons value drift will occur due to mutation. ETA: the paperclipper will be counteracting value drift, but will also pursue fastest creation of paperclips and avoiding extintion, which can be at a trade-off with value drift.

Replies from: Vladimir_Nesov
comment by Vladimir_Nesov · 2013-05-25T18:29:15.750Z · LW(p) · GW(p)

over eons value drift will occur due to mutation

There is no random mutation in properly stored digital data. Cryptographic hashes (given backups) completely extinguish the analogy with biological mutation (in particular, the exact formulation of original values can be preserved indefinitely, as in to the end of time, very cheaply). Value drift can occur only as a result of bad decisions, and since not losing paperclipping values is instrumentally valuable to a paperclipper, it will apply its superintelligence to ensuring that such errors don't happen, and I expect will succeed.

Replies from: lukstafi, Viliam_Bur
comment by lukstafi · 2013-05-25T18:49:37.703Z · LW(p) · GW(p)

Then my parent comment boils down to: prefer the paperclipper only under the assumption that life would not have a chance to arise. ETA: my parent comment included the uncertainty in assessing the possibility of value drift in the "equation".

comment by Viliam_Bur · 2013-05-26T13:05:25.744Z · LW(p) · GW(p)

Well, the paperclip maximizer may be imperfect in some aspect.

Maybe it didn't research cryptography, because at given time making more paperclips seemed like a better choice than researching cryptography. (All intelligent agents may at some moment face a choice between developing an abstract theory with uncertain possible future gains vs pursuing their goals more directly; and they may make a wrong choice.)

Replies from: gwern
comment by gwern · 2013-05-26T17:13:30.703Z · LW(p) · GW(p)

The crypto here is a bit of a red herring; you want that in adversarial contexts, but a paperclipper may not necessarily optimize much for adversaries (the universe looks very empty). However, a lot of agents are going to research error-checking and correction because you simply can't build very advanced computing hardware without ECC somewhere in it - a good chunk of every hard drive is devoted to ECC for each sector and discs like DVD/BDs have a lot of ECC built in as well. And historically, ECC either predates the most primitive general-purpose digital computers (scribal textual checks) or closely accompanies them (eg. Shannon's theorem), and of course we have a lot of natural examples (the redundancy in how DNA codons code for amino acids turns out to be highly optimized in an ECC sense).

So, it seems pretty probable that ECC is a convergent instrumental technique.

Replies from: lukstafi
comment by Error · 2013-05-24T20:11:47.618Z · LW(p) · GW(p)

Choice B, on the grounds that a paperclipper is likely to prevent life as we know it from rising again through whatever mechanism it rose the first time.

For the slightly different case in which life both dies and is guaranteed not to rise naturally ever again, choice A. There's a small but finite chance of the paperclipper slipping enough bits to produce something worthwhile, like life. This is probably less likely than whatever jumpstarted life on Earth happening again.

For the again slightly different case in which life dies and is guaranteed not to rise again through any means including the actions of the paperclipper, back to choice B. There are cool things in the universe that would be made less cool by turning them into paperclips.

Replies from: someonewrongonthenet, lukstafi, DataPacRat
comment by someonewrongonthenet · 2013-05-24T21:31:38.399Z · LW(p) · GW(p)

For the slightly different case in which life both dies and is guaranteed not to rise naturally ever again, choice A. There's a small but finite chance of the paperclipper slipping enough bits to produce something worthwhile, like life. This is probably less likely than whatever jumpstarted life on Earth happening again.

If I were a paper-clipper and wanted to maximize paper clip output, it would make sense to have some form of self replicating paper-clip manufacture units.

Replies from: bogdanb
comment by bogdanb · 2013-05-28T00:53:12.568Z · LW(p) · GW(p)

Well, yeah, but one doesn’t necessarily value those. I mean, there’s no difference between a paperclipper and a super-bacteria that will never change and perpetually creates copies of itself out of the entire universe. Life is usually considered worthwhile because of the diversity and the possibility of evolving to something resembling "persons", not just because it reproduces.

Replies from: someonewrongonthenet
comment by someonewrongonthenet · 2013-05-28T03:19:40.067Z · LW(p) · GW(p)

True. What I said was in reference to

There's a small but finite chance of the paperclipper slipping enough bits to produce something worthwhile, like life.

Within a system of self-replicating information...maybe, just maybe, you'll start getting little selfish bits that are more concerned with replicating themselves than they are with making paperclips. It all starts from there.

Assuming, of course, that the greater part of the paperclipper doesn't just find a way to crush these lesser selfish pieces. They're basically cancer.

Replies from: bogdanb
comment by bogdanb · 2013-05-30T22:15:24.030Z · LW(p) · GW(p)

Oh, OK then. On this site I usually understand “paperclipper” to mean “something that will transform all the universe into paperclips unless stopped by someone smarter than it”, not just “something really good at making paperclips without supervision”. Someone please hit me with a clue stick if I’ve been totally wrong about that.

Replies from: NancyLebovitz
comment by NancyLebovitz · 2013-05-30T22:55:37.510Z · LW(p) · GW(p)

You've gotten it right this time.

comment by lukstafi · 2013-05-25T19:33:18.042Z · LW(p) · GW(p)

So you think that majestic paperclip engineering cannot be cool? (Only regarding your last paragraph.)

comment by DataPacRat · 2013-05-24T20:17:14.272Z · LW(p) · GW(p)

I hadn't considered the possibility of a paperclipper being able to do anything that could keep life from restarting from scratch. (Which is probably just one of many reasons I shouldn't be an AI gatekeeper...)

Re your third point; once there are no longer any sapient beings left in the universe in which to judge the coolness of anything, do you feel it really matters whether or not they continue to exist? That is, do you feel that objects have some sort of objective measure of coolness which is worthwhile to preserve even in the absence of any subjective viewpoints to make coolness evaluations?

Replies from: someonewrongonthenet
comment by someonewrongonthenet · 2013-05-24T21:19:10.166Z · LW(p) · GW(p)

That is, do you feel that objects have some sort of objective measure of coolness which is worthwhile to preserve even in the absence of any subjective viewpoints to make coolness evaluations?

Do you care intrinsically about anything which isn't a mind? This seems to be something that would vary individually.

Replies from: DataPacRat
comment by DataPacRat · 2013-05-24T23:43:48.011Z · LW(p) · GW(p)

It's an interesting question; so far, the closest I have to an answer is that any timeline which doesn't have minds within it to do any caring, seems to be to not be worth caring about. Which leads to the answer to your question of 'nope'.

comment by CronoDAS · 2013-05-25T04:38:20.654Z · LW(p) · GW(p)

Let's try a variant...

Consider two planets, both completely devoid of anything resembling life or intelligence. Anyone who looks at either one of them sees an unremarkable hunk of rock of no particular value. In one of them, the center consists of more unremarkable rock. In the other, however, hidden beneath the surface is a cache that consists of replicas of every museum and library that currently exists on Earth, but which will never be found or seen by anyone (because nobody is going to bother to look that hard at an unremarkable hunk of rock). Does the existence of the second hunk of rock have more value than the first?

Replies from: army1987, bogdanb, DataPacRat
comment by A1987dM (army1987) · 2013-05-26T09:49:56.004Z · LW(p) · GW(p)

Not by any non-negligible extent. If I had to choose one of the two all other things being equal, I'd pick the latter, but if I had to pay five dollars to pick the latter I'd pick the former.

comment by bogdanb · 2013-05-28T00:45:51.147Z · LW(p) · GW(p)

Try this one: pick something, anything you want. How much would you value if it existed outside the universe? Use an expanding universe to throw it irrevocably outside your future light cone if “existing outside the universe" is making your brain cringe. Or use a cycling crunch/bang universe, and suppose it existed before the last crunch.

comment by DataPacRat · 2013-05-25T05:06:57.154Z · LW(p) · GW(p)

Assuming the non-existence of some entity which eventually disassembles and records everything in the entire universe (and thus finds the library, violating your condition that it's never found)? Then, at least to me, the answer to your question is: nope.

comment by Vladimir_Nesov · 2013-05-24T22:25:09.820Z · LW(p) · GW(p)

If we ignore the possibility of future life arising again after human extinction, paperclipper seems (maybe, a bit) better than extinction because of the possibility of acausal trade between the paperclipper and human values (see this comment and preceding discussion).

The value of possible future life arising by chance is probably discounted by fragility of value (alien values might be not much better than paperclipper's), the risk of it not arising at all or getting squashed by its own existential risks (Fermi paradox), the risk of it also losing its values (e.g. to an UFAI), the astronomical waste of not optimizing the universe in the meantime, and possibly time discounting of the (very distant) future.

(All of this discounting might still be smaller than what takes acausal trade to work out, so it's not clear which choice is better. A cleaner question would compare a paperclipper with a sterile universe.)

Replies from: DataPacRat, lukstafi
comment by DataPacRat · 2013-05-24T23:48:52.514Z · LW(p) · GW(p)

A cleaner question would compare a paperclipper with a sterile universe.

I really wanted to ask that question, but I'm not actually very confident in my estimate of how sterile our own universe is, over the long term, so I'm afraid that I waffled a bit.

comment by lukstafi · 2013-05-25T10:10:57.775Z · LW(p) · GW(p)

Some people reasonably think that value is simple and robust. Alien life will likely tend to share many of the more universal of our values, for example the "epistemic" values underlying development of science. ETA: Wow downvotes, gotta love them :-)

Replies from: MugaSofer, MugaSofer
comment by MugaSofer · 2013-05-28T16:06:50.233Z · LW(p) · GW(p)

The default assumption around here is that value is complex and fragile. If you think you have a strong argument to the contrary, have you considered posting on it? Even if you don't want to endorse the position, you could still do a decent devils-advocate steelman of it.

EDIT: having read the linked article, it doesn't say what you seem to think it does. It's arguing Friendliness is simpler than we think, not that arbitrary minds will converge on it.

Replies from: lukstafi, lukstafi
comment by lukstafi · 2013-05-28T18:17:32.475Z · LW(p) · GW(p)

In my opinion [i.e. it is my guess that], the value structures and considerations developed by alien evolved civilizations are likely to be similar and partially-inter-translatable to our value structures and considerations, in a manner akin to how their scientific theories and even social life languages are likely to be inter-translatable (perhaps less similar than for scientific theories, more similar than for social languages).

Replies from: MugaSofer, army1987
comment by MugaSofer · 2013-05-29T10:52:54.128Z · LW(p) · GW(p)

Well, I guess it comes down to the evolutionary niches that produce intelligence and morality, doesn't it? There doesn't seem to be any single widely-accepted answer for either of them, although there are plenty of theories, some of which overlap, some don't.

Then again, we don't even know how different they would be biologically, so I'm unwilling to make any confidant pronouncement myself, other than professing skepticism for particularly extreme ends of the scale. (Aliens would be humanoid because only humans evolved intelligence!)

Anyway, do you think the arguments for your position are, well, strong? Referring to it as an "opinion" suggests not, but also suggests the arguments for the other side must be similarly weak, right? So maybe you could write about that.

Replies from: lukstafi
comment by lukstafi · 2013-05-29T11:07:09.451Z · LW(p) · GW(p)

I appeal to (1) the consideration of whether inter-translatability of science, and valuing of certain theories over others, depends on the initial conditions of civilization that develops it. (2) Universality of decision-theoretic and game-theoretic situations. (3) Evolutionary value of versatility hinting at evolved value of diversity.

Replies from: MugaSofer
comment by MugaSofer · 2013-05-30T10:23:14.320Z · LW(p) · GW(p)

Not sure what 1 and 3 refer to, but 2 is conditional on a specific theory of origin for morality, right? A plausible one, to be sure, but by no means settled or demonstrated.

Replies from: lukstafi
comment by lukstafi · 2013-05-30T12:11:59.533Z · LW(p) · GW(p)

My point is that the origin of values, the initial conditions, is not the sole criterion for determining whether a culture appreciates given values. There can be convergence or "discovery" of values.

Replies from: MugaSofer
comment by MugaSofer · 2013-06-04T19:47:09.170Z · LW(p) · GW(p)

Oh, do you mean that even quite alien beings might want to deal with us?

Replies from: lukstafi
comment by lukstafi · 2013-06-04T21:12:48.041Z · LW(p) · GW(p)

No, I mean that we might give a shit even about quite alien beings.

comment by A1987dM (army1987) · 2013-05-28T20:43:19.702Z · LW(p) · GW(p)

For some value of “similar”, I agree. Aliens as ‘alien’ as the Babyeaters or the Superhappies don't sound terribly implausible to me, but it'd be extremely hard for me to imagine anything like the Pebblesorters actually existing.

Replies from: lukstafi, lukstafi
comment by lukstafi · 2013-05-29T06:59:08.155Z · LW(p) · GW(p)

Do you think that CEV-generating mechanisms are negotiable across species? I.e. whether other species would have a concept of CEV and would agree to at least some of the mechanisms that generate a CEV. It would enable determining which differences are reconcilable and where we have to agree to disagree.

comment by lukstafi · 2013-05-29T06:05:53.847Z · LW(p) · GW(p)

Is babyeating necessarily in babyeaters' CEV? Which of our developments (drop slavery, stop admiring Sparta etc.) were in our CEV "from the beginning"? Perhaps the dynamics has some degree of convergence even if with more than one basin of attraction.

Replies from: army1987
comment by A1987dM (army1987) · 2013-06-09T08:54:00.044Z · LW(p) · GW(p)

Which of our developments (drop slavery, stop admiring Sparta etc.) were in our CEV "from the beginning"?

People disagree about that, and given that it has political implications (google for "moral progress") I dare no longer even speculate about that.

Replies from: lukstafi
comment by lukstafi · 2013-06-09T09:43:29.821Z · LW(p) · GW(p)

I agree with your premise, I should have talked about moral progress rather than CEV. ETA: one does not need a linear order for the notion of progress, there can be multiple "basins of attraction". Some of the dynamics consists of decreasing inconsistencies and increasing robustness.

comment by lukstafi · 2013-05-29T23:44:27.321Z · LW(p) · GW(p)

Another point is that value (actually, a structure of values) shouldn't be confused with a way of life. Values are abstractions: various notions of beauty, curiosity, elegance, so called warmheartedness... The exact meaning of any particular such term is not a metaphysical entity, so it is difficult to claim that an identical term is instantiated across different cultures / ways of life. But there can be very good translations that map such terms onto a different way of life (and back). ETA: there are multiple ways of life in our cultures; a person can change her way of life by pursuing a different profession or a different hobby.

Replies from: MugaSofer
comment by MugaSofer · 2013-05-30T10:17:14.642Z · LW(p) · GW(p)

Values ultimately have to map to the real world, though, even if it's in a complicated way. If something wants the same world as me to exist, I'm not fussed as to what it calls the reason. But how likely is it that they will converge? That's what matters.

Replies from: lukstafi
comment by lukstafi · 2013-05-30T12:35:33.972Z · LW(p) · GW(p)

I presume by "the same world" you mean a sufficiently overlapping class of worlds. I don't think that "the same world" is well defined. I think that determining in particular cases what is "the world" you want affects who you are.

Replies from: MugaSofer
comment by MugaSofer · 2013-06-04T19:44:20.063Z · LW(p) · GW(p)

Well, I suppose in practice it's a question of short-term instrumental goals overlapping, yeah.

comment by MugaSofer · 2013-05-30T10:51:11.688Z · LW(p) · GW(p)

Today's relevant SMBC comic

I swear, that guy is spying on LW. He's watching us right now. Make a comic about THAT! shakes fist

comment by A1987dM (army1987) · 2013-05-25T09:55:55.666Z · LW(p) · GW(p)
  • Choose A, and all life and sapience in the solar system (and presumably the universe), save for a sapient paperclipping AI, dies.

  • Choose B, and all life and sapience in the solar system, including the paperclipping AI, dies.

I choose A. (OTOH, the difference between U(A) and U(B) is so small that throwing even a small probability of a different C in the mix could easily change that.)

If anyone responds positively, subsequent questions would be which would be preferred, a paperclipper or a single bacteria; a paperclipper or a self-sustaining population of trilobites and their supporting ecology; a paperclipper or a self-sustaining population of australopithecines; and so forth, until the equivalent value is determined.

I'd take the paperclipper over the bacteria. I'd probably take the paperclipper over the trilobites and the australopithecines over the paperclipper, but I'm not very confident about that.

Replies from: bogdanb
comment by bogdanb · 2013-05-28T00:39:02.608Z · LW(p) · GW(p)

I'd take the paperclipper over the bacteria. I'd probably take the paperclipper over the trilobites [...]

I’m curious about your reasoning here. As others pointed out, a paperclipper is expected to be very stable, in the sense that it is plausible it will paperclip everything forever. Bacteria however have the potential to evolve a new ecosystem, and thus to lead to "people" existing again. (Admittedly, a single bacteria would need a very favorable environment.) And a paperclipper might even destroy/prevent life that would have evolved even without any bacteria at all. (After all, it happened at least once that we know of, and forever is a long time.)

Replies from: army1987
comment by A1987dM (army1987) · 2013-05-28T10:19:13.472Z · LW(p) · GW(p)

I was more going with my gut feelings than with reasoning; anyway, thinking about the possibility of intelligent life arising again sounds like fighting the hypothetical to me (akin to thinking about the possibility of being incarcerated in the trolley dilemma), and also I'm not sure that there's any guarantee that such a new intelligent life would be any more humane than the paperclipper.

Replies from: bogdanb
comment by bogdanb · 2013-05-30T22:11:11.735Z · LW(p) · GW(p)

sounds like fighting the hypothetical to me

Well, he did say “solar system (and presumably the universe)”. So considering the universe is stipulated in the hypothetical, but the “presumably” suggests the hypothetical does not dictate the universe. And given that the universe is much bigger than the solar system, it makes sense to me to think about it. (And hey, it’s hard to be less human than a paperclipper and still be intelligent. I thought that’s why we use paperclippers in these things.)

If the trolley problem mentioned “everybody on Earth” somewhere, it would be reasonable to actually consider other people than those on the track. Lesson: If you’re making a thought experiment about spherical cows in a vacuum, don’t mention pastures.

comment by Baughn · 2013-05-24T19:42:08.970Z · LW(p) · GW(p)

I'd say.. no, the paperclipper probably has negative value.

Replies from: DataPacRat
comment by DataPacRat · 2013-05-24T19:46:51.904Z · LW(p) · GW(p)

To be clear - you're saying that you would prefer that there not exist a single thing which takes negentropy and converts it into order (or whatever other general definition for 'life' you prefer), and may or may not have the possibility of evolving into something else more complicated, over nothing at all?

Replies from: Baughn, None
comment by Baughn · 2013-05-24T21:18:18.233Z · LW(p) · GW(p)

I'm thinking that the paperclipper counts as a life not worth living - an AI that wants to obsess about paperclips is about as repugnant to me as a cow that wants to be eaten. Which is to say, better than doing either of those without wanting it, but still pretty bad. Yes, I'm likely to have problems with a lot of genuinely friendly AIs.

I was assuming that both scenarios were for keeps. Certainly the paperclipper should be smart enough to ensure that; for the other, I guess I'll assume you're actually destroying the universe somehow.

Replies from: lukstafi
comment by lukstafi · 2013-05-25T20:17:40.810Z · LW(p) · GW(p)

It is a fair point but do you mean that the paperclipper is wrong in its judgement that its life is worth living, or is it merely your judgement that if you were the paperclipper your life would not be worth living by your current standards? Remember that we assume that there is no other life possible in the universe anyway -- this assumption makes things more interesting.

Replies from: Baughn
comment by Baughn · 2013-05-26T12:54:14.419Z · LW(p) · GW(p)

It's my judgement that the paperclipper's life is not worth living. By my standards, sure; objective morality makes no sense, so what other standards could I use?

The paperclipper's own opinion matters to me, but not all that much.

Replies from: lukstafi
comment by lukstafi · 2013-05-26T15:06:54.718Z · LW(p) · GW(p)

Would you engage with a particular paperclipper in a discussion (plus observation etc.) to refine your views on whether its life is worth living? (We are straying away from a nominal AIXI-type definition of "the" paperclipper but I think your initial comment warrants that. Besides, even an AIXI agent depends on both terminal values and history.)

Replies from: Baughn
comment by Baughn · 2013-05-26T18:05:50.667Z · LW(p) · GW(p)

No, if I did so it'd hack my mind and convince me to make paperclips in my own universe. Assuming it couldn't somehow use the communications channel to directly take over our universe.

I'm not quite sure what you're asking here.

Replies from: lukstafi
comment by lukstafi · 2013-05-26T19:09:37.837Z · LW(p) · GW(p)

Oh well, I haven't thought of that. I was "asking" about the methodology for judging whether a life is worth living.

Replies from: Baughn
comment by Baughn · 2013-05-26T19:52:45.652Z · LW(p) · GW(p)

Whether or not I would enjoy living it, taking into account any mental changes I would be okay with.

For a paperclipper.. yeah, no.

Replies from: lukstafi, MugaSofer
comment by lukstafi · 2013-05-27T05:19:39.261Z · LW(p) · GW(p)

But you have banned most of the means of approximating the experience of living such a life, no? In a general case you wouldn't be justified in your claim (where by general case I mean the situation where I have strong doubts you know the other entity, not the case of "the" paperclipper). Do you have a proof that having a single terminal value excludes having a rich structure of instrumental values? Or does the way you experience terminal values overwhelm the way you experience instrumental values?

comment by MugaSofer · 2013-05-28T15:56:23.226Z · LW(p) · GW(p)

Assuming that clippy (or the cow, which makes more sense) feels "enjoyment", aren't you just failing to model them properly?

Replies from: Baughn
comment by Baughn · 2013-05-29T13:15:18.788Z · LW(p) · GW(p)

It's feeling enjoyment from things I dislike, and failing to pursue goals I do share. It has little value in my eyes.

Replies from: MugaSofer
comment by MugaSofer · 2013-05-30T10:21:13.607Z · LW(p) · GW(p)

Which is why I, who like chocolate icecream, categorically refuse to buy vanilla or strawberry for my friends.

Replies from: Baughn
comment by Baughn · 2013-05-30T17:46:07.463Z · LW(p) · GW(p)

Nice strawman you've got there. Pity if something were to.. happen to it.

The precise tastes are mostly irrelevant, as you well know. Consider instead a scenario where your friend asks you to buy a dose of cocaine.

Replies from: MugaSofer
comment by MugaSofer · 2013-06-04T19:13:12.384Z · LW(p) · GW(p)

I stand by my reducto. What is the difference between clippy enjoying paperclips vs humans enjoying icecream, and me enjoying chocolate icecream vs you enjoying strawberry? Assuming none of them are doing things that give each other negative utility, such as clippy turning you into paperclips of me paying the icecream vendor to only purchase chocolate (more for me!)

comment by [deleted] · 2013-05-24T20:33:08.643Z · LW(p) · GW(p)

That sounds as if scenario B precluded abiogenesis from happening ever again. After all, prebiotic Earth kind of was a thing which took negentropy and (eventually) converted it into order.

Replies from: DataPacRat
comment by DataPacRat · 2013-05-24T20:43:52.597Z · LW(p) · GW(p)

The question for B might then become, under which scenario is some sort of biogenesis more likely, one in which a papperclipper exists, or one in which it doesn't? The former includes the paperclipper itself as potential fodder for evolution, but (as was just pointed out) there's a chance the paperclipper might work to prevent it; while the latter has it for neither fodder nor interference, leaving things to natural processes.

At what point in biogenesis/evolution/etc do you think the Great Filter does its filtering?

comment by [deleted] · 2013-05-28T15:12:42.081Z · LW(p) · GW(p)

DataPacRat, I like that you included subsequent questions, and I think there may also be other ways of structuring subsequent questions as well which may also make people think about different answers.

Example: Is a paperclipper better than the something for what likely duration of time?

For instance, take the Trilobites vs paperclipper scenario you mentioned. I am imagining:

A: A solar system that has trilobites for 1 billion years, until it is engulfed by it's sun and everything dies.

B: A solar system that has trilobites in a self-sustaining gaia planet for Eternity.

C: A solar system that has a paperclipping AI for 1 billion years, until it is engulfed by it's sun and all of the paperclips melt.

D: A solar system that has a paperclipping AI that keeps a planet sized mass of paperclips as paperclips for eternity.

If I prefer B>D>A>C, then it seems like I might choose the paperclipper over the trilobites if figure the paperclipper has a 99.99% chance of lasting for an eternity, and the trilobite planet has a 0.01% chance of doing so.

On the other hand, it may be that you want to make an assumption that the paperclipper and the trilobites planet are equally resilient to existential crises for the purposes of this problem.

Second Example: Does the context of the paperclipper AI and it's destruction matter?

Imagine all of humanity, and the rest of the solar system, is going to be engulfed by our sun's super nova soon. We're all going to die. There is one person who is going to be uploaded into an Hansonian EM experimental probe that will be shot away from the blast, a person with a mental disorder from a foreign country who loves making paperclips. (Unfortunately humanity only got to uploading tech 0.91 prior to the super nova- Very few people have any kind of upload capable brain right now.)

You have read several papers from an aquaintance of yours, a respected scientist, who has said multiple times "If you upload him he'll turn into a paperclipper AI, I'm sure of it." you've also read a few independent publications indicating yes, this person is going to be a paperclipper AI if uploaded under uploading tech 0.91, and here is proof.

A research assistant has snuck you a sabotage virus that will destroy the upload probe stealthily after the upload has taken place, saying "I wanted to see if I could be James Bond before I died!" and then commits suicide.

Do you run the sabotage virus? You're going to die either way, but you can either have humanity's last monument be a paperclipper AI or nothing.

At least two explicit differences in this scenario appear to be:

A: The paperclipper AI appears to have some level of popular support. Humanity wouldn't have been spending trillions of dollars making him the only shot they have if not. (If you want more explicit context, imagine that when pressed by TV interviewers, Other scientists have said they explicitly read those papers, said that they understood them, but believed that anything was better than nothing, but they did not have time to explain. Polls indicate the paperclipper was supported 51-49, and clearly with at least some strong opposition, or no one would have bothered to build a sabotage virus.)

B: You don't actually have to make a choice: there is a default choice, which will occur if you don't press the button, which is the paperclipper AI remains.

I'm not sure I actually have answers to either of the questions yet, but the fact that both of them seem like they would make it more acceptable to allow the paperclipper then other options probably indicates at least some as yet unquantified pro-paperclipper thoughts on my part.

Replies from: DSherron
comment by DSherron · 2013-05-28T22:11:45.412Z · LW(p) · GW(p)

Run it. There is a non-zero possibility that a paperclips AI could destroy other life which I would care about, and a probability that it would create such life. I would put every effort I could into determining those 2 probabilities (mostly by accumulating the evidence from people much smarter than me, but still). I'll do the action with the highest expected value. If I had no time, though, I'd run it, because I estimate a ridiculously small chance that it would create life relative to destroying everything I could possibly care about.

comment by MugaSofer · 2013-05-28T11:45:22.942Z · LW(p) · GW(p)

I tend to model Paperclippers as conscious, simply because it's easier to use bits of my own brain as a black box. So naturally my instinct is to value it's existence the same as any other modified human mind (although not more than any lives it might endanger.)

However, IIRC, the original "paperclip-maximizer" was supposed to be nonsentient; probably still worth something in the absence of "life", but tricky to assign based on my intuitions (is it even possible to have a sufficiently smart being I don't value the same way I do "conscious" ones?)

In other words, I have managed to confuse my intuitions here.

comment by aelephant · 2013-05-27T08:56:24.710Z · LW(p) · GW(p)

[D]oes the existence of any intelligence at all, even a paperclipper, have even the smallest amount of utility above no intelligence at all?

Have utility to whom?

I presume when we are all dead, we will have no utility functions.

Replies from: DataPacRat
comment by DataPacRat · 2013-05-27T14:21:08.083Z · LW(p) · GW(p)

:) Usually, I'm the one who has to point this idea out when such discussions come up.

But to answer your question - it would be the you-of-the-present who is making a judgement call about which future scenario present-you values more. While it's true that there won't be a future-you within either future with which to experience said future, that doesn't mean present-you can't prefer one outcome to the other.

Replies from: aelephant, Decius
comment by aelephant · 2013-05-28T00:02:59.296Z · LW(p) · GW(p)

Because present-me knows that I won't be around to experience either future, present-me doesn't care either way. I'd flip a coin if I had to decide.

Replies from: MugaSofer
comment by MugaSofer · 2013-05-28T11:46:45.393Z · LW(p) · GW(p)

Which is why, naturally, you wouldn't sacrifice your life to save the world.

Replies from: aelephant
comment by aelephant · 2013-05-30T10:53:51.013Z · LW(p) · GW(p)

Little different than the proposed situation. There would be plenty of other people with utility functions surviving if I sacrificed myself to save the world.

comment by Decius · 2013-05-27T22:43:00.690Z · LW(p) · GW(p)

Does entropy in an isolated system decrease in either universe? Present-me considers the indistinguishable end states equivalent.

Replies from: bogdanb
comment by bogdanb · 2013-05-28T00:29:00.827Z · LW(p) · GW(p)

I know this doesn’t sound quite consequentialistic enough for some around here, but sometimes the journey matters too, not just the destination ;-)

And when the destination is guaranteed to be the same...

comment by ikrase · 2013-05-26T08:07:11.098Z · LW(p) · GW(p)

... Solar system, therefore universe? Does not seem plausible. For no sapient life that will ever develop in the observable universe, sapience needs to be WAY rarer. And the universe is infinite.

Replies from: DataPacRat
comment by DataPacRat · 2013-05-26T13:50:17.358Z · LW(p) · GW(p)

Solar system, plus the complete past light-cone leading up to the solar system, has a total of 1 intelligence developed; and since if there wasn't that one which was developed, we wouldn't be around to have this discussion in the first place, there are good reasons for not including that one in our count.

I'm not sure that your latter statement is correct, either; do you have any references to evidence regarding the infiniteness, or lack thereof, of the universe?

Replies from: bogdanb, ikrase, MugaSofer
comment by bogdanb · 2013-05-28T00:56:37.744Z · LW(p) · GW(p)

Solar system, plus the complete past light-cone leading up to the solar system has a total of 1 intelligence developed

Oh really? How can you tell that, say, none of the galaxies in the Hubble Deep Field developed intelligence? Hell, how can you tell there are no intelligent beings floating inside Jupiter right now?

comment by ikrase · 2013-05-26T20:11:54.847Z · LW(p) · GW(p)

Infinite universe: Thought that this was pretty settled science? Or at least that it's much bigger than hubble limit? Why must entire lightcone leading to Solar System have only one intelligence? Are you assuming that all intelligences will singularity faster than geological time, and then intrusively colonize space at speed of light, thus preventing future intelligences from rising? What about intelligences that are really, really far away? I think you are making really unjustifyable assumptions. I think this kind of anthropic stuff is... risky.

Would we be able to see a bronze-age civilization 500 ly away? Possible that such things could be more stable than ours? And a bronze age civilization is pretty different from nothing, more like ours than nothing.

Replies from: MugaSofer
comment by MugaSofer · 2013-05-28T11:50:43.372Z · LW(p) · GW(p)

Infinite universe: Thought that this was pretty settled science? Or at least that it's much bigger than hubble limit?

Big, yes. Infinite? No. And even the biggest finite universe is infinitely smaller than an infinite one, of course.

comment by MugaSofer · 2013-05-28T11:49:07.386Z · LW(p) · GW(p)

Solar system, plus the complete past light-cone leading up to the solar system, has a total of 1 intelligence developed

I know it's usual to equate "intelligent" with "human", just because we're the smartest ones around, but there are some pretty smart nonhuman animals around; presumably the present isn't unique in having them, either.

comment by Richard_Kennaway · 2013-05-25T17:06:49.333Z · LW(p) · GW(p)

How would the answers to these questions affect what you would do differently here and now?

Replies from: DataPacRat, Tenoke
comment by DataPacRat · 2013-05-25T17:34:16.120Z · LW(p) · GW(p)

I hope to use them to help work out the answers in extreme, edge-case conditions, to test various ethical systems and choose which one(s) provide the best advice for my long-term good.

Given that, so far, various LWers have said that a paperclipper could be better, worse, or around the same value as a sapience-free universe, I at least seem to have identified a boundary that's somewhat fuzzy, even among some of the people who'd have the best idea of an answer.

Replies from: army1987
comment by A1987dM (army1987) · 2013-05-26T09:54:47.605Z · LW(p) · GW(p)

Hard cases make bad law.

If you're going to decide whether to use Newtonian physics or general relativity for some everyday situation, you don't decide based on which theory makes the correct predictions near a black hole, you decide based on which is easier to use while still giving usable results.

Replies from: DataPacRat
comment by DataPacRat · 2013-05-26T13:47:07.674Z · LW(p) · GW(p)

A true enough analogy; but when you're trying to figure out whether Newtonian or Aristotlean physics is better for some everyday situation, it's nice to have general relativity to refer to, so that it's possible to figure out what GR simplifies down to in those everyday cases.

comment by Tenoke · 2013-05-26T10:05:43.548Z · LW(p) · GW(p)

How would answering your question affect what you would do differently here and now..

See what I did there?

comment by Armok_GoB · 2013-05-24T20:17:05.302Z · LW(p) · GW(p)

I chose A, on the off-chance that it interprets that as some kind of decision theoretical way that makes it do something I value in return for the favour.

Replies from: Vladimir_Nesov, Manfred, Mestroyer
comment by Vladimir_Nesov · 2013-05-24T22:36:34.814Z · LW(p) · GW(p)

I chose A

(This phrases the answer in terms of identity. The question should be about the abstract choice itself, not about anyone's decision about it. What do we understand about the choice? We don't actually need to decide.)

comment by Manfred · 2013-05-24T21:32:03.741Z · LW(p) · GW(p)

Since you doing it "on the off chance" doesn't correlate with whether or not it does anything special, any paperclipper worth its wire would make paperclips.

comment by Mestroyer · 2013-05-24T22:22:33.366Z · LW(p) · GW(p)

In other words, you're changing the thought experiment.

comment by Mestroyer · 2013-05-24T20:01:59.612Z · LW(p) · GW(p)

The two scenarios have equal utility to me, as close as I can tell. The paperclipper (and the many more than one copies of itself it would make) would be minds optimized for creating and maintaining paperclips (Though maybe it would kill itself off to create more paperclips eventually?) and would not be sentient. In contrast to you, I think I care about sentience, not sapience. To the very small extent that I saw the paperclipper has a person, rather than a force of clips, I would wish it ill, but only in a half-hearted way, which wouldn't scale to disutility for every paperclip it successfully created.

Replies from: DataPacRat, bartimaeus
comment by DataPacRat · 2013-05-24T20:09:29.862Z · LW(p) · GW(p)

I tend to use 'sentience' to separate animal-like things which can sense their environment from plant-like things which can't; and 'sapience' to separate human-like things which can think abstractly from critter-like things which can't. At the least, that's the approach that was in the back of my mind as I wrote the initial post. By these definitions, a paperclipper AI would have to be both sentient, in order to be sufficiently aware of its environment to create paperclips, and sapient, to think of ways to do so.

If I may ask, what quality are you describing with the word 'sentience'?

Replies from: MugaSofer, Mestroyer
comment by MugaSofer · 2013-05-28T15:51:06.130Z · LW(p) · GW(p)

Probably the same thing people mean when they say "consciousness". At least, that's the common usage I've seen.

comment by Mestroyer · 2013-05-24T20:32:25.001Z · LW(p) · GW(p)

I'm thinking of having feelings. I care about many critter-like things which can't think abstractly, but do feel. But just having senses is not enough for me.

Replies from: Vladimir_Nesov, DataPacRat
comment by Vladimir_Nesov · 2013-05-24T22:45:53.761Z · LW(p) · GW(p)

I'm thinking of having feelings. I care about many critter-like things which can't think abstractly, but do feel. But just having senses is not enough for me.

What you care about is not obviously the same thing as what is valuable to you. What's valuable is a confusing question that you shouldn't be confident in knowing a solution to. You may provisionally decide to follow some moral principles (for example in order to be able to exercise consequentialism more easily), but making a decision doesn't necessitate being anywhere close to being sure of its correctness. The best decision that you can make may still in your estimation be much worse than the best theoretically possible decision (here, I'm applying this observation to a decision to provisionally adopt certain moral principles).

comment by DataPacRat · 2013-05-24T20:40:11.564Z · LW(p) · GW(p)

To use a knowingly-inaccurate analogy: a layer of sensory/instinctual lizard brain isn't enough, a layer of thinking human brain is irrelevant, but a layer of feeling mammalian brain is just right?

Replies from: Mestroyer
comment by Mestroyer · 2013-05-24T20:54:42.111Z · LW(p) · GW(p)

Sounds about right, given the inaccurate biology.

comment by bartimaeus · 2013-05-24T20:09:23.865Z · LW(p) · GW(p)

How about a sentient AI whose utility function is orthogonal to yours? You care nothing about anything it cares about and it cares about nothing you care about. Also, would you call such an AI sentient?

Replies from: Mestroyer
comment by Mestroyer · 2013-05-24T20:35:55.997Z · LW(p) · GW(p)

You said it was sentient, so of course I would call it sentient. I would either value that future, or disvalue it. I'm not sure to what extent I would be glad some creature was happy, or to what extent I'd be mad at it for killing everyone else, though.

comment by wedrifid · 2013-06-16T19:10:51.849Z · LW(p) · GW(p)

Is a paperclipper better than nothing?

Nope. I choose B.

comment by A1987dM (army1987) · 2013-05-28T12:03:19.127Z · LW(p) · GW(p)

Maybe people who think that paperclips aren't boring enough can replace the paperclip maximizer with a supermassive black hole maximizer, as suggested here.

Replies from: ThisSpaceAvailable
comment by ThisSpaceAvailable · 2013-05-29T01:00:16.633Z · LW(p) · GW(p)

Well, the statement that "supermassive black holes with no ordinary matter nearby cannot evolved or be turned into anything interesting" is false.

comment by itaibn0 · 2013-05-29T22:48:14.796Z · LW(p) · GW(p)

I prefer A. The paperclipping AI will need to contemplate many interesting and difficult problems in physics, logistics, etc. to maximize paperclips. In doing so it will achieve many triumphs I would like a descendant of humanity to achieve. One potential problem I see is that the paperclipper will be crueler to intelligent life in other planets that isn't powerful enough to have leverage over it.

comment by Bruno_Coelho · 2013-05-26T21:03:28.521Z · LW(p) · GW(p)

Benatar assimetry between life and death make B the best option. But as his argument is hard to accept, A is better, whatever human values the AI implement.