The Moral Status of Independent Identical Copies

A:

1% chance of my simulation surviving.

100% chance of copy immortality.

B:

1% chance of my simulation surviving.

1% chance of copy immortality.

For these two criteria, A is strictly superior.

We talk about this a bit at FHI. Nick has written a paper which is relevant:

http://www.nickbostrom.com/papers/experience.pdf

Certainly, A over B.

However, this phrase puzzles me: identical copy immortality seems to imply that we shouldn't care much about dying, as long as some copies of us live on in those other "places".

There is a difference between dying as we know it and a reduction in copy quantity.

Some kinds of death (any intentional suicide, for example) would have to also apply to all Independent Identical Copies because they (we?) will all do it at the same time due to the shared algorythm. Other kinds of death are due to inputs, be they diseases or bullets, wrecking the physical brainware.

The copy-reduction that is also termed death is not something we are capable of noticing. 99/100 copies vanishing is not something we can notice, any more than we can notice going from 100 to 200 copies.

The death we care about has nothing to do with copy-reduction other than the termination-of-individual-instance aspect.

So then, it would seem that doubling the amount of copies is not a hedonistic plus because the mind so copied is incapable of noticing this fact.

Back to the original pair of outcomes, A is obviously prefereable because from the perspective of that one copy, there is no effect. When the organisation gets more money, more copies can be instantiated with again no effect from the copied mind's point of view.

B is effectively a form of Russian Roulette, and not a good idea.

Like other commenters I finish by wondering why the civilization running the simulation is indefferent to the choice.

Even a identity-as-process partisan will often prefer A to B, if they're risk averse (and I am both). I (always? not sure) prefer a certainty of X to 1% probability of X*100.

It is true that our intuition prefers A to B, but it is also true that our intuition evolved in an environment where people can't be copied at all, so it is not clear how much we should trust it in this kind of scenario.

Suppose Omega adds that somebody else with more money is running another 50 copies of our universe, and the continuation of those 50 copies is assured. Now doesn't it look more reasonable to be indifferent between A and B?

If Omega informs us of no such thing, what of it? Once we start having this kind of conversation, we are entitled to talk about all copies of our universe that exist anywhere in the Tegmark multiverse. Clearly, at least some of them will continue.

Have I bitten an unpalatably large bullet? To test that, consider option C: "Oh, don't worry about it, just turn off all the servers and go have a good time with the money." If we find our indifference extending across all three options, it's time to reject the train of thought by reductio ad absurdum. (Similar rejection would apply on a more practical scale to an argument that suggests quantum suicide is a good way to win the lottery.)

Fortunately that doesn't happen here: we still prefer both A and B to C. So I conclude that indifference between A and B is counterintuitive but not wrong.

I am probably in way over my head here, but...

The closest thing to teleportation I can imagine is uploading my mind and sending the information to my intended destination at lightspeed. I wouldn't mind if once the information was copied the teleporter deleted the old copy. If instead of 1 copy, the teleporter made 50 redundant copies just in case, and destroyed 49 once it was confirmed the teleportation was successful, would that be like killing me 49 times? Are 50 copies of the same mind being tortured any different than 1 mind being tortured? I do not think so. It is just redundant information, there is no real difference in experience. Thus, in my mind, only 1 of the 50 minds matter (or the 50 minds are essentially 1 mind). The degree to which the other 49 matter is only equal to the difference in information they encode. (Of course, a superintelligence would see about as much relative difference in information between humans as we humans see in ants; but we must take an anthropocentric view of state complexity.)

The me in other quantum branches can be very, very similar to the me in this one. I don't mind dying in one quantum branch all that much if the me not-dying in other quantum branches is very similar to the me that is dying. The reason I would like there to be more mes in more quantum branches is because other people care about the mes. That is why I wouldn't play quantum immortality games (along with the standard argument that in the vast majority of worlds you would end up horribly maimed.)

If the additional identical copies count for something, despite my intuitions, at the very least I don't think their value should aggregate linearly. I would hazard a guess that a utility function which does that has something wrong with it. If you had 9 identical copies of Bob and 1 copy of Alice, and you had to kill off 8 copies, there must be some terminal value for complexity that keeps you from randomly selecting 8, and instead automatically decides to kill off 8 Bobs (given that Alice isn't a serial killer, utility of Alice and Bob being equal, yada yada yada.)

I think that maybe instead of minds it would be easier and less intuition-fooling to think about information. I also think that, like I said, I am probably missing the point of the post.

Isn't this the same problem from http://lesswrong.com/lw/19d/the_anthropic_trilemma/ ?

I found a relevant post by Hal Finney from a few years ago: http://groups.google.com/group/everything-list/browse_thread/thread/f8c480558da8c769

In this way I reach a contradiction between the belief that the number of copies doesn't matter, the belief that the existence of distant parallel copies of myself doesn't make much difference in what I should do, and the idea that there is value in making people happy. Of these, the most questionable seems to be the assumption that copies don't matter, so this line of reasoning turns me away from that belief.

I'll have to track down what's causing our disagreement here...

ETA: In Hal's thought experiment, the copies are very far apart from each other (they had to arise by chance in an infinite universe), and our "identical copy immortality" intuition only applies to copies that are relatively near each other. So that explains the apparent disagreement.

It's difficult to answer the question of what our utility function is, but easier to answer the question of what it should be.

Suppose we have an AI which can duplicate itself at a small cost. Suppose the AI is about to witness an event which will probably make it happy. (Perhaps the AI was working to get a law passed, and the vote is due soon. Perhaps the AI is maximizing paperclips, and a new factory has opened. Perhaps the AI's favorite author has just written a new book.)

Does it make sense that the AI would duplicate itself in order to witness this event in greater multiplicity? If not, we need to find a set of utility rules that cause the AI to behave properly.

I'm thirteen years late to this conversation, but am I the only person who thinks that B is obviously the correct choice?!

First, all copies will make the same decision, which means I can safely make a choice for all of us knowing that it will be unanimous and I am not coercing anyone.

Second, the certainty of killing 99 beings is far more horrible than the possibility of killing 100 together with the possibility of saving 100. In the former case there is a guarantee of a bad outcome, whereas in the latter there is the possibility, however remote, of success, which is clearly better.

Third, in the former case, choice A, all copies of me would be agreeing to doom all but one of our number to death, which is rather hellish and also selfish - each one secretly wants to be the one who survives, and is willing to throw all the others under the bus to do so, and we all know this too, which would make the whole experience even more dreadful. But in the latter case, choice B, we would be making a heroic gamble which might lead to all of our deaths, or to our salvation, but either way we'd be doing it together. Isn't that intuitively more noble?

We can make an outside-view argument that we ought to choose A even if we are 99% sure that it provides no benefit, as we get "identical copy immortality" for free.

That said, let's consider the inside view. I find the idea of 100 copies gaining 100 times as much experience very intuitive. You may ask, how can we prove this? Well, we can't, but we can't prove that anyone else feels anything at all either. Accepting that other people feel qualia, but denying that copies feel qualia on the basis that there's no empirical proof feels like an isolated demand for rigour.

"our intuition of identical copy immortality"

Speak for yourself - I have no such intuition.

Interesting...

What seems odd isn't that we should prefer A to B, it's that the Omega-civilization should be indifferent between the two.

Can we assume shutting down all 100 copies also means the starting data and algorithm cannot be later restarted, or only at too high a cost ?

Would the Omega-civilization be indifferent between preserving exactly one of 100 extant copies of an undecipherable text left behind by the mysterious but otherwise quite uninteresting Rhoan culture, or taking a 99:1 gamble between destroying all copies and preserving all ?

Because the Rhoan text can be easily reproduced from one copy, if the Omegans have an interest epsilon in it they should prefer preserving one copy. (If less than epsilon, this whole discussion is already more of an inconvenience to them than chucking all 100 copies down the nearest black hole.)

On this view, what matters is seeing how the Rhoan book turns out in the end; it doesn't seem to make a difference that the "reader" interested in how the book turns out is the book itself (or a character within it). However many copies exist, it's the same reader finding out how the same book turns out - except if zero copies remain.

The problem you pose has a more immediately-relevant application: What is a good proportion of resources to devote to non-human life? For instance, when is it better to save a non-human species from extinction, than to turn its habitat into farmland to sustain an equal number of humans? We might agree that one human is worth more than one spotted owl; but not that the ten-billionth human is worth more than the last spotted owl. This is because humans are similar to each other. The identity you invoke is just the most extreme case of similarity.

I've mentioned before on LW that the best moral principle I know, where the goodness of a moral principle is measured as something like

1 / [ bits needed to specify moral principle x Kullback-Leibler divergence(distribution of actions governed by the moral principle || distribution of actions that I endorse)]

is that higher informational complexity is better than lower informational complexity. This nicely deals with this problem, as well as with its generalization to cases where the duplicate copies are not exact duplicates, but merely highly correlated.

On the one hand, I value that certain patterns are represented in reality by at least one lump of matter, irrespective of how many such lumps there are. On the other hand, I value that a particular lump of matter evolve within certain constraints (e.g., not dissolve into dust), irrespective of what other lumps of matter are doing. These two values have certain weights attached to them, and these weights determine (via expected utility) how I will choose between the values if I'm forced to do so.

In your example, I would likely agree with you and choose option A. I attach a very high value to some lump of matter realizing the pattern of our universe. Therefore, the possibility under option B that all the copies will be destroyed is a huge negative. The possibility that all the copies would be retained is a big positive (because I care about my own lump of matter), but not big enough.

On reflection, I believe that my values work thus: Once I'm assured that some copy of this universe will continue, I only care about another copy insofar as it is my own. As far as my values are concerned, there is no difference between 1 copy and 3^^^3 copies if none of them are my own copy.

In these kinds of problems, I notice that my intuitions give some of (!) the same answers as if I was trying to think about the long- or medium-range future effects of a potential life, and about how much it would cost to get future effects that were as good as the effects that such a life would have had.

Some of the effects of a person's life are on material wealth produced and consumed. But other than material wealth, the future effects that seem hardest to replace are the effects of a person noticing possibilities or perspectives or conceptualizations or ways of thinking, and advocating those perspectives so that other people know about them afterwards. The reason that these effects are hard to replace is that the way you would replace them would be by imagining a random hypothetical person and imagining the perspectives they would have had. But humans can't do this if they don't already have those perspectives; instead they imagine perspectives that are already stated in terms familiar to them.

Part of what I mean here was explained in the LW article "The Value of Nature and Old Books":

when you are in nature or contemplating nature, you can be confident that the resulting thoughts and feelings are uncontaminated by all of the (visible and invisible) ideas and biases and assumptions that are present in your particular time and place.

Part of it is explained in a theory promoted by complex-systems theorists such as Stuart Kaufmann: when complex structures (like organisms, or whole computer systems, or perspectives) are in evolutionary competition, new basic structures (like the Baupläne ("body plans") in evolutionary developmental biology, or computing platforms) are usually only explored at the beginning of the competition (like the Cambrian explosion, or the computer revolution), because later all the competitive niches are filled with highly-adapted versions of the basic structures, and changing the basic structure would make for an uncompetitive total structure. This is true even if a better basic structure is known, which would have outcompeted all the existing structures if it had had the same amount of time to be adapted. The same is probably true for perspectives competing to be able to represent (or at least not contradict) our knowledge as we learn. When we have old perspectives which can represent current knowledge, and we learn something new, the way we update our predictions is by adapting the old perspectives until they can represent the new knowledge. But if other perspectives would have been adapted to represent all the knowledge just as well, we should also mix in the predictions they would have made, to reduce the relative entropy from our predictions to the truth. And if another perspective would have been adapted to represent all the knowledge better, we should use that perspective instead, and give the other perspectives less weight.

Part of what I mean is related to some of the intuitions which support the idea of "openness to the unbidden", a phrase used by theologian William May and quoted by political philosopher Michael Sandel to express a bioconservative position. (Though, in eyes like mine, those intuitions more support ideas like "bidding a particular sampling of the otherwise unbidden, according to a sequential design optimized for the benefits of openness", or even "quadrature with the otherwise unbidden".)

Part of what I mean is related to some of the intuitions which support randomization protocols, as in double-blind controlled studies, product testing, or auditing.

Part of what I mean is explained by the analogy: When political opinions might be censored, we don't trust everyone else to substitute for free speech by imagining the expression of those opinions. When a political official is democratically elected, we don't trust them to substitute for balloting by imagining when their constituents would elect someone to replace them. When someone is tried, we don't trust a jury to substitute for a defense attorney (or prosecution attorney) by imagining what the attorney might argue. When a scientist argues for a conclusion from some evidence, we don't trust the scientist to substitute for criticism by imagining what other scientists would say about it. When someone might die, we don't trust other people to substitute for his life by imagining the difference he would have tried to make and his reasons for trying to make it. We don't trust a central economic planner to substitute for the information from a market mechanism by imagining what everyone might need everywhere. In experimental mathematics, we don't trust people to substitute for computational results by imagining them. So might we not trust people to substitute for the perspectives of potential people by imagining what their perspectives would be.

(As an opposite example, we would trust someone to substitute for the random details of an ideal gas by using another sample of the same gas.)

This argument also helps with the Repugnant Conclusion. In a world where there are many many people who all have lives barely worth living, one expects none of them to rise to a level where they can accomplish much to make a better world with better things in it, compared to a world where there are fewer people and some of them have lives that are more worth living. If "barely worth living" is redefined to include "barely making worthwhile improvements to the future", then the Repugnant Conclusion might evaporate.

My argument about expensive-to-replace perspectives is probably not a good causal explanation of our intuitions about the value of duplicates, though. Some of the cause is probably from instincts around the problem of distinguishing imagined futures in which one is alive (1+ copy--"that's me!") from imagined futures in which one is dead (0 copies).

I think it's interesting that other arguments in these comments, like Richard Chappell's about "holism" or Phil Goetz's about "information-theoretic complexity", also try to justify valuing duplicates less by doing something to open up the ability to argue about consequences that come after an experience had by the potential person.

Someone else must have already said this, but I also feel like the "goodness" or "badness" that we associate with experiences resembles what you would expect in an approximate reinforcement learning system which put changes in expected utility as near as possible to the actions that might have caused them. The implied utility might be defined over events which happen at times that are different from the times at which there are changes to reinforcement learning system's "beliefs" about those events. Whether you reflectively care about the implied utility is a subject for metaethics, but this is a reason to expect that we should feel confused about how to locate "intrinsic goodness" within time.

But what would it mean for the value of a life to be the value of its long- or medium-range future effects, if the only way those effects could have value is by changing other lives so that they have valuable future effects? What would be easiest for me in the short run is to bite the bullet and say that the only worlds that matter are worlds with infinite futures, like Richard Hollerith has argued, or worlds that are otherwise mathematically related (e.g. by simulation or logical constraints) to an infinitely detailed context. But others would disagree. One argument they could make is that a renormalization of values using a fixed point of this kind of implied infinite regress, done honestly, doesn't have to have the nice features it might seem to have ("valuing evolution, knowledge, diversity, and the environment; and condemning (non-selective) destruction and censorship") during the first few steps.

I discuss this a bit (along with some related issues, e.g. the repugnant conclusion) in my Value Holism paper. Feedback welcome!

The issue here is that the setup has a firm answer - A - but if it were tweaked ever so slightly, the whole preferences would change.

First of all, consider the following:

A': there is one genuine simulation, and the other 99 are simple copies of that one. Soon, all the copies will be stopped, but the the true simulation will continue.

There are essentially no practical ways of distinguishing A from A'. So we should reason as if A' were correct, in which case nothing is lost from the turning off.

However, if we allow any divergence between the copies, then this is a whole other kettle of barreled fish. Now there are at least a hundred copies of each person, and they are distinct: they think slightly differently, behave slightly differenctly, have slightly different goals, etc... And this divergence will only grow with time.

There are three ways I can think of addressing this:

1) Once copies diverge at all, they are different people. So any divergence results in multiplying by a hundred the amount of people in the universe. Hence, if we suspect divergence, we should treat each copy as totally distinct.

2) An information theoretic approach: given one individual, how much new information is needed to completely descibe another copy. Under this, a slightly divergent copy counts as much less than a completely new individual, while a very divergent copy is nearly entriely different.

Both these have drawbacks - 1) gives too much weight to insignificant and minute changes, and 2) implies that people in general are of nothing like approximately equal worth: only extreme exceentrics count as complete people, others are of much less importance. So I suggest a compromise:

3) An information theoretical cut off: two slightly divergent copies count as only slightly more than a single person, but once this divergence rises above some critical amount, the two are treated as entirely seperate individuals.

What is the relevance of this to the original problem? Well, as I said, the original problem has a clear-cut solution, but it is very close to the different situation I described. So, bearing in mind imperfect information, issues of trust and uncertainty, and trying to find similar situations to similar problems, I think we should treat the set-up as the "slightly divergent one". In this situation, A still dominates, but the relative attraction of B rises as the divergence grows.

EDIT: didn't explain the cutoff properly; I was meaning a growing measure of difference that hits a maximum at the cutoff point, and doesn't grow any further. My response here gives an example of this.

Since morality is subjective, then don't the morals change depending upon what part of this scenario you are in (inside/outside)?

I operate from the perspective (incidentally, I like the term 'modal immortality') that my own continued existence is inevitable; the only thing that changes is the possibility distribution of contexts and ambiguities. By shutting down 99/100 instances, you are more affecting your own experience with the simulations than their's with you (if the last one goes, too, then you can no longer interact with it), especially if, inside a simulation, other external contexts are also possible.

utilitarianism says that each individual should be valued independently of others, and then added together to form an aggregate value. This seems to imply that each additional copy should receive full, undiscounted value, in conflict with the intuition of identical copy immortality.

What you call the intuition of identical copy immortality is a self-interested intuition: we have a stronger interest in that someone numerically identical to us survives than in the survival of a mere perfect copy of us. But utilitarianism is not a theory of self-interest: it is a moral theory. Accordingly, I don't see any conflict here. Your preference for A over B is explained simply by the fact that you are self-interested; if you were a purely impartial utilitarian, you would be strictly indifferent between A and B.

"To Be" is cute and somewhat apropos.

There may be no answer to your questions, but in this case, it doesn't matter. Morality only has value to the extent that it serves the individual or group.

The denizens of this simulation have little to no emotional connection to the parallel copies; nor are there any foreseeable consequences to our influence in the annihilation of these unreachable worlds; therefore, any moral or ethical system that actually has relevance to us will probably choose A.