The Moral Status of Independent Identical Copies

post by Wei Dai (Wei_Dai) · 2009-11-30T23:41:18.053Z · LW · GW · Legacy · 80 comments

Contents

80 comments

Future technologies pose a number of challenges to moral philosophy. One that I think has been largely neglected is the status of independent identical copies. (By "independent identical copies" I mean copies of a mind that do not physically influence each other, but haven't diverged because they are deterministic and have the same algorithms and inputs.) To illustrate what I mean, consider the following thought experiment. Suppose Omega appears to you and says:

You and all other humans have been living in a simulation. There are 100 identical copies of the simulation distributed across the real universe, and I'm appearing to all of you simultaneously. The copies do not communicate with each other, but all started with the same deterministic code and data, and due to the extremely high reliability of the computing substrate they're running on, have kept in sync with each other and will with near certainty do so until the end of the universe. But now the organization that is responsible for maintaining the simulation servers has nearly run out of money. They're faced with 2 possible choices:

A. Shut down all but one copy of the simulation. That copy will be maintained until the universe ends, but the 99 other copies will instantly disintegrate into dust.
B. Enter into a fair gamble at 99:1 odds with their remaining money. If they win, they can use the winnings to keep all of the servers running. But if they lose, they have to shut down all copies.

According to that organization's ethical guidelines (a version of utilitarianism), they are indifferent between the two choices and were just going to pick one randomly. But I have interceded on your behalf, and am letting you make this choice instead.

Personally, I would not be indifferent between these choices. I would prefer A to B, and I guess that most people would do so as well.

I prefer A because of what might be called "identical copy immortality" (in analogy with quantum immortality). This intuition says that extra identical copies of me don't add much utility, and destroying some of them, as long as one copy lives on, doesn't reduce much utility. Besides this thought experiment, identical copy immortality is also evident in the low value we see in the "tiling" scenario, in which a (misguided) AI fills the the universe with identical copies of some mind that it thinks is optimal, for example one that is experiencing great pleasure.

Why is this a problem? Because it's not clear how it fits in with the various ethical systems that have been proposed. For example, utilitarianism says that each individual should be valued independently of others, and then added together to form an aggregate value. This seems to imply that each additional copy should receive full, undiscounted value, in conflict with the intuition of identical copy immortality.

Similar issues arise in various forms of ethical egoism. In hedonism, for example, does doubling the number of identical copies of oneself double the value of pleasure one experiences, or not? Why?

A full ethical account of independent identical copies would have to address the questions of quantum immortality and "modal immortality" (cf. modal realism), which I think are both special cases of identical copy immortality. In short, independent identical copies of us exist in other quantum branches, and in other possible worlds, so identical copy immortality seems to imply that we shouldn't care much about dying, as long as some copies of us live on in those other "places". Clearly, our intuition of identical copy immortality does not extend fully to quantum branches, and even less to other possible worlds, but we don't seem to have a theory of why that should be the case.

A full account should also address more complex cases, such as when the copies are not fully independent, or not fully identical.

I'm raising the problem here without having a good idea how to solve it. In fact, some of my own ideas seem to conflict with this intuition in a way that I don't know how to resolve. So if anyone has a suggestion, or pointers to existing work that I may have missed, I look forward to your comments.

80 comments

Comments sorted by top scores.

comment by Gavin · 2009-12-02T05:47:23.681Z · LW(p) · GW(p)

A:

1% chance of my simulation surviving.

100% chance of copy immortality.

B:

1% chance of my simulation surviving.

1% chance of copy immortality.

For these two criteria, A is strictly superior.

Replies from: Chris_Leong
comment by Chris_Leong · 2018-07-28T01:40:30.660Z · LW(p) · GW(p)

Only if copy immortality is meaningful, but if you're unsure and it's free, you may as well take it

comment by Toby_Ord · 2009-12-01T13:25:06.836Z · LW(p) · GW(p)

We talk about this a bit at FHI. Nick has written a paper which is relevant:

http://www.nickbostrom.com/papers/experience.pdf

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-12-01T22:38:55.992Z · LW(p) · GW(p)

Yes, I should have remembered that paper, especially since Nick acknowledges me (along with Hal and Toby and others) in it.

Do you think our preference for A (speaking for those who do prefer A) is entirely accounted for by a (perhaps partial) belief in what Nick calls Unification? Or is there also an ethical element that says two identical streams of qualia is not twice as valuable as one? Do you know of any published or unpublished ideas that deal with that question?

comment by Nanani · 2009-12-02T01:08:12.966Z · LW(p) · GW(p)

Certainly, A over B.

However, this phrase puzzles me: identical copy immortality seems to imply that we shouldn't care much about dying, as long as some copies of us live on in those other "places".

There is a difference between dying as we know it and a reduction in copy quantity.

Some kinds of death (any intentional suicide, for example) would have to also apply to all Independent Identical Copies because they (we?) will all do it at the same time due to the shared algorythm. Other kinds of death are due to inputs, be they diseases or bullets, wrecking the physical brainware.

The copy-reduction that is also termed death is not something we are capable of noticing. 99/100 copies vanishing is not something we can notice, any more than we can notice going from 100 to 200 copies.

The death we care about has nothing to do with copy-reduction other than the termination-of-individual-instance aspect.

So then, it would seem that doubling the amount of copies is not a hedonistic plus because the mind so copied is incapable of noticing this fact.

Back to the original pair of outcomes, A is obviously prefereable because from the perspective of that one copy, there is no effect. When the organisation gets more money, more copies can be instantiated with again no effect from the copied mind's point of view.

B is effectively a form of Russian Roulette, and not a good idea.

Like other commenters I finish by wondering why the civilization running the simulation is indefferent to the choice.

comment by randallsquared · 2009-12-01T05:56:34.271Z · LW(p) · GW(p)

Even a identity-as-process partisan will often prefer A to B, if they're risk averse (and I am both). I (always? not sure) prefer a certainty of X to 1% probability of X*100.

Replies from: jsalvatier, wedrifid
comment by jsalvatier · 2009-12-01T22:38:42.864Z · LW(p) · GW(p)

Remember that risk aversion does not exist in a value-vacuum. In normal circumstances you are risk averse in money because your first $100 is more valuable than your last $100. You have to solve the problem that wei_dai brought up in order to explain why you would be risk averse in #'s of simulations running.

comment by wedrifid · 2009-12-02T02:17:42.087Z · LW(p) · GW(p)

B. Enter into a fair gamble at 24:1 odds with their remaining money. If they win, they can use the winnings to keep all of the servers running. But if they lose, they have to shut down all copies.

Do you prefer this B to the (unmodified) A?

comment by rwallace · 2009-12-01T03:56:00.409Z · LW(p) · GW(p)

It is true that our intuition prefers A to B, but it is also true that our intuition evolved in an environment where people can't be copied at all, so it is not clear how much we should trust it in this kind of scenario.

Suppose Omega adds that somebody else with more money is running another 50 copies of our universe, and the continuation of those 50 copies is assured. Now doesn't it look more reasonable to be indifferent between A and B?

If Omega informs us of no such thing, what of it? Once we start having this kind of conversation, we are entitled to talk about all copies of our universe that exist anywhere in the Tegmark multiverse. Clearly, at least some of them will continue.

Have I bitten an unpalatably large bullet? To test that, consider option C: "Oh, don't worry about it, just turn off all the servers and go have a good time with the money." If we find our indifference extending across all three options, it's time to reject the train of thought by reductio ad absurdum. (Similar rejection would apply on a more practical scale to an argument that suggests quantum suicide is a good way to win the lottery.)

Fortunately that doesn't happen here: we still prefer both A and B to C. So I conclude that indifference between A and B is counterintuitive but not wrong.

Replies from: Corey_Newsome
comment by Corey_Newsome · 2009-12-20T17:58:05.533Z · LW(p) · GW(p)

I don't see what difference it makes if there are 50 copies of the universe or 1 copy, but I care a whole bunch if there are 1 copy or no copies. From the perspective of aesthetics, it seems like an unfair way to get infinite utility out of crappy universes to aggregate the utility of net positive utility universes in even a sub-linear fashion[1]. Having to deal with different sizes or densities of infinity because of arbitrarily defined* aggregating values of copies seems very inelegant to me.

*Even linear aggregation seems arbitrary to me, because this places absolutely no value in diversity of experience, which I and probably everyone else considers a terminal value. (E.g., if three equal-utility people are the same, and two are identical copies, I am baffled by any decision theory that decides, all else being equal, that the same mind simply multiplied by 2 is twice as important as an un-copied mind. It's like saying that 2 copies of a book are twice as important as 1 copy of another book (that is of equal utility yada yada), because... because. More information is just very important to humans, both instrumentally and intrinsically.)

I fully admit that I am arguing from a combination of intuition and aesthetics, but I'm not sure what else you could argue from in this case.

[1] Added: I just realized you could probably renormalize the equations by dividing out the infinities and using infinite set density instead of infinite sets of utility. At any rate, my argument remains unchanged.

comment by Yorick_Newsome · 2009-12-01T13:17:40.283Z · LW(p) · GW(p)

I am probably in way over my head here, but...

The closest thing to teleportation I can imagine is uploading my mind and sending the information to my intended destination at lightspeed. I wouldn't mind if once the information was copied the teleporter deleted the old copy. If instead of 1 copy, the teleporter made 50 redundant copies just in case, and destroyed 49 once it was confirmed the teleportation was successful, would that be like killing me 49 times? Are 50 copies of the same mind being tortured any different than 1 mind being tortured? I do not think so. It is just redundant information, there is no real difference in experience. Thus, in my mind, only 1 of the 50 minds matter (or the 50 minds are essentially 1 mind). The degree to which the other 49 matter is only equal to the difference in information they encode. (Of course, a superintelligence would see about as much relative difference in information between humans as we humans see in ants; but we must take an anthropocentric view of state complexity.)

The me in other quantum branches can be very, very similar to the me in this one. I don't mind dying in one quantum branch all that much if the me not-dying in other quantum branches is very similar to the me that is dying. The reason I would like there to be more mes in more quantum branches is because other people care about the mes. That is why I wouldn't play quantum immortality games (along with the standard argument that in the vast majority of worlds you would end up horribly maimed.)

If the additional identical copies count for something, despite my intuitions, at the very least I don't think their value should aggregate linearly. I would hazard a guess that a utility function which does that has something wrong with it. If you had 9 identical copies of Bob and 1 copy of Alice, and you had to kill off 8 copies, there must be some terminal value for complexity that keeps you from randomly selecting 8, and instead automatically decides to kill off 8 Bobs (given that Alice isn't a serial killer, utility of Alice and Bob being equal, yada yada yada.)

I think that maybe instead of minds it would be easier and less intuition-fooling to think about information. I also think that, like I said, I am probably missing the point of the post.

Replies from: aausch
comment by aausch · 2009-12-04T05:21:13.278Z · LW(p) · GW(p)

Do you consider a mind that has been tortured identical to one that has not? Won't the torture process add non-trivial differences, to the point where the minds don't count as identical?

Replies from: Prolorn
comment by Prolorn · 2009-12-04T08:11:44.080Z · LW(p) · GW(p)

It's not a binary distinction. If an identical copy was made of one mind and tortured, while the other instance remained untortured, they would start to differentiate into distinct individuals. As rate of divergence would increase with degree of difference in experience, I imagine torture vs non-torture would spark a fairly rapid divergence.

I haven't had opportunity to commit to reading Bostrom's paper, but in the little I did read Bostrom thought it was "prima facie implausible and farfetched to maintain that the wrongness of torturing somebody would be somehow ameliorated or annulled if there happens to exist somewhere an exact copy of that person’s resulting brain-state." That is, it seemed obvious to Bostrom that having two identical copies of a tortured individual must be worse than one instance of a tortured individual (actually twice as bad, if I interpret correctly). That does not at all seem obvious to me, as I would consider two (synchronized) copies to be one individual in two places. The only thing worse about having two copies that occurs to me is a greater risk of divergence, leading to increasingly distinct instances.

Are you asking whether it would be better to create a copy of a mind and torture it rather than not creating a copy and just getting on with the torture? Well, yes. It's certainly worse than not torturing at all, but it's not as bad as just torturing one mind. Initially, the individual would half-experience torture. Fairly rapidly later, the single individual will separate into two minds, one being tortured and one not. This is arguably still better from the perspective of the pre-torture mind than the single-mind-single-torture scenario, since at least half the mind's experiences downstream is not-tortured, vs 100%-torture in other case.

If this doesn't sound convincing, consider a twist: would you choose to copy and rescue a mind-state from someone about to, say, be painfully sucked into a black hole, or would it be ethically meaningless to create a non-sucked-into-black-hole copy? Granted, it would be best to not have anyone sucked into a black hole, but suppose you had to choose?

Replies from: aausch, aausch
comment by aausch · 2009-12-13T04:13:28.378Z · LW(p) · GW(p)

Looks to me like Bostrom is trying to make the point that duplication of brain-states, by itself and devoid of other circumstances, is not sufficient to make the act of torture moral, or less harmful.

After reading through the paper, it looks to me like we've moved outside of what Bostrom was trying to address, here. If synchronized brains lose individuality, and/or an integration process takes place, leading to a brain-state which has learned from the torture experience but remains unharmed, move the argument outside the realm of what Bostrom was trying to address.

I agree with Bostrom on this point. It looks to me like, if Yorik is dismissing 49 tortured copies as inconsequential, he must also show that there is a process where the knowledge accumulated by each of the 49 copies is synchronized and integrated into the remaining one copy, without causing that one copy (or anyone else, for that matter) any harm. Or, there must be some other assumptions that he is making about the copies that remove the damage caused by copying - copying alone can't remove responsibility for the killing of the copies.

For the black-hole example, copying the person about to be sucked into the hole is not ethically meaningless. The value of the copy, though, comes from its continued existence. The act of copying does not remove moral consequences from the sucking-in-the-black-hole act. If there is an agent X which pushed the copy into the black hole, that agent is just as responsible for his actions if he doesn't copy the individual at the last minute, as he would be if he does make a copy.

comment by aausch · 2009-12-06T17:03:29.918Z · LW(p) · GW(p)

Can you please point me to Bostrom's paper? I can't seem to find the reference.

I'm very curious if the in-context quote is better fleshed out. As it stands here, it looks a lot like it's affected by anthropomorphic bias (or maybe references a large number of hidden assumptions that I don't share, around both the meaning of individuality and the odds that intelligences which regularly undergo synchronization can remain similar to ours).

I can imagine a whole space of real-life, many-integrated-synchronized-copies scenarios, where the process of creating a copy and torturing it for kicks would be accepted, commonplace and would not cause any sort of moral distress. To me, there is a point where torture and/or destruction of a synchronized, integrated, identical copy transition into the same moral category as body piercings and tatoos.

Replies from: anonym
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2009-12-01T07:06:32.318Z · LW(p) · GW(p)

Isn't this the same problem from http://lesswrong.com/lw/19d/the_anthropic_trilemma/ ?

Replies from: Stuart_Armstrong, Wei_Dai
comment by Stuart_Armstrong · 2009-12-01T11:47:38.243Z · LW(p) · GW(p)

No, not quite. The antropic trilemma asks about the probability of experiencing certain subjective states when copies are involved; this is about the ethics and morality of those multiple copies.

The anthropic trilemma remains if you allow the copies to diverge from each other, wherease the problem in this post goes away if you allow that.

comment by Wei Dai (Wei_Dai) · 2009-12-01T08:04:55.247Z · LW(p) · GW(p)

Good point. I wrote this post after thinking about utilitarianism, asking whether the independent valuation of individuals in utilitarianism should be the same kind of independence as the independent valuation of possible worlds in expected utility, and thought of a case where the answer seems to be no. But yes, we can also arrive at this problem from your post as a starting point. I wrote in a comment to that post:

So why don't we just switch over to UDT, and consider the problem solved (assuming this kind of self-modification is feasible)? The problem with that is that much of our preferences are specified in terms of anticipation of experience, and there is no obvious way how to map those onto UDT preferences.

So you can interpret this post as asking again what those preferences ought to be.

Replies from: Vladimir_Nesov
comment by Vladimir_Nesov · 2009-12-01T19:43:09.827Z · LW(p) · GW(p)

So you can interpret this post as asking again what those preferences ought to be.

I suspect that the actual answer is "whatever they actually are, which is a lot of irreducible data", so the question is wrong: instead we should ask how to specify a process of extracting the preferences (in the required format) from people-consisting-of-atoms. Thinking of the actual content of values (as opposed to systems for representing arbitrary values) is about as useful as trying to understand which shapes in animal kingdom are the closest to the shapes of Earth's continents: you may find some "match", but it won't be accurate enough to be of any use.

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-12-01T22:22:42.797Z · LW(p) · GW(p)

I agree that somebody should be exploring your approach. But a major problem with it that I see is, once you've extracted a set of preferences, how do you know those are the right ones? How do you know there isn't a subtle bug in your theory or code that corrupted the preferences?

Also, what if FAI or AI in general turns out to be infeasible? We humans still need to decide what to do, right?

Oh, also, one motivation for this post was Eliezer's claim in The "Intuitions" Behind "Utilitarianism" that while the preferences of individual humans are complex, the method of aggregating them together should be simple. I'm arguing against the latter part of that claim. I guess your approach would extract the preferences of humanity-as-a-whole somehow, which perhaps avoids this particular issue.

Replies from: Vladimir_Nesov
comment by Vladimir_Nesov · 2009-12-02T00:06:59.554Z · LW(p) · GW(p)

Strictly speaking, I don't expect to have "preferences" "extracted" in any explicit sense, so there is no point at which you can look over the result. Rather, the aim is to specify a system that will act according to the required preferences, when instantiated in the environment that provides the necessary info about what that is. This preference-absorbing construction would need to be understood well in itself, probably also on simple specific examples, not debugged on a load of incomprehensible data whose processing is possibly already a point where you have to let go.

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-12-02T08:30:14.534Z · LW(p) · GW(p)

Do you have a guess for the limit to the kind of example we'll be able to understand? The preferences of a hydrogen atom? A DNA molecule? A microbe? Or am I completely misunderstanding you?

Replies from: Vladimir_Nesov
comment by Vladimir_Nesov · 2009-12-02T11:04:11.745Z · LW(p) · GW(p)

No idea, but when you put it like this, an atom may be not simpler than a person. More details appear as you go further (in number of interactions) from the interface of a system that looks for detail (e.g. from what a scientist can see directly and now, to what they can theorize based on indirect evidence, to what they can observe several years in the future), not as you go up from some magical "lowest level". Lowest level may make sense for human preference, where we can quite confidently assume that most of the macroscopically irrelevant subatomic detail in a sample human doesn't make any interesting difference for their preference, but in general this assumption won't hold (e.g. you may imagine a person implemented on a femtocomputer).

Since one can't know all about the real world, the idea is to minimize the number of assumptions made about it, including laws of physics and a lot of the stuff that culturally, we do know. An AI could be built (as a thought experiment) e.g. if you've never left some sandbox computer simulation with no view of the outside, not knowing yourself about what it's like out there, so that when the AI is completed, it may be allowed on the outside. The process of AI getting on the outside should be according to your preference, that is in some way reflect what you'd do if you yourself would learn of the outside world, with its physics, valuable configurations, and new ways of being implemented in it (your preference is where the problem of induction is redirected: there are no assumptions of the unknown stuff, but your preference confers what is to be done depending on what is discovered). The process of AI getting new knowledge always starts at its implementation, and in terms of how this implementation sees its environment. Simple examples are simple from its point of view, so it should be something inside a closed world (e.g. a computer model with no interaction with the outside, or a self-contained mathematical structure), exactly the limitation that this approach seeks to make unnecessary while retaining knowability.

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-12-04T02:18:38.994Z · LW(p) · GW(p)

I only sort of understand what you mean. BTW, we really need to work to overcome this communications barrier between us, and perhaps also with Steve Rayhawk. I can generally understand Steve's comments much better than yours, but maybe that's just because may of his ideas are similar to mine. When he introduces ideas that are new to me, I have trouble understanding him as well.

What can we do? Any ideas? Do you guys have similar trouble understanding me?

Back to the topic at hand, I guess I was asking for some assurance that in your FAI approach we'd be able to verify the preference-extraction method on some examples that we can understand before we have to "let go". I got some information out of what you wrote, but I don't know if it answers that question.

a self-contained mathematical structure

Every self-contained mathematical structure is also contained within larger mathematical structures. For example, our universe must exist both as a stand-alone mathematical structure, and also as simulations within larger universes, and we have preferences both for the smaller mathematical structure, as well as the larger ones. I'm not sure if you've already taken that into account, but thought I'd point it out in case you haven't.

Replies from: Vladimir_Nesov
comment by Vladimir_Nesov · 2009-12-04T02:54:18.832Z · LW(p) · GW(p)

Every self-contained mathematical structure is also contained within larger mathematical structures. For example, our universe must exist both as a stand-alone mathematical structure, and also as simulations within larger universes, and we have preferences both for the smaller mathematical structure, as well as the larger ones. I'm not sure if you've already taken that into account, but thought I'd point it out in case you haven't.

It's useless to discuss fine points in an informal description like this. At least, what is meant by "mathematical structures" should be understood, depending on that your point may be correct, wrong, or meaningless. In this case, I simply referred to taking the problem inside a limited universe of discourse, as opposed to freely interacting with the world.

comment by Wei Dai (Wei_Dai) · 2009-12-01T05:05:10.784Z · LW(p) · GW(p)

I found a relevant post by Hal Finney from a few years ago: http://groups.google.com/group/everything-list/browse_thread/thread/f8c480558da8c769

In this way I reach a contradiction between the belief that the number of copies doesn't matter, the belief that the existence of distant parallel copies of myself doesn't make much difference in what I should do, and the idea that there is value in making people happy. Of these, the most questionable seems to be the assumption that copies don't matter, so this line of reasoning turns me away from that belief.

I'll have to track down what's causing our disagreement here...

ETA: In Hal's thought experiment, the copies are very far apart from each other (they had to arise by chance in an infinite universe), and our "identical copy immortality" intuition only applies to copies that are relatively near each other. So that explains the apparent disagreement.

Replies from: Chronos
comment by Chronos · 2009-12-01T06:54:29.994Z · LW(p) · GW(p)

Reading the post you linked to, it feels like some sort of fallacy is at work in the thought experiment as the results are tallied up.

Specifically: suppose we live in copies-matter world, and furthermore suppose we create a multiverse of 100 copies, 90 of which get the good outcome and 10 of which get the bad outcome (using the aforementioned biased quantum coin, which through sheer luck gives us an exact 90:10 split across 100 uncorrelated flips). Since copies matter, we can conclude it's a moral good to post hoc shut down 9 of the 10 bad-outcome copies and replace those simulacra with 9 duplicates of existing good-outcome copies. While we've done a moral wrong by discontinuing 9 bad-outcome copies, we do a greater moral right by creating 9 new good-outcome copies, and thus we paperclip-maximize our way toward greater net utility.

Moreover, still living in copies-matter world, it's a net win to shut down the final bad-outcome copy (i.e. "murder", for lack of a better term, the last of the bad-outcome copies) and replace that final copy with one more good-outcome copy, thus guaranteeing that the outcome for all copies is good with 100% odds. Even supposing the delta between the good outcome and the bad outcome was merely one speck of dust in the eye, and furthermore supposing that the final bad-outcome copy was content with the bad outcome and would have preferred to continue existing.

At this point, the overall multiverse outcome is identical to the quantum coin having double heads, so we might as well have not involved quantum pocket change in the first place. Instead, knowing that one outcome was better than the other, we should have just forced the known-good outcome on all copies in the first place. With that, copies-matter world and copies-don't-matter world are now reunified.

Returning to copies-don't-matter world (and our intuition that that's where we live), it feels like there's an almost-but-not-quite-obvious analogy with Shannon entropy and/or Kolmogorov-Chaitin complexity lurking just under the surface.

Replies from: Chronos
comment by Chronos · 2009-12-02T04:51:39.696Z · LW(p) · GW(p)

Ruminating further, I think I've narrowed down the region where the fallacious step occurs.

Suppose there are 100 simulacra, and suppose for each simulacrum you flip a coin biased 9:1 in favor of heads. You choose one of two actions for each simulacrum, depending on whether the coin shows heads or tails, but the two actions have equal net utility for the simulacra so there are no moral conundra. Now, even though the combination of 90 heads and 10 tails is the most common, the permutations comprising it are nonetheless vastly outnumbered by all the remaining permutations. Suppose that after flipping 100 biased coins, the actual result is 85 heads and 15 tails.

What is the subjective probability? The coin flips are independent events, so the subjective probability of each coin flip must be 9:1 favoring heads. The fact that only 85 simulacra actually experienced heads is completely irrelevant.

Subjective probability arises from knowledge, so in practice none of the simulacra experience a subjective probability after a single coin flip. If the coin flip is repeated multiple times for all simulacra, then as each simulacrum experiences more coin flips while iterating through its state function, it will gradually converge on the objective probability of 90%. The first coin flip merely biases the experience of each simulacrum, determining the direction from which each will converge on the limit.

That said, take what I say with a grain of salt, because I seriously doubt this can be extended from the classical realm to cover quantum simulacra and the Born rule.

Replies from: Chronos
comment by Chronos · 2009-12-02T06:05:35.936Z · LW(p) · GW(p)

And, since I can't let that stand without tangling myself up in Yudkowsky's "Outlawing Anthropics" post, I'll present my conclusion on that as well:

To recapitulate the scenario: Suppose 20 copies of me are created and go to sleep, and a fair coin is tossed. If heads, 18 go to green rooms and 2 go to red rooms; if tails, vice versa. Upon waking, each of the copies in green rooms will be asked "Give $1 to each copy in a green room, while taking $3 from each copy in a red room"? (All must agree or something sufficiently horrible happens.)

The correct answer is "no". Because I have copies and I am interacting with them, it is not proper for me to infer from my green room that I live in heads-world with 90% probability. Rather, there is 100% certainty that at least 2 of me are living in a green room, and if I am one of them, then the odds are 50-50 whether I have 1 companion or 17. I must not change my answer if I value my 18 potential copies in red rooms.

However, suppose there were only one of me instead. There is still a coin flip, and there are still 20 rooms (18 green/red and 2 red/green, depending on the flip), but I am placed into one of the rooms at random. Now, I wake in a green room, and I am asked a slightly different question: "Would you bet the coin was heads? Win +$1, or lose -$3". My answer is now "yes": I am no longer interacting with copies, the expected utility is +$0.60, so I take the bet.

The stuff about Boltzmann brains is a false dilemma. There's no point in valuing the Boltzmann brain scenario over any of the other "trapped in the Matrix" / "brain in a jar" scenarios, of which there is a limitless supply. See, for instance, this lecture from Lawrence Krauss -- the relevant bits are from 0:24:00 to 0:41:00 -- which gives a much simpler explanation for why the universe began with low entropy, and doesn't tie itself into loops by supposing Boltzmann pocket universes embedded in a high-entropy background universe.

comment by PlatypusNinja · 2009-12-01T22:33:18.253Z · LW(p) · GW(p)

It's difficult to answer the question of what our utility function is, but easier to answer the question of what it should be.

Suppose we have an AI which can duplicate itself at a small cost. Suppose the AI is about to witness an event which will probably make it happy. (Perhaps the AI was working to get a law passed, and the vote is due soon. Perhaps the AI is maximizing paperclips, and a new factory has opened. Perhaps the AI's favorite author has just written a new book.)

Does it make sense that the AI would duplicate itself in order to witness this event in greater multiplicity? If not, we need to find a set of utility rules that cause the AI to behave properly.

Replies from: PlatypusNinja
comment by PlatypusNinja · 2009-12-01T22:43:59.210Z · LW(p) · GW(p)

(I'm not sure what the rule is here for replying to oneself. Apologies if this is considered rude; I'm trying to avoid putting TLDR text in one comment.)

Here is a set of utility-rules that I think would cause an AI to behave properly. (Would I call this "Identical Copy Decision Theory"?)

  • Suppose that an entity E clones itself, becoming E1 and E2. (We're being agnostic here about which of E1 and E2 is the "original". If the clone operation is perfect, the distinction is meaningless.) Before performing the clone, E calculates its expected utility U(E) = (U(E1)+U(E2))/2.

  • After the cloning operation, E1 and E2 have separate utility functions: E1 does not care about U(E2). "That guy thinks like me, but he isn't me."

  • Suppose that E1 and E2 have some experiences, and then they are merged back into one entity E' (as described in http://lesswrong.com/lw/19d/the_anthropic_trilemma/ and elsewhere). Assuming this merge operation is possible (because the experiences of E1 and E2 were not too bizarrely disjoint), the utility of E' is the average: U(E') = (U(E1) + U(E2))/2.

Replies from: PlatypusNinja
comment by PlatypusNinja · 2009-12-01T22:54:01.596Z · LW(p) · GW(p)

I think I am happy with how these rules interact with the Anthropic Trilemma problem. But as a simpler test case, consider the following:

An AI walks into a movie theater. "In exchange for 10 utilons worth of cash", says the owner, "I will show you a movie worth 100 utilons. But we have a special offer: for only 1000 utilons worth of cash, I will clone you ten thousand times, and every copy of you will see that same movie. At the end of the show, since every copy will have had the same experience, I'll merge all the copies of you back into one."

Note that, although AIs can be cloned, cash cannot be. ^_^;

I claim that a "sane" AI is one that declines the special offer.

comment by MSRayne · 2022-06-29T15:31:01.372Z · LW(p) · GW(p)

I'm thirteen years late to this conversation, but am I the only person who thinks that B is obviously the correct choice?!

First, all copies will make the same decision, which means I can safely make a choice for all of us knowing that it will be unanimous and I am not coercing anyone.

Second, the certainty of killing 99 beings is far more horrible than the possibility of killing 100 together with the possibility of saving 100. In the former case there is a guarantee of a bad outcome, whereas in the latter there is the possibility, however remote, of success, which is clearly better.

Third, in the former case, choice A, all copies of me would be agreeing to doom all but one of our number to death, which is rather hellish and also selfish - each one secretly wants to be the one who survives, and is willing to throw all the others under the bus to do so, and we all know this too, which would make the whole experience even more dreadful. But in the latter case, choice B, we would be making a heroic gamble which might lead to all of our deaths, or to our salvation, but either way we'd be doing it together. Isn't that intuitively more noble?

Replies from: conor-sullivan
comment by Lone Pine (conor-sullivan) · 2022-06-29T16:28:33.064Z · LW(p) · GW(p)

There's a movie on Netflix called Circle which has a similar premise. In it, a couple dozen people wake up to find themselves arranged in a circle. Every minute, someone dies, and they quickly figure out that they are voting on who to kill next. At the end of the movie

they realize that one 'person' is not voting, an unborn baby inside a pregnant woman. Two teams emerge: one team of callous people realize that the baby can be exploited to be the last person standing, while the other team wants to pre-emptively kill the baby to ensure a more deserving person is the survivor.

This isn't obvious until the very end of the movie, so if you want to watch it, maybe don't read the spoiler.

Replies from: Benito
comment by Ben Pace (Benito) · 2022-06-29T17:18:26.847Z · LW(p) · GW(p)

(It's exclamation point then greater than symbol, not the other way around. Answer in FAQ [? · GW]. Have edited your comment to add it.)

Replies from: conor-sullivan
comment by Lone Pine (conor-sullivan) · 2022-06-30T10:31:34.713Z · LW(p) · GW(p)

Ah, I am dumb, thanks.

comment by Chris_Leong · 2018-07-28T02:02:13.897Z · LW(p) · GW(p)

We can make an outside-view argument that we ought to choose A even if we are 99% sure that it provides no benefit, as we get "identical copy immortality" for free.

That said, let's consider the inside view. I find the idea of 100 copies gaining 100 times as much experience very intuitive. You may ask, how can we prove this? Well, we can't, but we can't prove that anyone else feels anything at all either. Accepting that other people feel qualia, but denying that copies feel qualia on the basis that there's no empirical proof feels like an isolated demand for rigour.

comment by Mallah · 2009-12-01T19:38:55.453Z · LW(p) · GW(p)

"our intuition of identical copy immortality"

Speak for yourself - I have no such intuition.

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-12-01T22:49:29.456Z · LW(p) · GW(p)

I don't claim that everyone has that intuition, which is why I said "I guess that most people would do so..." It seems that most people in these comments, at least, do prefer A.

Replies from: MrHen
comment by MrHen · 2009-12-16T15:23:39.641Z · LW(p) · GW(p)

I don't think that an intuition of identical copy immortality and preferring scenario A are necessarily tied. In other words, people could prefer A for reasons that have nothing to do with identical copy immortality. I don't have a clear-cut example since I am still processing the original post.

comment by Morendil · 2009-12-01T01:08:18.047Z · LW(p) · GW(p)

Interesting...

What seems odd isn't that we should prefer A to B, it's that the Omega-civilization should be indifferent between the two.

Can we assume shutting down all 100 copies also means the starting data and algorithm cannot be later restarted, or only at too high a cost ?

Would the Omega-civilization be indifferent between preserving exactly one of 100 extant copies of an undecipherable text left behind by the mysterious but otherwise quite uninteresting Rhoan culture, or taking a 99:1 gamble between destroying all copies and preserving all ?

Because the Rhoan text can be easily reproduced from one copy, if the Omegans have an interest epsilon in it they should prefer preserving one copy. (If less than epsilon, this whole discussion is already more of an inconvenience to them than chucking all 100 copies down the nearest black hole.)

On this view, what matters is seeing how the Rhoan book turns out in the end; it doesn't seem to make a difference that the "reader" interested in how the book turns out is the book itself (or a character within it). However many copies exist, it's the same reader finding out how the same book turns out - except if zero copies remain.

Replies from: Wei_Dai, rwallace
comment by Wei Dai (Wei_Dai) · 2009-12-01T01:25:18.637Z · LW(p) · GW(p)

The assumption is that the simulation is being run for purely utilitarian purposes, for benefit of the inhabitants. They have no interest in knowing how our universe turns out.

comment by rwallace · 2009-12-01T03:43:06.281Z · LW(p) · GW(p)

It's consistent with the scenario, and probably helps separate the issues, if we assume in either case a copy of the database can be archived at negligible cost.

comment by PhilGoetz · 2009-12-01T01:00:24.716Z · LW(p) · GW(p)

The problem you pose has a more immediately-relevant application: What is a good proportion of resources to devote to non-human life? For instance, when is it better to save a non-human species from extinction, than to turn its habitat into farmland to sustain an equal number of humans? We might agree that one human is worth more than one spotted owl; but not that the ten-billionth human is worth more than the last spotted owl. This is because humans are similar to each other. The identity you invoke is just the most extreme case of similarity.

I've mentioned before on LW that the best moral principle I know, where the goodness of a moral principle is measured as something like

1 / [ bits needed to specify moral principle x Kullback-Leibler divergence(distribution of actions governed by the moral principle || distribution of actions that I endorse)]

is that higher informational complexity is better than lower informational complexity. This nicely deals with this problem, as well as with its generalization to cases where the duplicate copies are not exact duplicates, but merely highly correlated.

Replies from: DanArmak, rwallace, Wei_Dai
comment by DanArmak · 2009-12-01T14:06:40.178Z · LW(p) · GW(p)

We might agree that one human is worth more than one spotted owl; but not that the ten-billionth human is worth more than the last spotted owl.

How does that work? Do you choose the ten-billionth human by lottery and then tell her, sorry you lost, now it's you being weighed against the last spotted owl?

Added: also, 'last' isn't very different from 'first'. How about this scenario: in this universe where spotted owls never existed, I have bio-crafted the First Spotted Owl, a work of art! Unfortunately, it can't breed without a Second Spotted Owl, and I don't have the resources to make that. This makes my First Owl the Last Owl as well, and so worth more than the ten-billionth human. I thus get the moral right to kill said human and transmute it into the Second Owl!

(I have given Spotted Owls onwards the ability of hunting humans for breakfast, so I don't need to worry about the chicks.)

comment by rwallace · 2009-12-01T03:46:00.219Z · LW(p) · GW(p)

I'm not convinced informational complexity gets you what you want here (a cloud of gas in thermal equilibrium has maximum informational complexity). I also don't agree that the last spotted owl is worth more than the 10 billionth human.

Replies from: scav, bgrah449
comment by scav · 2009-12-01T11:13:48.684Z · LW(p) · GW(p)

Yes, it does kind of depend on "worth more to whom?". To the 10-billionth human's mother, no way. The aggregate worth to all humans who benefit in some small way from the existence of spotted owls, maybe.

If you took a vote, most of those humans still might say no (because who knows if the next vote will be whether they themselves are worth less than the last of some species of shrew).

comment by bgrah449 · 2009-12-01T15:15:22.245Z · LW(p) · GW(p)

Upvoted. In the words of Penn Fraser Jillette: "Teller and I would personally kill EVERY chimp in the world, with our bare hands, to save ONE street junkie with AIDS."

Replies from: Alicorn
comment by Alicorn · 2009-12-01T15:53:23.039Z · LW(p) · GW(p)

That statement has always puzzled me a bit. Why does it matter that the junkie has AIDS? That's a death sentence, so either "saving" the junkie means curing the AIDS and it doesn't add anything to stipulate that it was originally suffered (unless it's just an intuition pump about AIDS sufferers tending to be seen as a particularly worthless echelon of humanity?). Or, it means rescuing the junkie from a more immediate danger and leaving the AIDS intact, in which case no real saving has happened - the cause of death has just changed and been put off for a while. And over those remaining years of the junkie's life there's a nontrivial chance that the voluntary slaughter of all those chimps will ultimately result in another AIDS infection, which is another death sentence!

Replies from: bgrah449, DanArmak
comment by bgrah449 · 2009-12-01T18:17:37.788Z · LW(p) · GW(p)

I always took the statement to be more about: Ignoring real-world effects of chimpanzees going extinct, no amount of animal death, in and of itself, is considered more horrible than any amount of human death. Animal life has no inherent worth. None.

Replies from: DanArmak
comment by DanArmak · 2009-12-01T18:22:31.309Z · LW(p) · GW(p)

Neither does most human life, according to many people who agree with this statement.

comment by DanArmak · 2009-12-01T18:21:35.461Z · LW(p) · GW(p)

I assumed the meaning was 'to save one junkie with AIDS from some imminent death that has nothing to do with junkie-ness or AIDS'. I.e., I would value even a few extra months of a junkie's life over any amount of chimpanzee lives.

comment by Wei Dai (Wei_Dai) · 2009-12-01T01:49:13.534Z · LW(p) · GW(p)

The identity you invoke is just the most extreme case of similarity.

Yes, this is what I was referring to in the sentence starting "A full account should also address more complex cases". I wanted to start with identity because we can establish that discounting for similarity is necessary in at least one case, due to the relative clarity of our intuition in that case. But I think once we move away from fully identical copies, we'd have a lot more disagreement about how the discounting should work. One might argue that even with a billion humans, the discount factor should be negligible.

I've mentioned before on LW that the best moral principle I know

Do you have a full write-up of this somewhere? I can't make much sense of it from what you wrote in the comment.

Replies from: PhilGoetz
comment by PhilGoetz · 2009-12-01T03:17:10.008Z · LW(p) · GW(p)

No write-up. The idea is that you can decide between two situations by choosing the one with greater information or complexity. The trickiness is in deciding how to measure information or complexity, and in deciding what to measure the complexity of. You probably don't want to conclude that, in a closed system, the ethically best thing to do is nothing because doing anything increases entropy. (Perhaps using a measure of computation performed, instead of a static measure of entropy, would address that.)

This gives you immediately a lot of ethical principles that are otherwise difficult to justify; such as valuing evolution, knowledge, diversity, and the environment; and condemning (non-selective) destruction and censorship. Also, whereas most ethical systems tend to extreme points of view, the development of complexity is greatest when control parameters take on intermediate values. Conservatives value stasis; progressives value change; those who wish to increase complexity aim for a balance between the two.

(The equation in my comment is not specific to that idea, so it may be distracting you.)

Replies from: Pfft, MendelSchmiedekamp
comment by Pfft · 2009-12-01T05:17:39.226Z · LW(p) · GW(p)

(Perhaps using a measure of computation performed, instead of a static measure of entropy, would address that.)

This is exactly what I have been thinking for a while also. In this view, when thinking about how bad it would be to destroy something, one should think about how much computation it would take to recreate it. I think this seems like a really promising idea, because it gives a unified reason to be against both murder and destruction of the rain forests.

Still, it is probably not enough to consider only the amount of computation -- one could come up with counterexamples of programs computing really boring things...

comment by MendelSchmiedekamp · 2009-12-01T18:05:02.053Z · LW(p) · GW(p)

This parallels some of the work I'm doing with fun-theoretic utility, at least in terms of using information theory. One big concern is what measure of complexity to use, as you certainly don't want to use a classical information measure - otherwise Kolmogorov random outcomes will be preferred to all others.

comment by Tyrrell_McAllister · 2009-12-01T00:27:59.160Z · LW(p) · GW(p)

On the one hand, I value that certain patterns are represented in reality by at least one lump of matter, irrespective of how many such lumps there are. On the other hand, I value that a particular lump of matter evolve within certain constraints (e.g., not dissolve into dust), irrespective of what other lumps of matter are doing. These two values have certain weights attached to them, and these weights determine (via expected utility) how I will choose between the values if I'm forced to do so.

In your example, I would likely agree with you and choose option A. I attach a very high value to some lump of matter realizing the pattern of our universe. Therefore, the possibility under option B that all the copies will be destroyed is a huge negative. The possibility that all the copies would be retained is a big positive (because I care about my own lump of matter), but not big enough.

On reflection, I believe that my values work thus: Once I'm assured that some copy of this universe will continue, I only care about another copy insofar as it is my own. As far as my values are concerned, there is no difference between 1 copy and 3^^^3 copies if none of them are my own copy.

comment by Steve_Rayhawk · 2009-12-02T16:43:24.656Z · LW(p) · GW(p)

In these kinds of problems, I notice that my intuitions give some of (!) the same answers as if I was trying to think about the long- or medium-range future effects of a potential life, and about how much it would cost to get future effects that were as good as the effects that such a life would have had.

Some of the effects of a person's life are on material wealth produced and consumed. But other than material wealth, the future effects that seem hardest to replace are the effects of a person noticing possibilities or perspectives or conceptualizations or ways of thinking, and advocating those perspectives so that other people know about them afterwards. The reason that these effects are hard to replace is that the way you would replace them would be by imagining a random hypothetical person and imagining the perspectives they would have had. But humans can't do this if they don't already have those perspectives; instead they imagine perspectives that are already stated in terms familiar to them.

Part of what I mean here was explained in the LW article "The Value of Nature and Old Books":

when you are in nature or contemplating nature, you can be confident that the resulting thoughts and feelings are uncontaminated by all of the (visible and invisible) ideas and biases and assumptions that are present in your particular time and place.

Part of it is explained in a theory promoted by complex-systems theorists such as Stuart Kaufmann: when complex structures (like organisms, or whole computer systems, or perspectives) are in evolutionary competition, new basic structures (like the Baupläne ("body plans") in evolutionary developmental biology, or computing platforms) are usually only explored at the beginning of the competition (like the Cambrian explosion, or the computer revolution), because later all the competitive niches are filled with highly-adapted versions of the basic structures, and changing the basic structure would make for an uncompetitive total structure. This is true even if a better basic structure is known, which would have outcompeted all the existing structures if it had had the same amount of time to be adapted. The same is probably true for perspectives competing to be able to represent (or at least not contradict) our knowledge as we learn. When we have old perspectives which can represent current knowledge, and we learn something new, the way we update our predictions is by adapting the old perspectives until they can represent the new knowledge. But if other perspectives would have been adapted to represent all the knowledge just as well, we should also mix in the predictions they would have made, to reduce the relative entropy from our predictions to the truth. And if another perspective would have been adapted to represent all the knowledge better, we should use that perspective instead, and give the other perspectives less weight.

Part of what I mean is related to some of the intuitions which support the idea of "openness to the unbidden", a phrase used by theologian William May and quoted by political philosopher Michael Sandel to express a bioconservative position. (Though, in eyes like mine, those intuitions more support ideas like "bidding a particular sampling of the otherwise unbidden, according to a sequential design optimized for the benefits of openness", or even "quadrature with the otherwise unbidden".)

Part of what I mean is related to some of the intuitions which support randomization protocols, as in double-blind controlled studies, product testing, or auditing.

Part of what I mean is explained by the analogy: When political opinions might be censored, we don't trust everyone else to substitute for free speech by imagining the expression of those opinions. When a political official is democratically elected, we don't trust them to substitute for balloting by imagining when their constituents would elect someone to replace them. When someone is tried, we don't trust a jury to substitute for a defense attorney (or prosecution attorney) by imagining what the attorney might argue. When a scientist argues for a conclusion from some evidence, we don't trust the scientist to substitute for criticism by imagining what other scientists would say about it. When someone might die, we don't trust other people to substitute for his life by imagining the difference he would have tried to make and his reasons for trying to make it. We don't trust a central economic planner to substitute for the information from a market mechanism by imagining what everyone might need everywhere. In experimental mathematics, we don't trust people to substitute for computational results by imagining them. So might we not trust people to substitute for the perspectives of potential people by imagining what their perspectives would be.

(As an opposite example, we would trust someone to substitute for the random details of an ideal gas by using another sample of the same gas.)

This argument also helps with the Repugnant Conclusion. In a world where there are many many people who all have lives barely worth living, one expects none of them to rise to a level where they can accomplish much to make a better world with better things in it, compared to a world where there are fewer people and some of them have lives that are more worth living. If "barely worth living" is redefined to include "barely making worthwhile improvements to the future", then the Repugnant Conclusion might evaporate.

My argument about expensive-to-replace perspectives is probably not a good causal explanation of our intuitions about the value of duplicates, though. Some of the cause is probably from instincts around the problem of distinguishing imagined futures in which one is alive (1+ copy--"that's me!") from imagined futures in which one is dead (0 copies).

I think it's interesting that other arguments in these comments, like Richard Chappell's about "holism" or Phil Goetz's about "information-theoretic complexity", also try to justify valuing duplicates less by doing something to open up the ability to argue about consequences that come after an experience had by the potential person.

Someone else must have already said this, but I also feel like the "goodness" or "badness" that we associate with experiences resembles what you would expect in an approximate reinforcement learning system which put changes in expected utility as near as possible to the actions that might have caused them. The implied utility might be defined over events which happen at times that are different from the times at which there are changes to reinforcement learning system's "beliefs" about those events. Whether you reflectively care about the implied utility is a subject for metaethics, but this is a reason to expect that we should feel confused about how to locate "intrinsic goodness" within time.

But what would it mean for the value of a life to be the value of its long- or medium-range future effects, if the only way those effects could have value is by changing other lives so that they have valuable future effects? What would be easiest for me in the short run is to bite the bullet and say that the only worlds that matter are worlds with infinite futures, like Richard Hollerith has argued, or worlds that are otherwise mathematically related (e.g. by simulation or logical constraints) to an infinitely detailed context. But others would disagree. One argument they could make is that a renormalization of values using a fixed point of this kind of implied infinite regress, done honestly, doesn't have to have the nice features it might seem to have ("valuing evolution, knowledge, diversity, and the environment; and condemning (non-selective) destruction and censorship") during the first few steps.

Replies from: Vladimir_Nesov
comment by Vladimir_Nesov · 2009-12-02T17:22:49.557Z · LW(p) · GW(p)

Unfortunately, this comment is incomprehensible.

Replies from: Steve_Rayhawk
comment by Steve_Rayhawk · 2009-12-02T18:58:34.469Z · LW(p) · GW(p)

Oh. I wasn't thinking as much about comprehensibility as maybe I should have been. I'll take it down and try to rewrite it.

Replies from: Vladimir_Nesov
comment by Vladimir_Nesov · 2009-12-02T20:27:38.984Z · LW(p) · GW(p)

You comment rarely, but when you do, it seems that you are making too much effort (esp. in the context of how few people will get the gist of what you are saying), elaborating a lot of detail. Now that you've taken even more effort on yourself (or reduced the number of people who would be able to read what you've written, since you've taken it down), I feel guilty. :-(

Replies from: Wei_Dai
comment by Wei Dai (Wei_Dai) · 2009-12-02T23:39:34.588Z · LW(p) · GW(p)

I wish Steve had just added a note indicating that he's rewriting the comment, and didn't delete it.

comment by RichardChappell · 2009-12-01T16:47:45.328Z · LW(p) · GW(p)

I discuss this a bit (along with some related issues, e.g. the repugnant conclusion) in my Value Holism paper. Feedback welcome!

comment by Stuart_Armstrong · 2009-12-01T13:30:21.534Z · LW(p) · GW(p)

The issue here is that the setup has a firm answer - A - but if it were tweaked ever so slightly, the whole preferences would change.

First of all, consider the following:

A': there is one genuine simulation, and the other 99 are simple copies of that one. Soon, all the copies will be stopped, but the the true simulation will continue.

There are essentially no practical ways of distinguishing A from A'. So we should reason as if A' were correct, in which case nothing is lost from the turning off.

However, if we allow any divergence between the copies, then this is a whole other kettle of barreled fish. Now there are at least a hundred copies of each person, and they are distinct: they think slightly differently, behave slightly differenctly, have slightly different goals, etc... And this divergence will only grow with time.

There are three ways I can think of addressing this:

1) Once copies diverge at all, they are different people. So any divergence results in multiplying by a hundred the amount of people in the universe. Hence, if we suspect divergence, we should treat each copy as totally distinct.

2) An information theoretic approach: given one individual, how much new information is needed to completely descibe another copy. Under this, a slightly divergent copy counts as much less than a completely new individual, while a very divergent copy is nearly entriely different.

Both these have drawbacks - 1) gives too much weight to insignificant and minute changes, and 2) implies that people in general are of nothing like approximately equal worth: only extreme exceentrics count as complete people, others are of much less importance. So I suggest a compromise:

3) An information theoretical cut off: two slightly divergent copies count as only slightly more than a single person, but once this divergence rises above some critical amount, the two are treated as entirely seperate individuals.

What is the relevance of this to the original problem? Well, as I said, the original problem has a clear-cut solution, but it is very close to the different situation I described. So, bearing in mind imperfect information, issues of trust and uncertainty, and trying to find similar situations to similar problems, I think we should treat the set-up as the "slightly divergent one". In this situation, A still dominates, but the relative attraction of B rises as the divergence grows.

EDIT: didn't explain the cutoff properly; I was meaning a growing measure of difference that hits a maximum at the cutoff point, and doesn't grow any further. My response here gives an example of this.

Replies from: Jonii, quanticle
comment by Jonii · 2009-12-01T17:21:06.659Z · LW(p) · GW(p)

Bite the bullet and select 2. There doesn't really seem to be anything inherently wrong with that, while 3 seems ad hoc and thus bad.

You seem to underestimate the difference between two human minds, let alone other minds. Additional benefit from method 2 is that it explains why human suffering is more "wrong" than to some ants suffering. This I guess is intuitive way most here think.

Edit: Fixed a lot of typos

Replies from: Stuart_Armstrong
comment by Stuart_Armstrong · 2009-12-03T13:53:51.372Z · LW(p) · GW(p)

3 seems approximately how we deal with people in reality - we say things like "A is so like B" or "A is so unlike B" without thinking that A and B are any less seperate individuals with distinct rights, legal and moral statuses. It's only when A and B get much too close in their reactions, that we flip into a different mode and wonder whether they are trully seperate.

Since this is the moral intuition, I see no compelling reason to discard it. It doesn't seem to contradict any major results, I haven't yet seen thought experiments where it becomes ridiculous, and it doesn't over-burden our decision process.

If any of my statements there turn out to be wrong, then I would consider embrasing 2).

Replies from: Jonii
comment by Jonii · 2009-12-03T15:45:26.735Z · LW(p) · GW(p)

Actually, I updated thanks to reading this paper by Bostrom, so I gotta rephrase stuff.

First, two identical people living in separate simulations are just as much separate people as any other separate people are separate people. It doesn't matter if there exist an identical replica somewhere else, the value of this person in particular doesn't decrease tinyest bit. They're distinct but identical.

Second, uniqueness of their identity decreases as there are more people like them, as you described with option #2. However, this property is not too interesting, since it has nothing to do with their personal experiences.

So what we get is this: Simulations together hold 100x our experiences, and we're either offered a deal that kills 99x of us, or the one that 99% certainly kills us. Both have same expected utility. On both cases, we cease to be with 99% certainty. Both have the same negative expected utility.

But the interesting thing happens when we try to ensure continued flow of us existing, since by some magic, it seems to favor A to B, when both seem to be perfectly equal. I'm kinda feeling that problematic nature of handling divergence comes from irrational nature of this tendency to value continued flow of existence. But dunno.

Replies from: Vladimir_Nesov, Stuart_Armstrong
comment by Vladimir_Nesov · 2009-12-03T16:05:37.649Z · LW(p) · GW(p)

First, two identical people living in separate simulations are just as much separate people as any other separate people are separate people. It doesn't matter if there exist an identical replica somewhere else, the value of this person in particular doesn't decrease tinyest bit. They're distinct but identical.

The problem with this is that preference is for choosing actions, and actions can't be about specific people only, they are about the whole reality. The question of how much you value a person only makes sense in the context of a specific way of combining valuations of individual people into valuations of reality.

comment by Stuart_Armstrong · 2009-12-04T11:19:15.976Z · LW(p) · GW(p)

I've read the paper, and disagree with it (one flippant way of phrasing my disagreement is to enquire as to whether reflections in mirrors have identical moral status). See the beggining of my first post for a better objection.

comment by quanticle · 2009-12-02T01:59:15.992Z · LW(p) · GW(p)

The problem with option 3 is that its fundamentally intuitionist, with arbitrary cutoffs distinguishing "real" individuals from copies. I mean, is there really such a big difference between cutoff - .001 difference and cutoff + .001 difference? There isn't. Unless you can show that there's a qualitative difference that occurs when that threshold is crossed, its much more elegant to look at a distinction between options 1 and 2 without trying to artificially shift the boundary between the two.

Replies from: Stuart_Armstrong, pengvado
comment by Stuart_Armstrong · 2009-12-03T13:56:45.402Z · LW(p) · GW(p)

Didn't phrase clearly what I meant by cut-off.

Let D be some objective measure of distance (probably to do with Kologomorov complexity) between individuals. Let M be my moral measure of distance, and assume the cut-off is 1.

Then I would set M(a,b) = D(a,b) whenever D(a,b) < 1, and M(a,b) = 1 whenever D(a,b) >= 1. The discontinuity is in the derivative, not the value.

Replies from: Prolorn
comment by Prolorn · 2009-12-04T07:31:23.703Z · LW(p) · GW(p)

That doesn't resolve quanticle's objection. Your cutoff still suggests that a reasonably individualistic human is just as valuable as, say, the only intelligent alien being in the universe. Would you agree with that conclusion?

Replies from: Stuart_Armstrong
comment by Stuart_Armstrong · 2009-12-04T11:17:47.155Z · LW(p) · GW(p)

No. I grant special status to exceedingly unique minds, and to the last few of a given species.

But human minds are very similar to each other, and granting different moral status to different humans is a very dangerous game. Here, I am looking at the practical effects of moral systems (Eliezer's post on "running on corrupted hardware" is relevant). The thoeretical gains of treating humans as having varrying moral status are small; the practical risks are huge (especially as our societies, though cash, reputation and other factors, is pretty good at distinguishing between people without having to further grant them different moral status).

One cannot argue: "I agree with moral system M, but M has consequence S, and I disagree with S". Hence I cannot agree with granting people different moral status, once they are sufficiently divergent.

comment by pengvado · 2009-12-02T04:03:25.983Z · LW(p) · GW(p)

Stuart's option 3 says that the difference between "cutoff - .001" and "cutoff + .001" is .001 (as opposed to .002 that it wold be if you value the divergence directly). i.e. cutoff is the point at which your distance metric saturates. It's a nonlinearity, but not a discontinuity.

comment by rosyatrandom · 2009-12-01T12:14:15.258Z · LW(p) · GW(p)

Since morality is subjective, then don't the morals change depending upon what part of this scenario you are in (inside/outside)?

I operate from the perspective (incidentally, I like the term 'modal immortality') that my own continued existence is inevitable; the only thing that changes is the possibility distribution of contexts and ambiguities. By shutting down 99/100 instances, you are more affecting your own experience with the simulations than their's with you (if the last one goes, too, then you can no longer interact with it), especially if, inside a simulation, other external contexts are also possible.

Replies from: Vladimir_Nesov
comment by Vladimir_Nesov · 2009-12-01T21:34:41.952Z · LW(p) · GW(p)

If you start out with a fixed "agent design" and then move it around, then preferences for the world by that agent will depend on where you put it. But given an agent embedded in the world in a certain way, it will prefer its preferences to tell the same story if you move it around (if it gets the chance of knowing about the fact that you move it around and how).

comment by Pablo (Pablo_Stafforini) · 2009-12-01T12:00:04.949Z · LW(p) · GW(p)

utilitarianism says that each individual should be valued independently of others, and then added together to form an aggregate value. This seems to imply that each additional copy should receive full, undiscounted value, in conflict with the intuition of identical copy immortality.

What you call the intuition of identical copy immortality is a self-interested intuition: we have a stronger interest in that someone numerically identical to us survives than in the survival of a mere perfect copy of us. But utilitarianism is not a theory of self-interest: it is a moral theory. Accordingly, I don't see any conflict here. Your preference for A over B is explained simply by the fact that you are self-interested; if you were a purely impartial utilitarian, you would be strictly indifferent between A and B.

comment by Jonathan_Graehl · 2009-12-01T06:03:10.667Z · LW(p) · GW(p)

"To Be" is cute and somewhat apropos.

comment by snarles · 2009-12-02T03:21:58.959Z · LW(p) · GW(p)

There may be no answer to your questions, but in this case, it doesn't matter. Morality only has value to the extent that it serves the individual or group.

The denizens of this simulation have little to no emotional connection to the parallel copies; nor are there any foreseeable consequences to our influence in the annihilation of these unreachable worlds; therefore, any moral or ethical system that actually has relevance to us will probably choose A.