The Anthropic Trilemma

I've been thinking about this topic, off and on, at least since September 1997, when I joined the Extropians mailing list, and sent off a "copying related probability question" (which is still in my "sent" folder but apparently no longer archived anywhere that Google can find). Both Eliezer and Nick were also participants in that discussion. What are the chances that we're still trying to figure this out 12 years later?

My current position, for what it's worth, is that anticipation and continuity of experience are both evolutionary adaptations that will turn maladaptive when mind copying/merging becomes possible. Theoretically, evolution could have programmed us to use UDT, in which case this dilemma wouldn't exist now, because anticipation and continuity of experience is not part of UDT.

So why don't we just switch over to UDT, and consider the problem solved (assuming this kind of self-modification is feasible)? The problem with that is that much of our preferences are specified in terms of anticipation of experience, and there is no obvious way how to map those onto UDT preferences. For example, suppose you’re about to be tortured in an hour. Should you make as many copies as you can of yourself (who won’t be tortured) before the hour is up, in order to reduce your anticipation of the torture experience? You have to come up with a way to answer that question before you can switch to UDT.

One approach that I think is promising, which Johnicholas already suggested, is to ask "what would evolution do?" The way I interpret that is, whenever there’s an ambiguity in how to map our preferences onto UDT, or where our preferences are incoherent, pick the UDT preference that maximizes evolutionary success.

But a problem with that, is that what evolution does depends on where you look. For example, suppose you sample Reality using some weird distribution. (Let’s say you heavily favor worlds where lottery numbers always come out to be the digits of pi.) Then you might find a bunch of Bayesians who use that weird distribution as their prior (or the UDT equivalent of that), since they would be the ones having the most evolutionary success in that part of Reality.

The next thought is that perhaps algorithmic complexity and related concepts can help here. Maybe there is a natural way to define a measure over Reality, to say that most of Reality is here, and not there. And then say we want to maximize evolutionary success under this measure.

How to define “evolutionary success” is another issue that needs to be resolved in this approach. I think some notion of “amount of Reality under one’s control/influence” (and not “number of copies/descendants”) would make the most sense.

↑ comment by davidad · 2021-12-10T12:25:50.480Z · LW(p) · GW(p)

I just happened to see this whilst it happens to be 12 years later. I wonder what your sense of this puzzle is now (object-level as well as meta-level).

Replies from: Wei_Dai

↑ comment by Wei Dai (Wei_Dai) · 2021-12-12T19:53:03.368Z · LW(p) · GW(p)

I'm not really aware of any significant progress since 12 years ago. I've mostly given up working on this problem, or most object-level philosophical problems, due to slow pace of progress and perceived opportunity costs. (Spending time on ensuring a future where progress on such problems can continue to be made [LW · GW], e.g., fighting against x-risk and value/philosophical lock-in or drift, seems a better bet even for the part of me that really wants to solve philosophical problems.) It seems like there's a decline in other LWer's interest in the problem, maybe for similar reasons?

Whatever the correct answer is, the first step towards it has to be to taboo words like "experience" in sentences like, "But if I make two copies of the same computer program, is there twice as much experience, or only the same experience?"

What making copies is, is creating multiple instances of the same pattern. If you make two copies of a pattern, there are twice as many instances but only one pattern, obviously.

Are there, then, two of 'you'? Depends what you mean by 'you'. Has the weight of experience increased? Depends what you mean by 'experience'. Think in terms of patterns and instances of patterns, and these questions become trivial.

I feel a bit strange having to explain this to Eliezer Yudkowsky, of all people.

To condense my response to a number of comments here:

It seems to me that there's some level on which, even if I say very firmly, "I now resolve to care only about future versions of myself who win the lottery! Only those people are defined as Eliezer Yudkowskys!", and plan only for futures where I win the lottery, then, come the next day, I wake up, look at the losing numbers, and say, "Damnit! What went wrong? I thought personal continuity was strictly subjective, and I could redefine it however I wanted!"

You reply, "But that's just because you're defining 'I' the old way in evaluating the anticipated results of the experiment."

And I reply, "...I still sorta think there's more to it than that."

To look at it another way, consider the Born probabilities. In this case, Nature seems to have very definite opinions about how much of yourself flows where, even though both copies exist. Now suppose you try to redefine your utility function so you only care about copies of yourself that see the quantum coin land heads up. Then you are trying to send all of your measure to the branch where the coin lands up heads, by exercising your right to redefine personal continuity howsoever you please; whereas Nature only wants to send half your measure there. Now flip the coin a hundred times. I think Nature is gonna win this one.

Tired of being poor? Redefine personal continuity so that tomorrow you continue as Bill Gates and Bill Gates continues as you - just better hope Gates doesn't swap again the next day.

It seems to me that experience and anticipation operate at a more primitive level than my utility function. Perhaps I am wrong. But I would like a cleaner demonstration of how I am wrong, than pointing out how convenient it would be if there were no question.

Of course it must be a wrong question - it is unanswerable, therefore, it is a wrong question. That is not the same as there being no question.

It's helpful in these sorts of problems to ask the question "What would evolution do?". It always turns out to be coherent, reality-based actions. Even though evolution, to the extent that it "values" things, values different things than I do, I'd like my actions to be comparably coherent and reality-based.

Regarding the first horn: Regardless of whether simple algorithms move "subjective experience" around like a fluid, if the simple algorithms take some resources, evolution would not perform them.

Regarding the second horn: If there was an organism that routinely split, merged, played lottery, and priced side-bets on whether it had won the lottery, then, given zero information about whether it had won the lottery, it would price the side-bet at the standard lottery odds. Splitting and merging, so long as the procedure did not provide any new information, would not affect its price.

Regarding the third horn: Evolution would certainly not create an organism that hurls itself off cliffs without fear. However, this is not because of it "cares" about any thread of subjective experience. Rather, this is because of the physical continuity. Compare this with evolution's choice in an environment where there are "transporters" that accurately convey entities by molecular disassembly/reassembly. Creatures which had evolved in that environment would certainly step through those transporters without fear.

I can't answer the fourth or fifth horns; I'm not sure I understand them.

When you wake up, you will almost certainly have won (a trillionth of the prize). The subsequent destruction of winners (sort of - see below) reduces your probability of being the surviving winner back to one in a billion.

Merging N people into 1 is the destruction of N-1 people - the process may be symmetrical but each of the N can only contribute 1/N of themself to the outcome.

The idea of being (N-1)/N th killed may seem a little odd at first, but less so if you compare it to the case where half of one person's brain is merged with half of a different person's (and the leftovers discarded).

EDIT: Note that when the trillion were told they won, they were actually being lied to - they had won a trillionth part of the prize, one way or another.

Whenever I read about "weight of experience", "quantum goo", "existentness" etc. I can't keep myself of also thinking of "vital spark", "phlogiston", "ether" and other similar stuff... And it somehow spoils the whole fun...

In the history of mankind, hard looking (meta-)physical dilemmas were much more often resolved by means of elimination rather than by introduction of new "essences". The moral of the history of physics so far is that relativity typically trumps absoluteness in the long run.

For example, I would not be surprised at all, if it turned out that experienced Born probabilities would not be absolute, but would depend on some reference frame (in a very high dimensional space) just like the experience of time, speed, mass, etc. depends on the relativistic frame of reference.

Following Nominull and Furcas, I bite the third bullet without qualms for the perfectly ordinary obvious reasons. Once we know how much of what kinds of experiences will occur at different times, there's nothing left to be confused about. Subjective selfishness is still coherent because you're not just an arbitrary observer with no distinguishing characteristics at all; you're a very specific bundle of personality traits, memories, tendencies of thought, and so forth. Subjective selfishness corresponds to only caring about this one highly specific bundle: only caring about whether someone falls off a cliff if this person identifies as such-and-such and has such-and-these specific memories and such-and-those personality traits: however close a correspondence you need to match whatever you define as personal identity.

The popular concepts of altruism and selfishness weren't designed for people who understand materialism. Once you realize this, you can just recast whatever it was you were already trying to do in terms of preferences over histories of the universe. It all adds up to, &c., &c.

Just an aside - this is obviously something that Eliezer - someone highly intelligent and thoughful - has thought deeply about, and has had difficulty answering.

Yet most of the answers - including my own - seem to be of the "this is the obvious solution to the dilemma" sort.

Bonus: if you're uncomfortable with merging/deleting copies, you can skip that part! Just use the lottery money to buy some computing equipment and keep your extra copies running in lockstep forever. Is this now an uncontroversial algorithm for winning the lottery, or what?

More near-equivalent reformulations of the problem (in support of the second horn):

• A trillion copies will be created, believing they have won the lottery. All but one will be killed (1/trillion that your current state leads directly to your future state). If you add some uniportant differentiation between the copies - give each one a speratate number - then the situation is clearer: you have one chance in a trillion that the future self will remember your number (so your unique contribution will have 1/trillion chance of happening), while he will be certain to believe he has won the lottery (he gets that belief from everyone.

• A trillion copies are created, each altruistically happy that one among the group has won the lottery. One of them at random is designated the lottery winner. Then everyone else is killed.

• Follow the money: you (and your copies) are not deriving utility from winning the lottery, but from spending the money. If each copy is selfish, there is no dilema: the lottery winnings divided amongst a trillion cancels out the trillion copies. If each copy is altruistic, then the example is the same as above; in which case there is a mass of utility generated from the copies, which vanish when the copies vanish. But this extra mass of utility is akin to the utility generated by: "It's wonderful to be alive. Quick, I copy myself, so now many copies feel it's wonderful to be alive. Then I delete the copies, so the utility goes away".

Time and (objective or subjective) continuity are emergent notions. The more basic notion they emerge from is memory. (Eliezer, you read this idea in Barbour's book, and you seemed to like it when you wrote about that book.)

Considering this: yes, caring about the well-being of "agents that have memories of formerly being me" is incoherent. It is just as incoherent as caring about the well-being of "agents mostly consisting of atoms that currently reside in my body". But in typical cases, both of these lead to the same well known and evolutionarily useful heuristics.

I don't think any of the above implies that "thread of subjective experience" is a ridiculous thing, or that you can turn into being Britney Spears. Continuity being an emergent phenomenon does not mean that it is a nonexistent one.

As for what happens ten seconds after that, you have no way of knowing how many processors you run on, so you shouldn't feel a thing

Here's the problem, as far as I can see. You shouldn't feel a thing, but that would also be true if none of you ever woke up again. "I won't notice being dead" is not an argument that you won't be dead, so lottery winners should anticipate never waking up again, though they won't experience it (we don't anticipate living forever in the factual world, even though no one ever notices being dead).

I'm sure there's some reason this is considered invalid, since quantum suicide is looked on so favorably around here. :)

I still have trouble biting that bullet for some reason. Maybe I'm naive, I know, but there's a sense in which I just can't seem to let go of the question, "What will I see happen next?" I strive for altruism, but I'm not sure I can believe that subjective selfishness - caring about your own future experiences - is an incoherent utility function; that we are forced to be Buddhists who dare not cheat a neighbor, not because we are kind, but because we anticipate experiencing their consequences just as much as we anticipate experiencing our own. I don't think that, if I were really selfish, I could jump off a cliff knowing smugly that a different person would experience the consequence of hitting the ground.

I don't really understand your reasoning here. It's not a different person that will experience the consequences of hitting the ground, it's Eliezer+5. Sure, Eliezer+5 is not identical to Eliezer, but he's really, really, really similar. If Eliezer is selfish, it makes perfect sense to care about Eliezer+5 too, and no sense at all to care equally about Furcas+5, who is really different from Eliezer.

And the third horn of the trilemma is to reject the idea of the personal future - that there's any meaningful sense in which I can anticipate waking up as myself tomorrow, rather than Britney Spears. Or, for that matter, that there's any meaningful sense in which I can anticipate being myself in five seconds, rather than Britney Spears. In five seconds there will be an Eliezer Yudkowsky, and there will be a Britney Spears, but it is meaningless to speak of the current Eliezer "continuing on" as Eliezer+5 rather than Britney+5; these are simply three different people we are talking about.

There are no threads connecting subjective experiences. There are simply different subjective experiences. Even if some subjective experiences are highly similar to, and causally computed from, other subjective experiences, they are not connected.

I still have trouble biting that bullet for some reason. Maybe I'm naive, I know, but there's a sense in which I just can't seem to let go of the question, "What will I see happen next?" I strive for altruism, but I'm not sure I can believe that subjective selfishness - caring about your own future experiences - is an incoherent utility function; that we are forced to be Buddhists who dare not cheat a neighbor, not because we are kind, but because we anticipate experiencing their consequences just as much as we anticipate experiencing our own. I don't think that, if I were really selfish, I could jump off a cliff knowing smugly that a different person would experience the consequence of hitting the ground.

I do bite the bullet, but I think you are wrong about the implications of biting this bullet.

Eliezer Yudkowsky cares about what happens to Eliezer Yudkowsky+5 seconds, in a way that he doesn't care about what happens to Ishaan+5 or Brittany+5.

E+5 holds a special place in E's utility function. To E, universes in which E+5 is happy are vastly superior to universes in which E+5 is unhappy or dead.

It makes no difference to E that E+5 is not identical to E. E still cares about E+5, and E aught not need any magic subjective thread connecting E and E+5 to justify this preference. It's not incoherent to prefer a future where certain entities that are causally connected to you continue to thrive - That's all "selfishness" really means.

E anticipates the universe that E+5 will experience. E+5 will carry the memory of this anticipation. If there are lotteries and clones, E will anticipate a universe with a 1% chance of a bunch of E+5 clones winning the lottery and a 99% chance of no E+5 clones winning the lottery. Anticipation is expectation concerning what you+5 will experience in the future. You're basically imagining your future self and experiencing a specialized and extreme version of "empathy". It doesn't matter whether or not there is a magical thread tying you to your future self. If you strip the emotional connotation on "anticipation" and just call it "prediction", you can even predict what happens after you die (it's just that there is no future version of you to "empathize" with anymore)

There are no souls. That holds spatially and temporally.

Oddly, I feel myself willing to bite all three bullets. Maybe I am too willing to bite bullets? There is a meaningful sense in which I can anticipate myself being one of the future people who will remember being me, though perhaps there isn't a meaningful way to talk about which of those many people I will be; I will be all of them.

I suggested that, in some situations, questions like "What is your posterior probability?" might not have answers, unless they are part of decision problems like "What odds should you bet at?" or even "What should you rationally anticipate to get a brain that trusts rational anticipation?". You didn't comment on the suggestion, so I thought about problems you might have seen in it.

In the suggestion, the "correct" subjective probability depends on a utility function and a UDT/TDT agent's starting probabilities, which never change. The most important way the suggestion is incomplete is that it doesn't itself explain something we do naturally: we care about the way our "existentness" has "flowed" to us, and if we learn things about how "existentness" or "experiencedness" works, we change what we care about. So when we experiment on quantum systems, and we get experimental statistics that are more probable under a Born rule with a power of 2 than (hand-waving normalization problems) under a Born rule with a power of 4, we change our preferences, so that we care about what happens in possible future worlds in proportion to their integrated squared amplitude, and not in proportion to the integral of the fourth power. But, if there were people who consistently got experimental statistics that were more probable under a Born rule with a power of 4 (whatever that would mean), we would want them to care about possible future worlds in proportion to the integral of the fourth power of their amplitude.

This can even be done in classical decision theory. Suppose you were creating an agent to be put into a world with Ebborean physics, and you had uncertainty about whether, in the law relating world-thickness ratios (at splitting time) to "existentness" ratios, the power was 2 or 4. It would be easy to put a prior probability of 1/2 on each power, and then have "the agent" update from measurements of the relative thicknesses of the sides of the split worlds it (i.e. its local copy) ended up on. But this doesn't explain why you would want to do that.

What would a UDT/TDT prior belief distribution or utility function have to look like in order to define agents that can "update" in this way, while only thinking in terms of copying and not subjective probability? Suppose you were creating an agent to be put into a world with Ebborean physics, and you had uncertainty about whether, in the relation between world thickness ratios and "existentness" ratios, the power was 2 or 4. And this time, suppose the agent was to be an updateless decision theory agent. I think a UDT agent which uses "probability" can be converted by an expected utility calculation into a behaviorally equivalent UDT agent which uses no probability. Instead of probability, the agent uses only "importances": relative strengths of its (linearly additive) preferences about what happens in the various deterministic worlds the agent "was" copied into at the time of its creation. To make such an agent in Ebborean physics "update" on "evidence" about existentness, you could take the relative importance you assigned to influencing world-sheets, split it into two halves, and distribute each half across world-sheets in a different way. Half of the importance would be distributed in proportion to the cumulative products of the squares of the worlds' thickness ratios at their times of splitting, and half of the importance would be distributed in proportion to the cumulative products of the fourth powers of the worlds' thickness ratios at their times of splitting. Then, in each world-sheet, the copy of the agent in that world-sheet would make some measurements of the relative thicknesses on its side of a split, and it would use use those measurements to decide what kinds of local futures it should prioritize influencing.

But, again, this doesn't explain why you would want to do that. (Maybe you wanted the agents to take a coordinated action at the end of time using the world-sheets they controlled, and you didn't know which kinds of world-sheets would become good general-purpose resources for that action?)

I think there was another way my suggestion is incomplete, which has something to do with the way your definition of altruism doesn't work without a definition of "correct" subjective probability. But I don't remember what your definition of altruism was or why it didn't work without subjective probability.

I still think the right way to answer the question, "What is the correct subjective probability?" might be partly to derive "Bayesian updating" as an approximation that can be used by computationally limited agents implementing an updateless or other decision theory, with a utility function defined over mathematical descriptions of worlds containing some number of copies of the agent, when the differences in utility that result from the agent's decisions fulfill certain independence and linearity assumptions. I need to mathematically formalize those assumptions. "Subjective probability" would then be a variable used in that approximation, which would be meaningless or undefined when the assumptions failed.

I'm terribly sorry, but I had to delete that comment because it used the name of M-nt-f-x, which, if spoken in plain text, causes that one to appear. He Googles his name.

Here was the original comment by outlawpoet:

Marc Geddes was a participant in the SL4 list back when it was a bit more active, and kind of proverbial for degenerating into posting very incoherent abstract theory. He's basically carrying on the torch for M-nt-f-x and Achimedes Plutonium and other such celebrities of incommunicable genius.

Now please stop replying to this thread!

The problem is that copying and merging is not as harmless as it seems. You are basically doing invasive surgery on the mind, but because it's performed using intuitively "non-invasive" operations, it looks harmless. If, for example, you replaced the procedure with rewriting "subjective probability" by directly modifying the brain, the fact that you'd have different "subjective probability" as a result won't be surprising.

Thus, on one hand, there is an intuition that the described procedure doesn't damage the brain, and on the other the intuition about what subjective probability should look like in an undamaged brain, no matter in what form this outcome is delivered (that is, probability is always the same, you can just learn about it in different ways, and this experiment is one of them). The problem is that the experiment is not an instance of normal experience to which one can generalize the rule that subjective probability works fine, but an instance of arbitrary modification of the brain, from which you can expect anything.

Assuming that the experiment with copying/merging doesn't damage the brain, the resulting subjective probability must be correct, and so we get a perception of modifying the correct subjective probability arbitrarily.

Thought experiments with doing strange things to decision-theoretic agents are only valid if the agents have an idea about what kind of situation they are in, and so can try to find a good way out. Anything less, and it's just phenomenology: throw a rat in magma and see how it burns. Human intuitions about subjective expectation are optimized for agents who don't get copied or merged.

I responded here http://meteuphoric.wordpress.com/2011/05/19/on-the-anthropic-trilemma/

The problem with anthropic reasoning and evidence is that unlike ordinary reasoning and evidence, it can't be transferred between observers. Even if "anthropic psychic powers" actually do work, you still should expect all other observers to report that they don't.

I bite the third bullet. I am not as gifted with words as you are to describe why biting it is just and good and even natural if you look at it from a certain point of view, but...

You are believing in something mystical. You are believing in personal identity as something meaningful in reality, without giving any reason why it ought to be, because that is how your algorithm feels from the inside. There is nothing special about your brain as compared to your brain+your spinal cord, or as compared to your brain+this pencil I am holding. How could there be? Reality doesn't know what a brain is. Brains are not baked into the fundaments of reality, they are emergent from within. How could there be anything meaningful about them?

Consider this thought experiment. We take "you", and for a brief timespan, say, epsilon seconds, we replace "you" with "Brittany Spears". Then, after the epsilon seconds have passed, we swap "you" back in. Does this have a greater than order epsilon effect on anything? If so, what accounts for this discontinuity?

Let's explore this scenario in computational rather than quantum language.

Suppose a computer with infinite working memory, running a virtual world with a billion inhabitants, each of whom has a private computational workspace consisting of an infinite subset of total memory.

The computer is going to run an unusual sort of 'lottery' in which a billion copies of the virtual world are created, and in each one, a different inhabitant gets to be the lottery winner. So already the total population after the lottery is not a billion, it's a billion billion, spread across a billion worlds.

Virtual Yu'el perceives that he could utilize his workspace as described by Eliezer: pause himself, then have a single copy restored from backup if he didn't win the lottery, but have a trillion copies made if he did. So first he wonders whether it's correct to see this as making his victory in the lottery all but certain. Then he notices that if after winning he then does a merge, the certain victory turns back into certain loss, and becomes really worried about the fundamental soundness of his decision procedures and understanding of probability, etc.

Stating the scenario in these concrete terms brings out, for me, aspects that aren't so obvious in the original statement. For example: If everyone else has the same option (the trillionfold copying), Yu'el is no longer favored. Is the trilemma partly due to supposing that only one lottery participant has this radical existential option? Also, it seems important to keep the other worlds where Yu'el loses in sight. By focusing on that one special world, where we go from a billion people, to a trillion people, mostly Yu'els, and then back to a billion, we are not even thinking about the full population elsewhere.

I think a lot of the assumptions going into this thought experiment as originally proposed are simply wrong. But there might be a watered-down version involving copies of decision-making programs on a single big computer, etc, to which I could not object. The question for me is how much of the impression of paradox will remain after the problem has been diluted in this fashion.

....you have to throw away the idea that your joint subjective probabilities are the product of your conditional subjective probabilities....If you win the lottery, the subjective probability of having still won the lottery, ten seconds later, is ~1.

If copying increases your measure, merging decreases it. When you notice yourself winning the lottery, you are almost certainly going to cease to exist after ten seconds.

I don't think you have the third horn quite right. It's not that you're equally likely to wake up as Brittany Spears. It's that the only meaningful idea of "you" exists only right now. Your subjective anticipation of winning the lottery in five minutes should be zero. You clearly aren't winning the lottery or in five minutes.

Also, isn't that more of a quintlemma?

Maybe I am too late to comment here and it is already covered in collapsed comments, but it looks like that it is possible to make this experiment in real life.

Imagine that instead of copying, I will use waking up. If I win, I will be waked up 3 times and informed that I won and will be given a drug which will make me forget the act of awakening. If I lose, I will be wakened only one time and informed that I lost. Now I have 3 to 1 observer moments where I informed about winning.

In such setup in is exactly the Sleeping beauty problem, with all its pro and contra, which I will not try to explore here.

Tafter thinking about the Anthropic Trilemma for awhile, I've come up with an alternative resolution which I think is better than, or at least simpler than, any of the other resolutions. Rather than try to construct a path for consciousness to follow inductively forwards through time, start at the end and go backwards: from the set of times at which an entity I consider to be a copy of me dies, choose one at random weighted by quantum measure, then choose uniformly at random from all paths ending there.

The trick is that this means that while copying your mind increases the probability of ending up in the universe where you're copied, merging back together cancels it out perfectly. This means that you can send information back in time and influence your path through the universe by self-copying, but only log(n) bits of information or influence for n copies that you can't get rid of without probably dying.

I deny that increasing the number of physical copies increases the weight of an experience. If I create N copies of myself, there is still just one of me, plus N other agents running my decision-making algorithms. If I then merge all N copies back into myself, the resulting composite contains the utility of each copy weighted by 1/(N+1).

My feeling about the Boltzmann Brain is: I cheerfully admit that there is some chance that my experience has been produced by a random experience generator. However, in those cases, nothing I do matters anyway. Thus I don't give them any weight in my decision-making algorithm.

This solution still works correctly if the N copies of me have slightly different experiences and then forget them.

Banned for attempted top-posting of Geddes-standard gibberish.

I don't know the answer either. My best guess is that the question turns out to involve comparing incommensurable things, but I haven't pinned down which things. (As I remarked previously, I think the best answer for policy purposes is to just optimize for total utility, but that doesn't answer the question about subjective experience.)

But one line of attack that occurs to me is the mysterious nature of the Born probabilities.

Suppose they are not fundamental, suppose the ultimate layer of physics -- maybe superstrings, maybe something else -- generates the various outcomes in its own way, like a computer (digital or analog) forking processes...

and we subjectively experience outcomes according to the Born probabilities because this is the correct answer to the question about subjective experience probability.

Is there a way to test that conjecture? Is there a way to figure out what the consequences would be if it were true, or if it were false?

The third option seems awfully close to the second. In the second, you anticipate winning the lottery for a few seconds, and then going back to not. In the third, the universe anticipates winning the lottery for a few seconds, and then going back to Britney.

I'm just wondering what would Britney Spears say when she reads this.

Now you could just bite this bullet. You could say, "Sounds to me like it should work fine." You could say, "There's no reason why you shouldn't be able to exert anthropic psychic powers." You could say, "I have no problem with the idea that no one else could see you exerting your anthropic psychic powers, and I have no problem with the idea that different people can send different portions of their subjective futures into different realities."

I think there are other problems that may prevent the "anthropic psychic powers" example from working (maybe copying doesn't duplicate measure, but splits it gradually as the copies become increasingly separated in information content or in location; I think my comment here [LW(p) · GW(p)] might provide a way to think about that), but "the idea that different people can send different portions of their subjective futures into different realities" is not one of the problems, as I believe it's implied to be possible by the "two Schrodinger's cats" thought experiment (https://phys.org/news/2019-11-quantum-physics-reality-doesnt.html, https://arxiv.org/abs/1604.07422, https://web.archive.org/web/20200215011940/https://curiosity.com/topics/adding-a-second-cat-to-schrodingers-cat-experiment-might-break-quantum-physics-curiosity/, and the Frauchiger-Renner thought experiment mentioned in https://www.quantamagazine.org/frauchiger-renner-paradox-clarifies-where-our-views-of-reality-go-wrong-20181203/). (I'm not completely confident in my understanding of this, so please let me know if I'm understanding that thought experiment incorrectly. My understanding of the experiment is that the different participants should rightly expect different things to happen, and I think the easiest explanation is that the participants have their measure going in different proportions to different outcomes.)

The odds of winning the lottery are ordinarily a billion to one. But now the branch in >which you win has your "measure", your "amount of experience", temporarily >multiplied by a trillion. So with the brief expenditure of a little extra computing power, >you can subjectively win the lottery - be reasonably sure that when next you open >your eyes, you will see a computer screen flashing "You won!"

As I see it, the odds of being any one of those trillion "me"s in 5 seconds is 10^21 to one(one trillion times one billion). since there are a trillion ways for me to be one of those, the total probability of experiencing winning is still a billion to one. To be more formal:

P("experiencing winning")=sum(P("winning"|"being me #n")P("being me #n")) =sum(P("winning" and "being me #n"))=10^12*10^-21=10^-9 since "being me #n" partitions the space.

Overall this means I:

• anticipate not winning at 5 sec.

• anticipate not winning at 15 sec.

• don't have super-psychic-anthropic powers

• don't see why anyone has an issue with this

Checking consistency just in case:

p("experience win after 15s") = p("experience win after 15s"|"experience win after >5s")p("experience win after 5s") + p("experience win after 15s"|"experience not-win >after 5s")p("experience not-win after 5s").

p("experience win after 15s") = (~1)*(10^-9) + (~0)(1-10^-9)=~10^-9=~p("experience win after 5s")

Additionally, I should note that the total amount of "people who are me who experience winning" will be 1 trillion at 5 sec. and exactly 1 at 15 sec. This is because those trillion "me"s must all have identical experiences for merging to work, meaning the merged copy only has one set of consistent memories of having won the lottery. I don't see this as a problem, honestly.

I'm coming in a bit late, and not reading the rest of the posts, but I felt I had to comment on the third horn of the trilemma, as it's an option I've been giving a lot of thought.

I managed to independently invent it (with roughly the same reasoning) back in high school, though I haven't managed to convince myself of it or, for that matter, to explain it to anyone else. Your explanation is better, and I'll be borrowing it.

At any rate. One of your objections seems to be "...to assert that you can hurl yourself off a cliff without fear, because whoever hits the ground will be another person not particularly connected to you by any such ridiculous thing as a "thread of subjective experience".

For that to make sense would require that, while you can anticipate subjective experiences from just about anywhere, you would only anticipate experiencing a limited subset of them; 1/N of the total, where N represents.. what? The total number of humans, and why? Of souls?

Things get simpler if you set N to 1. Then your anticipation would be to experience Eliezer+5, Britney+5 and Cliffdiver+5, as well as every other subjective experience available for experiencing; sidestepping the cliffdiver problem, and more importantly removing any need to explain the value of N.

There's still the alternate option of it being infinity. I feel relatively certain that this is not the case, but I'm not sure this isn't simply wishful thinking. Help?

I figured it out! Roger Penrose is right about the nature of the brain!

just kidding.

In quantum copying and merging, every "branch" operation preserves the total measure of the original branch,

Maybe branch quantum operations don't make new copies, but represent already existing but identical copies "becoming" no longer identical?

In the computer program analogy: instead of having one program at time t and n slightly different versions at time t+1, start out with n copies already existing (but identical) at time t, and have each one change in the branching. If you expect a t+2, you need to start with at least n^2 copies.

(That may mean a lot more copies of everything than would otherwise be expected even under many worlds, but even if it's enough to give this diabolical monster bed-wetting nightmares, by the Occam's razor that works for predicting physical laws, that's absolutely fine).

Come to think of it... if this interpretation isn't true...

or for that matter, even if this is true but it isn't true that someone who runs redundantly on three processors gets three times as much weight as someone who runs on one processor...

then wouldn't we be vastly likely to be experiencing the last instant of experienceable existence in the Universe, because that's where the vast majority of distinct observers would be? Omega-point simulation hypothesis anyone? :-)

The "second horn" seems to be phrased incorrectly. It says:

"you can coherently anticipate winning the lottery after five seconds, anticipate the experience of having lost the lottery after fifteen seconds, and anticipate that once you experience winning the lottery you will experience having still won it ten seconds later."

That's not really right - the fate of most of those agents that experience a win of the lottery is to be snuffed out of existence. They don't actually win the lottery - and they don't experience having won it eleven seconds later either. The chances of the lottery staying won after it has been experienced as being won are slender.

Either that "horn" needs rephrasing - or another "horn" needs to be created with the correct answer on it.

I strive for altruism, but I'm not sure I can believe that subjective selfishness - caring about your own future experiences - is an incoherent utility function; that we are forced to be Buddhists who dare not cheat a neighbor, not because we are kind, but because we anticipate experiencing their consequences just as much as we anticipate experiencing our own. I don't think that, if I were really selfish, I could jump off a cliff knowing smugly that a different person would experience the consequence of hitting the ground.

It strikes me that this is a problem of comparing things at different levels of meta. You are talking about your "motivations" as if they were things that you could freely choose based on your a priori determinations about how one well and truly ought to act. You sound, forgive me for saying this, I do respect you deeply, almost Kantian here.

The underlying basis for your ethical system, or even further, your motivational system, does not lie in this abstract observer that is Eliezer today and Brittany Spears tomorrow. I think I'm ripping this off of some videogame I've played, but think of the brain as a container, and this "observerstuff" (dare I call it a soul?) as water poured into the container. Different containers have different shapes and shape the water differently, even though it is the same water. But the key point is, motivations are a property of the container, not of the water, and their referents are containers, not water. Eliezer-shaped water cares about what happens to the Eliezer-container, not what happens to the water in it. That's just not how the container is built.

I will bite the first horn of the trilemma. I'm will argue that the increase in subjective probability results from losing information and that it is no different from other situations where you lose information in such a way to make subjective probabilities seem higher. For example, if you watch the lotto draw, but then forget every number except those that match your ticket, your subjective probability that you won will be much higher than originally.

Let's imagine that if you win the lottery that a billion copies of you will be created.

t=0: The lottery is drawn t=1: If you won the lottery, then a billion clones are created. The original remembers that they are the original as they see the clones being created, but if clones were made, they don't know they are clones and don't know that the original knows that they are the original, so they can't figure it out that way. t=2: You have a bad memory and so you forget whether you are an original or a clone. t=3: If any clones exist, they are all killed off t=4: Everyone is informed about whether or not they ran the lottery.

Let's suppose that you know you are the original and that you are at t=1. Your chances of winning the lottery are still 1 in a million as the creation of clones does not affect your probability of waking up to a win at t=4 if you know that you are not a clone.

Now let's consider the probability at t=2. Your subjective odds of winning the lottery have rise massively, since you most probably are a copy. Even though there is only a one in a million chance that copies will be made, the fact that a billion copies will be made more than cancels this out.

What we have identified is that it is the information loss that is the key feature. Of course you can increase your subjective probabilities by erasing any information that is contrary. What is interesting about cloning is that if we are able to create clones with the exact same information, we are able to effectively remove knowledge without touching your brain. That is, if you know that you are not a clone, after we have cloned you exactly, then you no longer know you are not a clone, unless someone tells you or you see it happen.

Now at t=3 we kill off/merge any remaining clones. If you are still alive, you've gained information when you learned that you weren't killed off. In fact, you've been retaught the same information you've forgotten.

1) the probability of ending up in the set of winners is

1/billion

2) the probability of being (a specific) one of the trillion is

1/(b * t)

the probability of being a 2) given you are awake is

p(2 | awake) = P(awake | 2) * p(2)

                    p(awake)


            = 1 * 1E-21
              ---------
                   1



            = very small

Buy a ticket. Suspend your computer program just before the lottery drawing - which should of course be a quantum lottery, so that every ticket wins somewhere. Program your computational environment to, if you win, make a trillion copies of yourself, and wake them up for ten seconds, long enough to experience winning the lottery. Then suspend the programs, merge them again, and start the result. If you don't win the lottery, then just wake up automatically.

How would you do that???

The odds of winning the lottery are ordinarily a billion to one. But now the branch in which you win has your "measure", your "amount of experience", temporarily multiplied by a trillion.

This seems perhaps too obvious, but how can branches multiply probability by anything greater than 1? Conditional branches follow the rules of conjunctive probability . . .

Probability in regards to the future is simply a matter of counting branches. The subset of branches in which you win is always only one in a billion of all branches - and any further events in a branch only create further sub-branches, so the probability of anything happening in that sub-branch can never be greater than 10^-9. The exact number of copies in this context is irrelevant - it could be infinite and it wouldn't matter.

Whether we accept identification with only one copy of ourself as in jschculter's result or we consider our 'self' to be all copies, the results still work out to 1 billion to 1 against winning.

Another way of looking at the matter: we should be wary of any non-objective decision process. If we substitute 'you' for 'person X' in the example, we wouldn't worry that person X splitting themselves into a trillion sub-copies only if they win the lottery would somehow increase their actual likelihood of winning.

The flaw is that anticipation should not be treated as a brute thing. Anticipation should be a tool used in the service of your decision theory. Once you bring in some particular decision theory and utility function, the question is dissolved (if you use TDT and your utility function is just the total quality of simulated observer moments, then you can reverse engineer exactly Nick Bostrom's notion of "anticipate." So if I had to go with an answer, that would be mine.)

Two people disagreeing about what they should anticipate is like two people arguing about whether a tree falling in an empty forest makes a sound. They disagree about what they anticipate, yes, but they behave identically.

I'm curious why no one mentioned Solomonoff prior here. Anticipation of subjective experience can be expressed as: what is a probability of experiencing X, given my prior experiences. Thus we "swap" ontological status of objective reality and subjective experience, then we can use Solomonoff prior to infer probabilities.

When one wakes up as a copy, one experiences instantaneous arbitrary space-time travel, thus Solomonoff prior for this experience should be lower, than that of wake-up-as-original one (if original one can wake up at all).

Given that approach, it seems that our subjective experience will tend to be as much "normal" as it allowed by simplest computable laws of physics.

a truly remarkable observation: quantum measure seems to behave in a way that would avoid this trilemma completely

Which is why Roger Penrose is so keen to show that consciousness is a quantum phenomenon.

We have a strong subjective sense of personal experience which is optimized for passing on genes, and which thus coincides with the Born probabilities. In addition, it seems biased toward "only one of me" thinking (evidence: most people's intuitive rejection of MWI as absurd even before hearing any of the physics, and most people's intuitive sense that if duplicated, 'they' will be the original and 'someone else' will be the copy). The plausible ev-psych explanation for this, ISTM, is that you won't ever encounter another version of your actual self, and that it's very bad to be tricked into really loving your neighbor as yourself. Thus the rigid sense of continuity of personal identity.

Thus, when complications like quantum suicide or splitting or merging of minds are introduced, the basic intuitions become extremely muddled. In particular, Nick Bostrom's solution prompts the objection of absurdity, even though it is made up of ingredients that seem reasonable (to rationalist materialists, anyhow) taken separately. That makes me suspicious that Bostrom might in fact be right, and that our objections stem more from the ev-psych than anything else.

The following thought experiment pumps my intuition: what concept of subjective probability might Ebborian-like creatures evolve? Of course, they'd split more frequently when resources were plentiful. Imagine that a quantum random event X would double an Ebborian's resources if it happened, and that X happened in half the branches of the wavefunction. If it's assumed that the Ebborian would split if and only if X happened, what subjective probability would ve evolve to assign to X? Well, since what really counts for the evolutionary process is the total 'population of descendants' averaged across all branches, ve should in fact weight more heavily the futures in which ve splits: ve should evolve to assign X probability 2/3, even though the splitting happens after observing X. And there's really no inconsistency with that: the single copy in half the branches feels a bit unlucky while the two copies in the other branches rejoice that the odds were in their favor. Repeat this a great many times, and most of the descendants will feel pretty well calibrated in their probabilities.

So I think that our sense of subjective probability has to be an evolved aid to decision-making, rather than an inherent aspect of conscious experience; and I have to go with Nick Bostrom's probabilities, as strange as they sound. (Nods to Tyrrell and Wei Dai, whose comments greatly helped my thought process.)

ETA: I just realized an ingredient of "what it would feel like": Ebborians would evolve to give the same probabilities we would for events that don't affect their splitting times, but all events that would make them richer or poorer would have subjective probability skewed in this fashion. Basically, Ebborians evolve so that each one just feels that being a consistently lucky frood is the natural state of things, and without that necessarily giving them the ego it would give us.

The third horn is a fake, being capable of being defined in or out of existence at will. It I am indifferent to my state ten seconds from now, it is true; if my current utility function includes a term for my state ten seconds from now, it is false.

The 'thread of subjective experience' is not the issue; whether I throw myself of the cliff, will depend on whether I am currently indifferent as to whether my future self will die.

Why is bullet three a fake reduction? I've bit it before and even see bullet three as something to aspire to.

"I still have trouble biting that bullet for some reason.... I don't think that, if I were really selfish, I could jump off a cliff knowing smugly that a different person would experience the consequence of hitting the ground."

Replace "jump off a cliff" with "heroin overdose." Sure you could, and many do. Not caring about the future is actually very common in humans, and it's harder for smart people to understand this attitude because many of us have very good impulse control. But I still find it strange that you seem to want continuity of self to be more than an illusion. Many more paradoxes occur when you insist on continuity of self than when you don't; Occam's Razor would tell you to throw away the concept.

Is there a contradiction in supposing that the total subjective weight increases as unconnected threads of subjective experience come into existence, but copies branching off of an existing thread of subjective experience divide up the weight of the parent thread?

For what it's worth, here is the latest attempt by philosophers of physics to prove the Born rule from decision theory.

Sentences in this comment asserted with only 95% probability of making sense, read on at own peril.

There's a mainstream program to derive the Born probabilities from physics and decision theory which David Wallace, especially, has done a lot of work on. If I remember correctly, he distinguishes two viewpoints:

• "Subjective Uncertainty", which says you're a stage of some 4D space-time worm and you're indexically uncertain which worm, because many of them have stages that are exactly alike
• "Objective Determinism", which says you'll continue as all your future continuations and you should just maximize utility over them and see if doing so involves something that, even though it doesn't express uncertainty, behaves like a probability

Wallace's opinion is that both SU and OD are correct ways to think. (Given his assumptions, both allow a proof of the Born probabilities, but it's easier with SU.) That strikes me as being parallel to Eliezer's claims here. If you think SU is a wrong way to think and OD is a correct way to think, that strikes me as being parallel to everyone else's claims here.

There's an argument that SU and OD have different implications, e.g., OD allows you to care about inter-branch diversity and SU doesn't. If something like OD gives you a way to reductionize "anticipation", but you still want a different kind of "anticipation" that accords more with your intuitions, then you may be in trouble if their roles ever overlap. It seems to me one has more to do with decision-making and the other has more to do with conscious experience; those are completely different things and it's important to keep them separate.

Anyway, since we can be reductionist about these 4D worms and we're already being reductionist about decision theory, it shouldn't be hard to figure out exactly how they relate.

In quantum copying and merging, every "branch" operation preserves the total measure of the original branch,

Maybe branch quantum operations don't make new copies, but represent already existing but identical copies "becoming" no longer identical?

In the computer program analogy: instead of having one program at time t and n slightly different versions at time t+1, start out with n copies already existing (but identical) at time t, and have each one change in the branching. If you expect a t+2, you need to start with at least n^2 copies.

(That may mean a lot more copies of everything than would otherwise be expected even under many worlds, but even if it's enough to give this diabolical monster bed-wetting nightmares, by the Occam's razor that works for predicting physical laws, that's absolutely fine).

Come to think of it... if this interpretation isn't true, or for that matter, even if this is true but it isn't true that identical copies get equal experience measure, then wouldn't we be vastly likely to be experiencing the last instant of experienceable existence in the Universe, because that's where the vast majority of distinct observers would be? Omega-point simulation hypothesis anyone? :-)

In quantum copying and merging, every "branch" operation preserves the total measure of the original branch,

Maybe branch quantum operations don't make new copies, but represent already existing but identical copies "becoming" no longer identical?

In the computer program analogy: instead of having one program at time t and n slightly different versions at time t+1, start out with n copies already existing (but identical) at time t, and have each one change in the branching. If you expect a t+2, you need to start with at least n^2 copies...

...and that may mean a lot more copies of everything than would otherwise be expected even under many worlds, but even if it's enough to give this diabolical monster bed-wetting nightmares, by the Occam's razor that works for predicting physical laws, that's absolutely fine.

Just an aside - this is obviously something that Eliezer - someone highly intelligent and thoughful - has thought deeply about, and has had difficulty answering.

Yet most of the answers - including my own - seem to be of the "this is the obvious solution to the dilemma" sort.

Since I have a theory of Correlated decision making, let's use it! :-)

Let's look longer at the Nick Bostrom solution. How much contribution is there towards "feeling I will have won the lottery ten seconds from now" from "feeling I have currently won the lottery"? By the rules of this set-up, each of the happy copies contributes one trillionth towards that result.

(quick and dirty argument to convince you of that: replace the current rules by one saying "we will take the average feeling of victory across the trillion copies"; since all the feelings are exactly correlated, this rule gives the same ultimate result, while making it clear that each copy contributes one trillionth of the final result).

Thus Nick's position is, I believe, correct.

As for dealing with the fourth horn, I've already written on how to do that: here you have partially correlated experiences contributing to future feelings of victory, which you should split into correlated and anti-correlated parts. Since the divergence is low, the anti-correlated parts are of low probability, and the solution is approxiamtely the same as before.

I've thought about this before and I think I'd have to take the second horn. Argument: assuming we can ignore quantum effects for the moment, imagine setting up a computer running one instance of some mind. There're no other instances anywhere. Shut the machine down. Assuming no dust theory style immortality (which, if there was such a thing, would seem to violate born stats, and given that we actually observe the validity of born stats...), total weight/measure/reality-fluid assigned to that mind goes from 1 to 0, so it looks reasonable that second horn type stuff is allowed to happen.

I'd say personal continuity is real, but is made up of stuff like memory, causality maybe, etc. I suspect those things explain it rather than explain it away.

However, given that in this instance it seems QM actually makes things behave in a saner way, there's one other option I think we ought consider, though I'm hesitant to bring it up:

Horn 5b: consciousness may be inherently quantum. This is not, on its own, an explanation of consciousness, but maybe we ought consider the possibility that the only types of physical processes that are "allowed" to be conscious are in some way tied to inherently quantum ones, and the only type of mind branching that's allowed is via quantum branching.

Given that, as you point out, it seems that the only form of branching that we experience (ie, quantum branching) is the one way that actually seems to (for some reason) automatically make it work out in a way that doesn't produce confusing weirdness, well...

(EDIT: main reason I'm bringing up this possibility is that it's an option that would actually help recover "it all adds up to normality")

(EDIT2: Ugh, I'm stupid: "normality" except for more or less allowing stuff that's pretty close to being p-zombies... so this doesn't actually improve the situation all that much as far as "normality" after all.)

Other than that, maybe when we explicitly solve the Born stats fully satisfactorily, when we see how nature is pulling the trick off, then we'll hopefully automatically see the consequences of this situation.

I have an Othello/Reversi playing program.

I tried making it better by applying probabilistic statistics to the game tree, quite like antropic reasoning. It then became quite bad at playing.

Ordinary minimax with A-B did very well.

Game algorithms that ignore density of states in the game tree, and only focus on minimaxing, do much better. This is a close analogy to the experience trees of Eliezer, and therefore a hint that antropic reasoning here has some kind of error.

Kim0

Re: "But if I make two copies of the same computer program, is there twice as much experience, or only the same experience? Does someone who runs redundantly on three processors, get three times as much weight as someone who runs on one processor?"

Do they get three times as much "weight" - in some moral system?

Er, that depends on the moral system in question.