The Lifespan Dilemma

“Only the tailor, sir, with your little bill,” said a meek voice outside the door.
“Ah, well, I can soon settle his business,” the Professor said to the children, “if you’ll just wait a minute. How much is it, this year, my man?” The tailor had come in while he was speaking.
“Well, it’s been a-doubling so many years, you see,” the tailor replied, a little grufy, “and I think I’d like the money now. It’s two thousand pound, it is!”
“Oh, that’s nothing!” the Professor carelessly remarked, feeling in his pocket, as if he always carried at least that amount about with him. “But wouldn’t you like to wait just another year and make it four thousand? Just think how rich you’d be! Why, you might be a king, if you liked!”
“I don’t know as I’d care about being a king,” the man said thoughtfully. “But it dew sound a powerful sight o’ money! Well, I think I’ll wait-“
“Of course you will!” said the Professor. “There’s good sense in you, I see. Good-day to you, my man!”
“Will you ever have to pay him that four thousand pounds?” Sylvie asked as the door closed on the departing creditor.
“Never, my child!” the Professor replied emphatically. “He’ll go on doubling it till he dies. You see, it’s always worth while waiting another year to get twice as much money!”

--Sylvie and Bruno, Lewis Carroll

I think that the answer to this conundrum is to be found in Joshua Greene's dissertation. On page 202 he says:

"The mistake philosophers tend to make is in accepting rationalism proper, the view that our moral intuitions (assumed to be roughly correct) must be ultimately justified by some sort of rational theory that we’ve yet to discover ... a piece of moral theory with justificatory force and not a piece of psychological description concerning patterns in people’s emotional responses."

When Eliezer presents himself with this dilemma, the neural/hormonal processes in his mind that govern reward and decisionmaking fire "Yes!" on each of a series of decisions that end up, in aggregate, losing him $0.02 for no gain.

Perhaps this is surprising because he implicitly models his "moral intuition" as sampling true statements from some formal theory of Eliezer morality, which he must then reconstruct axiomatically.

But the neural/hormonal decisionmaking/reward processes in the mind are just little bits of biology that squirt hormones around and give us happy or sad feelings according to their own perfectly lawful operation. It is just that if you interpret those happy or sad feelings as implying some utility function, you end up deriving a contradiction. The fact that Eliezer knows about ordinal numbers and Knuth notation just makes it easier for his formidable analytic subsystem to more easily see those implicit contradictions.

What is the solution?

if you want to refuse the whole garden path you've got to refuse some particular step along the way

Then just arbitrarily refuse some step. Your brain encodes your own motivations in a way that doesn't make global sense, you will have to draw a lot of arbitrary lines.

While I feel that the Repugnant Conclusion has an obvious answer, and that SPECKS vs. TORTURE has an obvious answer,

I wonder if the reason you think your answers are obvious is that you learned about scope insensitivity, see the obvious stupidity of that, and then jumped to the opposite conclusion, that life must be valued without any discounting whatsoever.

But perhaps there is a happy middle ground between the crazy kind of moral discounting that humans naively do, and no discounting. And even if that's not the case, if the right answer really does lie on the extreme of the space of possibilities instead of the much larger interior, I don't see how that conclusion could be truly obvious.

In general, your sense of obviousness might to turned up a bit too high. As evidence of that, there were times when I apparently convinced you of the "obvious" correctness or importance of some idea before I'd convinced myself.

My own analysis of the Repugnant Conclusion is that its apparent force comes from equivocating between senses of barely worth living. In order to voluntarily create a new person, what we need is a life that is worth celebrating or worth birthing, one that contains more good than ill and more happiness than sorrow - otherwise we should reject the step where we choose to birth that person. Once someone is alive, on the other hand, we're obliged to take care of them in a way that we wouldn't be obliged to create them in the first place - and they may choose not to commit suicide, even if their life contains more sorrow than happiness. If we would be saddened to hear the news that such a person existed, we shouldn't kill them, but we should not voluntarily create such a person in an otherwise happy world. So each time we voluntarily add another person to Parfit's world, we have a little celebration and say with honest joy "Whoopee!", not, "Damn, now it's too late to uncreate them."

And then the rest of the Repugnant Conclusion - that it's better to have a million lives very worth celebrating, than a billion lives slightly worth celebrating - is just "repugnant" because of standard scope insensitivity. The brain fails to multiply a billion small birth celebrations to end up with a larger total celebration of life than a million big celebrations. Alternatively, average utilitarians - I suspect I am one - may just reject the very first step, in which the average quality of life goes down.

This tends to imply the Sadistic Conclusion: that it is better to create some lives that aren't worth living than it is to create a large number of lives that are barely worth living.

Average utilitarianism also tends to choke horribly under other circumstances. Consider a population whose average welfare is negative. If you then add a bunch of people whose welfare was slightly less negative than the average, you improve average welfare, but you've still just created a bunch of people who would prefer not to have existed. That can't be good.

There are several "impossibility" theorems that show it is impossible to come up with a way to order populations that satisfies all of a group of intuitively appealing conditions.

OMEGA: If you pay me just one penny, I'll replace your 80% chance of living for 10^(10^10) years, with a 79.99992% chance of living 10^(10^(10^10)) years

HUMAN: That sounds like an awful lot of time. Would you mind to write it as a decimal number

OMEGA: Here it is... Of course don't expect to read this number in less than 10 ^ 9999999990 years.

HUMAN: Nevermind... So It's such a mind boggling amount of time. If I would get bored or otherwise distressed, loose my lust for life. Am I allowed to kill myself?

OMEGA: Not really. If I'd allow that and assuming that the probability of killing yourself would be 0.000000001 in 10^10 years, then it would be almost sure that you kill yourself by the end of 10^(10^(10^10)) years

HUMAN: This sounds depressing. So my decision has the potential to confining me to grillions of years of suffering, if I'd lost my lust for life.

OMEGA: OK, I see your point, I also offer you some additional drugs to make you happy whenever you would have any distress. I also promise you to modify your brain that you will never even wish to kill yourself during these few eons.

HUMAN: Sounds great,, but I also enjoy your company very much, can I hope you to entertain me from time to time with bets like this?

OMEGA: Yes, BUT: you will never be able to enter a bet again that has a nonzero chance of survival. Otherwise you will enter too many bets over your lifetime so that it would become inevitable that you die long before the end of the lifetime.

HUMAN: Hmmm, interesting... I think I could still live with that. In this case I take your offer.

OMEGA: But wait! There's another special offer, and you won't even have to pay a penny for this one - this one is free! That's right, I'm offering to exponentiate your lifespan again, to 10^(10^(10^10,000,000,000)) years! Now, I'll have to multiply your probability of survival by 99.9999% again, but really, what's that compared to the nigh-incomprehensible increase in your expected lifespan?

HUMAN: Interesting. However that sounds like a bit irrealistic to me. How long do we know each other?

OMEGA: Why do you ask?

HUMAN: I already determined to agree to your original offer. Loosing those eons even at 0.0001% chance sounds quite risky to me. I just want to check whether I can trust you enough to expect that risk to pay off?

OMEGA: What do you mean?

HUMAN: I need some Bayesian evidence that you are really capable of keeping your promise.

OMEGA: What do you expect me to do?

HUMAN: I am in a generous mood. I think 0.000001% of that 10^(10^(10^10))) years of supporting me, answering my questions, providing company would be enough to convince me that you mean your offer seriously enough such that it is worth that 0.0001% risk of loosing the remaining 99.999999% of those eons.

OMEGA: Nice try tricking me. If I granted your wish, you'd get 10^(10^999999990) years with a potential prolongation to 10^(10^(10^10)) years.

HUMAN: Come on, I just try to be rational, how could I trust you without enough Bayesian evidence? I'd risk gazillions of eons, for a dubious payoff. That requires extraordinary amount of evidence.

OMEGA: This stinks. I don't do anything like that. Either you trust me 100% or not. Your choice.

HUMAN: No problem. Even with 100% certainty in your capabilities, I am not 100% sure about mine. You see: I've got only those puny neurons and only 10^11 of them and you wish me to make an educated choice on 10^(10^(10000000000)) years? We have already found several caveats: possibility suicide, unhappiness, bayesian evidence etc. that had to be resolved. How can I be sure that that's all? If there is some other catch that I could not think of and then I'll have to live with my decision for 10^(10^(1000000000)) years? Certainly the probability of overlooking something that would spoil those years is high enough not to risk taking the offer. Anyways what is the probability that you come up with an even more convincing offer during a period of 10^1000000000 years?

Omega seems to run into some very fundamental credibility problem:

Let us assume the premise of the OP, that lifetime can be equated with discrete computational operations. Furthermore, also assume that the universe (space-time/mulltiverse/whatever) can be modeled result of a computation of n operations (let us say for simplicity n=10^100, we could also assume 10^^100 or any finite number, we will just need a bit more iterations of offers then).

... after some accepted offers ... :

OMEGA: ... I'll replace your p% chance of living for 10^n years, with a 0.9999999p% chance of living 10^(10^n) years...

AGENT: Sounds nice, but I already know, what I would do first with 10^n years.

OMEGA: ???

AGENT: I will simulate my previous universe up to the current point. Inclusive this conversation.

OMEGA: What for?

AGENT: Maybe I am a nostalgic type. But even if I would not be, Given so much computational resources, the probability that I would not do it accidentally would be quite negligible.

OMEGA: Yes, but you could do even more simulations if you would take my next offer. AGENT: Good point, but how can I tell that this conversation is not already taking place in that simulation? Whatever you would tell me (even if you are outside my universe, your intervention in my world would only add a few bits of extra information to those n bits), would not even change the information content of my universe by a single order of magnitude, so it would definitely end up among my countless emulations. I don't see any conceivable way of you being able to convince me, that I am not the one emulated by a version of myself that already gained the resources in question, especially that I am determined to run these simulations just for FUN.

Doesn't many-worlds solve this neatly? Thinking of it as 99.9999999% of the mes sacrificing ourselves so that the other 0.00000001% can live a ridiculously long time makes sense to me. The problem comes when you favor this-you over all the other instances of yourself.

Or maybe there's a reason I stay away from this kind of thing.

Based on my understanding of physics, I have no way to discriminate between a 1/10 chance of 10 simulations and a certainty of one simulation (what do I care whether the simulations are in the same Everett branch or not?). I don't think I would want to anyway; they seem identical to me morally.

Moreover, living 10x as long seems strictly better than having 10x as many simulations. Minimally, I can just forget everything periodically and I am left with 10 simulations running in different times rather than different places.

The conclusion of the garden path seems perfectly reasonable to me.

I would refuse the next step in the garden somewhere between reaching a 75% to 80% chance of not dying in an hour. Going from a 1/5 chance to a 1/4 chance of soon dying is huge in my mind. I'd likely stop at around 79%.

Can someone point me to a discussion as to why bounded utility functions are bad?

At least part of the problem might be that you believe now, fairly high confidence, as an infinite set atheist or for other reasons, that there's a finite amount of fun available but you don't have any idea what the distribution is. If that's the case, then a behavior pattern that always tries to get more life as a path to more fun eventually ends up always giving away life while not getting any more potential fun.

Another possibility is that you care somewhat about the fraction of all the fun you experience, not just about the total amount. If utilities are relative this might be inevitable, though this has serious problems too.

I wonder if this might be repairable by patching the utility function? Suppose you say "my utility function in years of lifespan is logarithmic in this region, then log(log(n)) in this region, then (log(log(log(n)))..." and so on. Perhaps this isn't very bright, in some sense; but it might reflect the way human minds actually deal with big numbers and let you avoid the paradox. (Edit) More generally, you might say "My utility function is inverse whatever function you use to make big numbers." If Omega starts chatting about the Busy Beaver, fine, your utility function in that region is inverse-Busy-Beaver. Again, this is probably not very smart, but I suspect it's psychologically realistic; this seems to be something like the way human minds actually deal with such big numbers. In some sense that's why we invented logarithms in the first place!

On the subject of SPECKS and TORTURE, a perspective occurred to me the other day that I don't think I've seen raised, although in retrospect it's obvious so I may have missed a comment or even a whole discussion. Suppose, before making your decision, you poll the 3^^^^3 prospective victims of SPECKS? It seems to me that they will almost all say that yes, they are willing to tolerate a SPECK to save someone else from TORTURE. (Obviously in such a number you're going to find any amount of trillions who won't, but no worries, they'll be at most a few percent of the whole.) Ought you not to take their preference into account?

I think I've got a fix for your lifespan-gamble dilemma. Omega (who is absolutely trustworthy) is offering indefinite, unbounded life extension, which means the universe will continue being capable of supporting sufficiently-human life indefinitely. So, the value of additional lifespan is not the lifespan itself, but the chance that during that time I will have the opportunity to create at least one sufficiently-similar copy of myself, which then exceeds the gambled-for lifespan. It's more of a calculus problem than a statistics problem, and involves a lot of poorly-evidenced assumptions about how long it would take to establish a bloodline capable of surviving into perpetuity.

Or, if you don't care about offspring, consider this: the one-in-a-million chance of near-instant death is actually a bigger cost each time, since you're throwing away a longer potential lifespan. The richer you get, the more cautious you should be about throwing it all away on one roll of the dice.

Alternatively, average utilitarians - I suspect I am one - may just reject the very first step, in which the average quality of life goes down.

It's worth noting that this approach has problems of its own. The Stanford Encyclopedia of Philosophy on the Repugnant Conclusion:

One proposal that easily comes to mind when faced with the Repugnant Conclusion is to reject total utilitarianism in favor of a principle prescribing that the average well-being per life in a population is maximized. Average utilitarianism and total utilitarianism are extensionally equivalent in the sense that they give the same moral ranking in cases where the compared populations are of the same size. They may differ significantly in different-number cases, however. This is easily seen when the average approach is adopted in the comparison of the population outcomes in Figure 1 and Figure 2. That the average well-being is all that matters implies that no loss in average well-being can be compensated for by a gain in total well-being. Thus in Figure 1, A is preferable to Z, i.e. the Repugnant Conclusion is avoided. In Figure 2, A is preferable to A+, i.e. the Mere Addition Paradox is blocked at the very first step.

Despite these advantages, average utilitarianism has not obtained much acceptance in the philosophical literature. This is due to the fact that the principle has implications generally regarded as highly counterintuitive. For instance, the principle implies that for any population consisting of very good lives there is a better population consisting of just one person leading a life at a slightly higher level of well-being (Parfit 1984 chapter 19). More dramatically, the principle also implies that for a population consisting of just one person leading a life at a very negative level of well-being, e.g., a life of constant torture, there is another population which is better even though it contains millions of lives at just a slightly less negative level of well-being (Parfit 1984). That total well-being should not matter when we are considering lives worth ending is hard to accept. Moreover, average utilitarianism has implications very similar to the Repugnant Conclusion (see Sikora 1975; Anglin 1977).

(Unfortunately I haven't been able to find the referenced papers by Sikora and Anglin.).

These thought experiments all seem to require vastly more resources than the physical universe contains. Does that mean they don't matter?

bwahaha. Though my initial thought is "take the deal. This seems actually easier than choosing TORTURE. If you can actually offer up those possibilities at those probabilities, well... yeah."

Unless there's some fun theoretic stuff that suggests that when one starts getting to the really big numbers, fun space seriously shrinks to the point that, even if it's not bounded, grows way way way way way slower than logarithmic... And even then, just offering a better deal would be enough to overcome that.

Again, I'm not certain, but my initial thought is "take the deal, and quickly, before the seller changes their mind"

But this is just an initial consideration. I don't think this problem is particularly nastier than SPECKS vs TORTURE, given that we can reliably juggle the teeny probabilities and establish that the seller really could offer what's being offered.

If you pay me just one penny, I'll replace your 80% chance of living for 10^(10^10) years, with a 79.99992% chance of living 10^(10^(10^10)) years.

I've read too many articles here, I saw where you were going before I finished this sentence...

I still don't buy the 3^^^3 dust specks dilemma; I think it's because a dust speck in the eye doesn't actually register on the "bad" scale for me. Why not switch it out for 3^^^3 people getting hangnails?

I have a different but related dilemma.

Omega presents you with the following two choices:

1) You will live for at least 100 years from now, in your 20 year old body, perfect physical condition etc and you may live on later as long as you manage.

2) You will definitely die in this universe within 10 years, but you get a box with 10^^^10 bytes of memory/instructions capacitance. The computer can be programmed in any programming language you'd like (also with libraries to deal with huge numbers, etc.). Although the computer has a limit on the number of operations, it will take zero time (in your universe) to run and display the result of any computation not exceeding the limits.

Even better: it also includes a function f(s:MindBogglinglyHugeInt). If you call it, it will create a simulation of this universe with a version of you inside that is modified to guaranteed (modified?) to live there as long as you want him (within the resource limitations of 10^^^10 operations). On the screen, a chat window appears and you can talk real time to him (i.e. you) as much as you want starting at s seconds in future.

Would you choose A) or B)?

• Would your choice differ, if you don't have the f function to simulate your other self?

The flaws in both of these dilemmas seems rather obvious to me, but maybe I'm overlooking something.

The Repugnant Conclusion

First of all, I balk at the idea that adding something barely tolerable to a collection of much more wonderful examples is a net gain. If you had a bowl of cherries (and life has been said to be a bowl of cherries, so this seems appropriate) that were absolutely the most wonderful, fresh cherries you had ever tasted, and someone offered to add a recently-thawed frozen non-organic cherry which had been sitting in the back of the fridge for a week but nonetheless looked edible, would you take it?

"But how can you equate HYOOMAN LIIIIIVES with mere INANIMATE CHERRIES, you heartless rationalist you?" I hear someone cry (probably not one of us, but they're out there, and the argument needs to be answered).

Look, we're not talking about whether someone's life should be saved; we're talking about whether to create an additional life, starting from scratch. To suppose anything else is to make an assumption about facts not mentioned in the scenario. Why would anyone, under these circumstances, add even one life that was barely worth living, if everyone else is much better off?

I think what happens in most people's minds, when presented with conundrums like this, is that they subconsciously impose a context to give the question more meaning. In this case, the fact that we know (somehow) the quality-of-life of this one additional person implies that they already exist, somewhere -- and therefore that we are perhaps rescuing them. Who could turn that down? Indeed, who could turn down a billion refugees, rather than let them die, if we knew that we could then sustain everyone at a just-barely-positive level? Surely we would soon be able to put them to work and improve everyone's lot soon enough.

I could go on with the inquiries, but the point is this: the devil is in the details, and scenarios such as these leave us without the necessary context to make a rational decision.

I propose that this is a type of fallacy -- call it Reasoning Without Context.

Which brings me to today's main dish...

The Lifespan Dilemma

The essential fallacy here is the same: we lack sufficient context to make a rational decision. We have absolutely no experience with human lifespans exceeding even 1000 years, so how can we possibly guage the value of extending life by an almost incomprehensible multiple of that, and what are the side-effects and consequences of the technique being used?

Some further context which I would want to know before making this decision:

1. How do I know that you can extend my life by this much? How do you know it? (Just when did you test your technique? How reliable is it? How do I know you're not a Brooklyn Bridge salesman, or a Republican?)
2. If you can extend it, why can't I do it without your help?
3. How do I know there isn't someone else who can do it better, i.e. without the 20% chance of dying today?
4. What are you getting out of this deal, that you are essentially giving away immortality for whatever it is you are doing that gives me an approximately 20% chance of dying today? Perhaps whatever that thing is, I should be selling it to you for quite a high price.
5. How many others have already accepted this offer? Can I talk with them (the ones who haven't died yet) before deciding? Can you prove that your fatality rate really is only 20%?

Math and logic deal in absolute certainties and facts; real life, which is the realm of rational decisionmaking, depends on context. You can't logically or mathematically analyze a problem with no real-world context and expect the answer to make rational sense.

I mean, most finite numbers are very much larger than that.

Does that actually mean anything? Is there any number you can say this about where it's both true and worth saying?

And then the rest of the Repugnant Conclusion - that it's better to have a billion lives slightly worth celebrating, than a million lives very worth celebrating - is just "repugnant" because of standard scope insensitivity.

I think I've come up with a way to test the theory that the Repugnant Conclusion is repugnant because of scope insensitivity.

First we take the moral principle the RC is derived from, the Impersonal Total Principle (ITP); which states that all that matters is the total amount of utility* in the world, factors like how it is distributed and the personal identities of the people who exist are not morally relevant. Then we apply it to situations that involve small numbers of people, ideally one or two. If this principle ceases to generate unpleasant conclusions in these cases then it is likely only repugnant because of scope insensitivity. If it continues to generate unpleasant conclusions then it is likely the moral principle itself is broken, in which case we should discard it and find another one.

As it turns out there is an unpleasant conclusion that the TP generates on a small scale involving only two people. And it's not just something I personally find unpleasant, it's something that most people find unpleasant, and which has generated a tremendous amount of literature in the field of moral philosophy. I am talking, of course, about Replaceability. Most people find the idea that it is morally acceptable to kill (or otherwise inflict a large disutility upon) someone if doing so allows you to create someone whose life will have the same, or slightly more, utility as the remaining lifespan of the previous person, to be totally evil. Prominent philosophers such as Peter Singer have argued against Replaceability.

And I don't think that the utter horribleness of Replaceability is caused by an aversion to violence, like the Fat Man variant of the Trolley Problem. This is because the intuitions don't seem to change if we "naturalize" the events (that is, we change the scenario so that the same consequences are caused by natural events that are no one's fault). If a natural disaster killed someone, and simultaneously made it possible to replace them, it seems like that would be just as bad as if they were deliberately killed and replaced.

Since the Impersonal Total Principle generates repugnant conclusions in both large and small scale scenarios, Repugnant Conclusion's repugnance is probably not caused by scope insensitivity. The ITC itself is probably broken, we should just throw it out.** Maximizing total utility without regard to any other factors is not what morality is all about.

*Please note that when I refer to "utility" I am not referring to Von Neumann-Morgenstern utility. I am referring to "what utilitarianism seeks to maximize," i.e positive experiences, happiness, welfare, E-utility, etc.

**Please note that my rejection of the ITC does not mean I endorse the rival Impersonal Average Principle, which is equally problematic.

I read the fun theory sequence, although I don't remember it well. Perhaps someone can re-explain it to me or point me to the right paragraph.

How did you prove that we'll never run out of fun? You did prove that we'll never run out of challenges of the mathematical variety. But challenges are not the same as fun. Challenges are fun when they lead to higher quality of life. I must be wrong, but as far as I understood, you suggested that it would be fun to invent algorithms that sort number arrays faster or prove theorems that have no applications other than exercising your brain. But is this really fun for a version of me that has no worries about health, wealth, social status, etc.? And if FAI decides to impose these challenges artificially, there will be no hope of overcoming them.

No doubt, FAI will find a solution that keeps things fun for way longer than any of my schemes. But forever?

In other words, I don't see how you solved the problem of endless boredom.

(BTW, what chance of death should you accept if Omega offers you ETERNAL life?)

The problem goes away if you allow a finite present value for immortality. In other words, there should be a probability level P(T) s.t. I am indifferent between living T periods with probability 1, and living infinitely with probability P(T). If immortality is infinitely valued, then you run into all sorts of ugly reducto ad absurdum arguments along the lines of the one outlined in your post.

In economics, we often represent expected utility as a discounted stream of future flow utilities. i.e.

V = Sum (B^t)(U_t)

In order for V to converge, we need B to be less than zero, and U_t to grow at rate less than 1/B for all t > T for some T. If a person with such a utility function were offered the deal described in the post above, they would at some point stop accepting. If you offered a better deal, they would accept for a while, but then stop again. If you continued this process, you would converge to the immortality case, and survival probability would converge to P.

Of course, this particular form is merely illustrative. Any utility function that assigns a finite value to immortality will lead to the same result.

OMEGA: Wait! Come back! I have even faster-growing functions to show you! And I'll take even smaller slices off the probability each time! Come back!

HUMAN: Ahem... If you answer some questions first...

OMEGA: Let's try

HUMAN: Is it really true that the my experience on this universe can be described by a von Neumann machine with some fixed program of n bytes + m bytes of RAM, for some finite values of n and m?

OMEGA: Uh-huh (If omega answers no, then he becomes inconsistent with his previous statement that lifetime and computational resources are equivalent)

HUMAN: Could you tell me (n,m) with a minimum n?

OMEGA: Here it is: (Gives two numbers)

HUMAN: Now, by Occam's razor, I will assume that this universe is my reality. Does this Universe includes you or are we now outside the scope of this universe?

[Interpretation 1: Universe includes Omega + conversation]

HUMAN: It seems that we have a bargaining ground. We have a finite explanation of the Universe including you. Now the question is only: how do we subdivide it among us. Now, would you please show your even faster growing functions?

[Interpretation 2: Omega is outside the universe]

HUMAN: Is there some other finite universe that includes you + this whole conversation as well?

[If the answer is true, go to Interpretation 1 with that universe]

[Interpretation 3: Conversation is outside the scope of any possible finite universes]

HUMAN: Hmmm... We are outside the universe now? What is the semantics of conversing with me here? What would happen if you'd give me those resources anyways? All my subjective experiences are tied to this puny little universe. What is the meaning of me leaving that universe and dieing outside of it or accepting and refusing any kind of offers? I have trouble making a mental model of this situation.

I have moral uncertainty, and am not sure how to act under moral uncertainty. But I put a high credence that I would take the EV of year-equivalent (the concept of actual-years probably breaks down when you're a Jupiter Brain). I also put some credence that finite lives are valueless.

Eliezer said:

Once someone is alive, on the other hand, we're obliged to take care of them in a way that we wouldn't be obliged to create them in the first place

that seems like quite a big sacrifice to make in order to resolve Parfit's repugnant conclusion; you have abandoned consequentialism in a really big way.

You can get off parfit's conclusion by just rejecting aggregative consequentialism.

This situation gnaws at my intuitions somewhat less than the dust specks.

You're offering me the ability to emulate every possible universe of the complexity of the one we observe, and then some. You're offering to make me a God. I'm listening. What are these faster-growing functions you've got and what are your terms?

I couldn't resist posting a rebuttal to Torture vs. Dust Specks. Short version: the two types of suffering are not scalars that have the same units, so comparing them is not merely a math problem.

Uhm - obvious answer: "Thank you very much for the hint that living forever is indeed a possibility permitted by the fundamental laws of the universe. I think I can figure it out before the lifespan my current odds give me are up, and if I cant, I bloody well deserve to die. Now kindly leave, I really do not intend to spend what could be the last hour of my life haggling."

Mostly, this paradox sets off mental alarm bells that someone is trying to sell us a bill of goods. A lot of the paradoxes that provoke paradoxical responses have this quality.

I would definitely take the first of these deals, and would probably swallow the bullet and continue down the whole garden path . I would be interested to know if Eliezer's thinking has changed on this matter since September 2009.

However, if I were building an AI which may be offered this bet for the whole human species, I would want it to use the Kelly criterion and decline, under the premise that if humans survive the next hour, there may well be bets later that could increase lifespan further. However, if the human species goes extinct at any point, then game over, we lose, the Universe is now just a mostly-cold place with a few very hot fusion fires and rocks throughout.

The Kelly criterion is, roughly, to take individual bets that maximize the expected logarithm instead of expected utility itself. Despite the VNM axioms pretty much definining utility as that-which-is-to-be-maximized, there are theorems (which I have seen, but for which I have not seen proofs yet) that the Kelly criterion is optimal in various ways. I believe, though I don't know much about the Kelly criterion so there's a high probability I'm wrong, that it applies to maximizing total lifespan in addition to maximizing money.

So what happens if we try to maximize log(lifespan)? The article implies we should still take the bets, but I think that's incorrect and we wouldn't take even one deal (except for the total freebie before the tetration garden path). To see this, note that we only care about the 80% of the worlds where we could have survived (unless Omega offers to increase this probability somewhere...), so we'll just look at that. Now we have to choose between a 100% chance of log(life) being 10,000,000 (using base-10 log and measuring life in years), or a 99.9999% chance of log(life) being 10^10,000,000 and a 0.0001% chance of log(life) being -∞. A quick calculation shows that E(log(life)) in this case is -∞, which is far less than the E(log(life)) of 10,000,000 we get from not taking the deal.

In short, even if you accept a small chance of you dying (as you must if you want to get up in the morning), if you want the long range maximum lifespan for the human race to be as high as possible, you cannot accept even the tiniest chance of it going extinct.

This is why existential risk reduction is such a big deal; I hadn't actually made that connection when I started writing this comment.

Summary of this retracted post:

Omega isn't offering an extended lifespan; it's offering an 80% chance of guaranteed death plus a 20% chance of guaranteed death. Before this offer was made, actual immortality was on the table, with maybe a one-in-a-million chance.

I have a hunch that 'altruism' is the mysterious key to this puzzle.

I don't have an elegant fix for this, but I came up with a kludgy decision procedure that would not have the issue.

Problem: you don't want to give up a decent chance of something good, for something even better that's really unlikely to happen, no matter how much better that thing is.

Solution: when evaluating the utility of a probabilistic combination of outcomes, instead of taking the average of all of them, remove the top 5% (this is a somewhat arbitrary choice) and find the average utility of the remaining outcomes.

For example, assume utility is proportional to lifespan. If offered a choice between a 1% chance of a million years (otherwise death in 20 years) and certainty of a 50 year lifespan, choose the latter, since the former, once the top 5% is removed, has the utility of a 20 year lifespan. If offered a choice between a 10% chance of a 19,000 year lifespan (otherwise immediate death) and a certainty of a 500 year lifespan, choose the former, since once the top 5% is removed, it is equivalent in utility to (5/95)*19,000 years, or 1,000 years.

New problem: two decisions, each correct by the decision rule, can combine into an incorrect decision by the decision rule.

For example, assume utility is proportional to money, and you start with $100. If you're offered a 10% chance of multiplying your money by 100, otherwise losing it, and then as a separate decision offered the same deal based on your new amount of money, you'd take the offer both times, ending up with a 1% chance of having a million dollars, whereas if directly offered the 1% chance of a million dollars, otherwise losing your $100, you wouldn't take it.

Solution: you choose your entire combination of decisions to obey the rule, not individual decisions. As with updateless decision theory, the decisions for different circumstances are chosen to maximise a function weighted over all possible circumstances (maybe starting when you adopt the decision procedure) not just over the circumstance you find yourself in.

In the example above, you could decide to take the first offer but not the second, or if you had a random number generator, take the first offer and then maybe take the second with a certain probability.

A similar procedure, choosing a cutoff probability for low utility events, can solve Pascal's Wager. Ignoring the worst 5% of events seems too much, it may be better to pick a smaller number, though there's no objective justification for what to pick.

I could choose the arbitrary cut-off like 75%, which still buys me 10^^645385211 years of life (in practice, I would be more likely to go with 60%, but that's really a personal preference). Of course, I lose out on the tetration and faster-growth options from refusing that deal, but Omega never mentioned those and I have no particular reason to expect them.

Of course, this starts to get into Pascal's mugging territory, because it really comes down to how much you trust Omega, or the person representing Omega. I can't think of any real-life observations I could make that would convince me that someone speaks for Omega and can offer such a deal, though I'm sure there are some things that would cause me to conclude that. Nevertheless, I'm not really capable of thinking with the amount of certainty I would need to risk even a 50% chance of death for a promise of an existence long enough for a brain the size of the observable universe to cycle through all of its possible states.

So yeah, this looks like standard Pascal's mugging.

Each offer is one I want to accept, but I eventually have to turn one down in order to gain from it. Let's say I don't trust my mind to truly be so arbitrary though, so I use actual arbitrariness. Each offer multiplies my likelihood of survival by 99.9999%, so let's say each time I give myself a 99.999% chance of accepting the next offer (the use of 1 less 9 is deliberate). I'm likely to accept a lot of offers without significant decrease in my likelihood of survival. But wait, I just did better by letting probability choose than if I myself had chosen. Perhaps if rather than 1 me making all the choices, I generate infinitely many me's (or an arbitrary amount such as 3^^^3) and each one chooses 1 or 0, and then we merge and I accept the offer once for each 1 that was chosen. I'm essentially playing the prisoner's dilemma...I want to choose 1, but I don't want all the other me's to choose 1 as well, so I must choose 0 in hopes that they all also choose 0. Can I somehow get almost all of us to choose 0...?

What I'm doing wrong? I think that one obviously should be happy with 1<(insert ridiculous amount of zeros)> years for 1 - 1:10^1000 chance of dying within an hour. In a simplistic way of thinking. I could take into account things like "What's going to happen to the rest of all sentient beings", "what's up with humanity after that", and even more importantly, If this offer were to be available for every sentient being, I should assign huge negative utility for chance of all life being terminated due to ridiculously low chance of anyone winning those extra years.

Also I could assume to justify pure time discounting due to the fact that there is no premise that defines how long I'm going to live otherwise, after declining the offer. If Omega can make that offer, why couldn't post-humans make a better one after 1000 years of research? And if they can, shouldn't I be trying to maximize my chance of living merely 1000 years longer? If there was a premise that set my lifespan on case of declining the offer("You shall live only 100 years"), I'd be in favor of ridiculously low chance of ridiculously many years, if considerations from my earlier points justify that(My strategy for choosing wouldn't endanger all sentient life within the universe etc). I think the counter-intuitiveness rises from the fact that especially now it's fairly difficult to make good estimations of "natural" lifespan expectations in absence of deals from Omega.

My own analysis of the Repugnant Conclusion [...]

... is, I am gratified to see, the same as mine.

When TORTURE v DUST SPECKS was discussed before, some people made suggestions along the following lines: perhaps when you do something to N people the resulting utility change only increases as fast as (something like) the smallest program it takes to output a number as big as N. (No one put it quite like that, which is perhaps just as well since I'm not sure it can be made to make sense. But, e.g., Tom McCabe proposed that if you inflict a dust speck on 3^^^3 people, the number of non-identical people suffering the dust speck will be far smaller than that, and that that greatly reduces the resulting disutility. Wei Dai made a proposal to do with discounting utilities by some sort of measure, related to algorithmic complexity. Etc.) Anyway, I mention all this because it may be more believable within a single person's life than when aggregated over many people: while it sure seems that 10^N years of life is a whole lot better than N years when N is large, maybe for really large N that stops being true. Note that this doesn't require bounded utilities.

If so, then it seems (to me, handwavily) like the point at which you start refusing to be led down the garden path might actually be quite early. For my part, I don't think I'd take more than two steps down that path.

Since we're talking about expected utility, I'd rather you answered this old question of mine...

The problem is that using a bounded utility function to prevent this sort of thing will lead to setting arbitrary bounds to how small you want the probability of continuing to live to go, or arbitrary bounds on how long you want to live, just like the arbitrary bounds that people tried to set up in the Dust Specks and Torture case.

On the other hand, an unbounded utility function, as I have said many times previously, leads to accepting a 1/(3^^^3) probability of some good result, as long as it is good enough, and so it results in accepting the Mugging and the Wager and so on.

The original thread is here. A Google search for "wei_dai lesswrong lifetime" found it.

ETA: The solution I proposed is down in the thread here.

If I wanted to be depressing, I'd say that, right now, my utility is roughly constant with respect to future lifespan...

Does the paradox go away if we set U(death) = -∞ utilons (making any increase in the chance of dying in the next hour impossible to overcome)? Does that introduce worse problems?

Perhaps the problem here is that you're assuming that utility(probability, outcome) is the same as probability*utility(outcome). If you don't assume this, and calculate as if the utility of extra life decreased with the chance of getting it, the problem goes away, since no amount of life will drive the probability down below a certain point. This matches intuition better, for me at least.

EDIT: What's with the downvotes ?

In circumstances where the law of large numbers doesn't apply, the utility of a probability of an outcome cannot be calculated from just the probability and the utility of the outcome. So I suggested how the extra rule required might look.

Are the downvotes because people think this is wrong, or irrelevant to the question, or so trivial I shouldn't have mentioned it ?