LessWrong 2.0 Reader

If you are a naturalist or physicalist about humans - these copies are not me, they are my identical twins. If you want to go beyond naturalism or physicalism, that is perfectly fine, but based on our current understanding, these are identical twins of me and in no sense are they me. So whatever happens in these other universes - it is not going to be me going through the timeline, it will be my identical twin.

An infinite or very large universe/multiverse is highly speculative and no matter what I decide in this universe, if there is an infinite universe, it makes essentially no difference if all universes are the same. There is going to be 10^1000000 and far beyond that of my identical twins and I have no power to influence anything. To say that you can influence anything in that scenario is worse than saying that you can move earth to another galaxy by jumping on it. You have 0.00...0000...0001% effect on it, and in an infinite universe you have effectively zero effect on the fate of your copies - so no matter what you decide, you will not have any influence over it.

Note also that those questions are included for the sake of giving the reader a sense of the general flavor of the subject. The actual course will use a different set of homework questions, focused on the topics listed above [LW(p) · GW(p)].

Right, that's a cliff-hanger from the screenshot. You can find the full list of questions used in the summer workshop linked before the screenshot too if you wanted to see more detail.

Agreed and well said. Playing a number of different strategies simultaneously is the smart move. I'm glad you're pursuing that line of research.

Sorry if I sound overconfident. My actual considered belief is that AGI this decade is quite possible, and it is crazy overconfident in longer timeline predictions to not prepare seriously for that possibility.

Multigenerational stuff needs a way longer timeline. There's a lot of space between three years and two generations.

There's a very plausible sense in which you may not actually get a choice to not exist.

In pretty much any sort of larger-than-immediately-visible universe, there are parts of the world (timelines, wavefunction sections, distant copies in an infinite universe, Tegmark ensembles, etc) in which you exist and have the same epistemic state as immediately prior to this choice, but weren't offered the choice. Some of those versions of you are going to suffer for billions of years regardless of you choosing to no longer exist in this fragment of the world.

Granted, there's nothing you can do about them - you can only choose your response in worlds where you get the choice.

From the wider point of view it may or may not change things. For example suppose you knew (or the superintelligence told you as follow-up information) that in worlds having an essentially identical "you" in them, 10% will be unconditionally tortured for billions of years, and 90% are offered the question (with a 2% chance of hell and 98% chance of utopia). The superintelligence knows that in most timelines leading to hellworlds there is no care for consent while utopias do, which is why conditional on consent the chance is only 2% rather than the overall 11.8%.

If you are the sort of person to choose "nonexistence" then 10% of versions of you go to hell and 90% die. If you choose "live" then in total 11.8% of you go to hell and 88.2% to utopia. The marginal numbers are the same, but you no longer get the option to completely save all versions of you from torture.

Is it still worthwhile for those who can to choose death? This is not rhetorical, it is a question that only you can answer for yourself. Certainly those in the 1.8% would regret being a "choose life" decider and joining the 10% who never got a choice.

To say that non-existence is a gamble too is kind of like saying that a person who does not gamble in a casino is gambling too - because they are missing on a chance to win millions of dollars - to me that is more a matter of definitions and if one wants to argue for that, sure, let's accept that every single thing in life is a gamble.

Your assertion that humans will be able to integrate over large timespans might be true given the current human brain - but here we are talking about superintelligence, even with relatively primitive AIs  we are already talking about new medication and cures, superintelligence that would want to cause widespread suffering or torture you and be able to build a Dyson sphere around the sun and a thousand of other advanced technologies will be able not just to figure out how to torture you persistently (so that your brain does not adapt to the new state of constant torture) but also to increase your pain levels by 1000x - not all animals feels the same pain, and there is no reason to think that current pain experience of humans cannot be increased by a huge amount.

I don't think that it is not rational to take the gamble when the odds are 1%, much less when the odds are 20% or 49% or 70%. Let's go with 1% because I am willing to give you favourable odds - so the post asks, would you be willing to be in a torture chamber now 1 hour for every 99 hours that you are in a really happy state? We can increase that to 20 hours (20%) or what have you. And here I am talking about real extreme torture, not you have a headache. So imagine the worst torture methods that currently exist, and it is not waterboarding - check worst torture methods of history and if you are objective, whatever odds you would be willing to accept, if you say 20% or 1%, would you be willing to be really tortured for that amount of time single every day?

Kelly always applies

Ah, I meant, would agree to pause once things came to a head. Pretty sure these political leaders are selfish enough that if they saw clear evidence of their imminent demise, and had a safer option, they'd take the out.

0. You people are somewhat crazy overconfident about humanity knowing enough to make AGI this decade. https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce [LW · GW]

1. One hope on the scale of decades is that strong germline engineering should offer an alternative vision to AGI. If the options are "make supergenius non-social alien" and "make many genius humans", it ought to be clear that the latter is both much safer and gets most of the hypothetical benefits of the former.