Saying the quiet part out loud: trading off x-risk for personal immortality

At this point timelines look short enough that you likely increase your personal odds of survival more by increasing the chance that AI goes well than by speeding up timelines. Also I don't see why you think cryonics doesn't make sense as alternative option.

I do not want the timing of AGI development to be based on my personal life-expectancy. Going further I think most decent people would be willing to sacrifice their own life to prevent their civilization from going extinct, and I think it would be a very honorable thing to do ^[1]. Every single generation to come is at stake, so I don't think my own life bears much on whether to defend all ${10}^{50+}$ of theirs.

1. ^{^}

   Not that I am aware of any such opportunity, to be clear.

I've seen this kind of opinion before (on Twitter, and maybe reddit?), and I strongly suspect that the average person would react with extreme revulsion to it. It most closely resembles "cartoon villain morality", in being a direct tradeoff between everyone's lives and someone's immortality. People strongly value the possibility of their children and grandchildren being able to have further children of their own, and for things in the world to continue on. And of course, the statement plays so well into stereotypes of politically-oriented age differences: Old people not sufficiently caring about what happens after they die, so they'll take decisions that let young people deal with catastrophes, young people thinking they'll never die and being so selfish that they discount the broader world outside themselves, etc. If anything, this is a "please speak directly into the microphone" situation, where the framing would pull people very strongly in the direction of stopping AGI.

I agree with the Statement. As strongly as I can agree with anything. I think the hope of current humans achieving... if not immortality, then very substantially increased longevity... without AI doing the work for us, is at most a rounding error. And ASI that was even close to aligned, that found it worth reserving even a billionth part of the value of the universe for humans, would treat this as the obvious most urgent problem and solve death pretty much if there's any physically possible way of doing so. And when I look inside, I find that I simply don't care about a glorious transhumanist future that doesn't include me or any of the particular other humans I care about. I do somewhat prefer being kind / helpful / benificent to people I've never met, very slightly prefer that even for people who don't exist yet, but it's far too weak a preference to trade off against any noticeable change to the odds of me and everyone I care about dying. If that makes me a "sociopath" in the view of someone or other, oh well.

I've been a supporter of MIRI, AI alignment, etc. for a long time, not because I share that much with EA in terms of values, but because the path to the future having any value has seemed for a long time to route through our building aligned ASI, which I consider as hard as MIRI does. But when the "pivotal act" framing started being discussed, rather than actually aligning ASI, I noticed a crack developing between my values and MIRI's, and the past year with advocacy for "shut it all down" and so on has blown that crack wide open. I no longer feel like a future I value has any group trying to pursue it. Everyone outside of AI alignment is either just confused and flailing around with unpredictable effects, or is badly mistaken and actively pushing towards turning us all into paperclips, but those in AI alignment are either extremely unrealistically optimistic about plans that I'm pretty sure, for reasons that MIRI has argued, won't work; or, like current MIRI, they say things like that I should stake my personal presence in the glorious transhumanist future on cryonics (and what of my friends and family members who I could never convince to sign up? What of the fact that, IMO, current cryonics practice probably doesn't even prevent info-theoretical death, let alone give one a good shot at actually being revived at some point in the future?)

I happen to also think that most plans for preventing ASI from happening soon, that aren't "shut it all down" in a very indiscriminate way, just won't work - that is, I think we'll get ASI (and probably all die) pretty soon anyway. And I think "shut it all down" is very unlikely to be societally selected as our plan for how to deal with AI in the near term, let alone effectively implemented. There are forms of certain actors choosing to go slower on their paths to ASI that I would support, but only if those actors are doing that specifically to attempt to solve alignment before ASI, and only if it won't slow them down so much that someone else just makes unaligned ASI first anyway. And of course we should forcibly stop anyone who is on the path to making ASI without even trying to align it (because they're mistaken about the default result of building ASI without aligning it, or because they think humanity's extinction is good actually), although I'm not sure how capable we are of stopping them. But I want an organization that is facing up to the real, tremendous difficulty of making the first ASI aligned, and trying to do that anyway, because no other option actually has a result that they (or I) find acceptable. (By the way, MIRI is right that "do your alignment homework for you" is probably the literal worst possible task to give to one's newly developed AGI, so e.g. OpenAI's alignment plan seems deeply delusional to me and thus OpenAI is not the org for which I'm looking.)

I'd like someone from MIRI to read this. If no one replies here, I may send them a copy, or something based on this.

Apparent moral alien here. Hi! I'm pretty astonished how many people apparently believe something like that statement. In my world, it's an off-the-deep-end perverse and sociopathic sort of value to hold. Of course, it's consistent. I can work with sociopaths, because to a first approximation I can work with anyone. But if there's enough delay, cooperation is going to become strained, because I'm quite happy with me dying, and you dying, and your loved ones dying (ideally painlessly having lived fulfilling lives!), if it turns out to be necessary in order for there to be a valuable future at all.

Despite introducing myself as an alien, I currently think most humans don't espouse your statement, because most humans' moral circle is big enough to preclude it. Other humans reject it on the basis of philosophical open/empty individualism. Others reject it for superstitious reasons or due to ethical injunctions. Others still for the deceptive reasons you mentioned. There are basically a lot of reasons either not buy it, or be quiet about it if you do!

As an aside, it appears that in certain circles, the deceptive/prudent reasons you mention to not espouse this view are inverted: a sort of neoliberal/objectivist-celebratory crowd would tend to praise this sort of thing.

A separate aside: there is a strong implicit assumption that AGI is a necessary prerequisite to functional immortality, which isn't at all obvious to me. I think it's an insufficiently-justified received wisdom from the days of sloppier, lower-stakes futurism.

Finally, I'm much less confident than you that a locked-in pause is likely happening!

I thought your analysis was generally pretty good otherwise.

I kind of agree with the statement, but I don't quite agree with the implications: my timelines are already quite short and I think that if we contract things more, we pay a high cost in risk for little benifit unless you expect yourself and many of your loved ones to die before then. In which case, what's the point of discussing it? 

Now if timelines were long, then yeah, it would make sense to accelerate AI insofar as it contributed to the continued surival of me and my loved ones. That could be through creating a FAI, or it could be via improved longevity tech, better disease treatment, improved cryonics, mind uploading etc. Which many people in this community are quite in favour of, even now. 

Fifth observation: Believing in the Statement is compatible with folk morality and revealed preferences of most of the population.

I think you underestimate how weird people find the desire for immortality. Why does pretty much no-one sign up for cryonics? Because it is unlikely to work? Nah, the odd's are maybe a few percent for current tech. Plenty of people should be willing to take a bet for continued life at those odds. Is it because of cost? Again, no. Life insurance isn't that expensive. The real reason is that it is weird. 

I wrote about this in Appendix A of this post [LW · GW].
------
One might look at the rough 50/50 chance at immortality given surviving AGI and think "Wow, I should really speed up AGI so I can make it in time!". But the action space is more something like:

1. Work on AI safety (transfers probability mass from "die from AGI" to "survive AGI")
   1. The amount of probability transferred is probably at least a few microdooms per person.
2. Live healthy and don't do dangerous things (transfers probability mass from "die before AGI" to "survive until AGI")
   1. Intuitively, I'm guessing one can transfer around 1 percentage point of probability by doing this.
3. Do nothing (leaves probability distribution the same)
4. Preserve your brain if you die before AGI (kind of transfers probability mass from "die before AGI" to "survive until AGI")
   1. This is a weird edge case in the model and it conditions on various beliefs about preservation technology and whether being "revived" is possible
5. Delay AGI (transfers probability from "die from AGI" to "survive AGI" and from "survive until AGI" to "die before AGI")
6. Accelerate AGI (transfer probability mass from "survive AGI" to "die from AGI" and from "die before AGI" to "survive until AGI")

I think working on AI safety and healthy living seem like a much better choice than accelerating AI. I'd guess this is true even for a vast majority of purely selfish people. 

For altruistic people, working on AI safety clearly trumps any other action in this space as it has huge positive externalities. This is true for people who only care about current human lives (as one microdoom $\approx$ 8,000 current human lives saved), and it's especially true for people who place value on future lives as well (as one microdoom = one millionth of the value of the entire long term future).

This is a very simplified view of what it means to accelerate or delay AGI, which ignores that there are different ways to shift AGI timelines that transfer probability mass differently. In this model I assume that as timelines get longer, our probability of surviving AGI increases monotonically, but there are various considerations that make this assumption shaky and not generally true for every possible way to shift timelines (such as overhangs [? · GW], different actors being able to catch up to top labs, etc.)

Two days ago I wrote a tweet with similar idea and it get unexpected boost - which means that it is important thing for people: 'Perhaps, we don't need superintelligent AI to solve aging quickly. Some more mundane level of AI will be enough." https://twitter.com/turchin/status/1719315694813130896 

EY wrote against the statement, saying that why not all these people not signing for cryonics

M. Batin wrote in favor of this statement and even made a counter-appearance during AI Pause meeting in SF https://twitter.com/MikhailBatin/status/1719948531233140754 

I also agree with the statement. I'm guessing most people who haven't been sold on longtermism would too.

When people say things like "even a 1% chance of existential risk is unacceptable", they are clearly valuing the long term future of humanity a lot more than they are valuing the individual people alive right now (assuming that the 99% in that scenario above is AGI going well & bringing huge benefits).

Related question: You can push a button that will, with probability P, cure aging and make all current humans immortal. But with probability 1-P, all humans die. How high does P have to be before you push? I suspect that answers to this question are highly correlated with AI caution/accelerationsim

You have a very strange understanding of politics, wherein you have laymen who want to advance their interests at the expense of other peoples, who realize that would be unpopular if they stated exactly what they were doing, and then as a consequence lie on Twitter about needing to do it for a different reason. This is insufficiently cynical. People almost always genuinely think their political platform is the Right and Just one. It's just that people also have strong instincts to align themselves with whatever tribe is saying they're the Good Guys, and that giving the Good Guys money is going to be great for the economy, and that the Bad Guys deserve to get their money taken away from them, the greedy pricks.

So the tradeoff you mention in your problem statement is simply too abstract for these instincts to kick in, even if it were real. The instincts kick in for Yann Lecun or Marc Andreesen, because their hindbrains don't have to spend any extra time to see the obvious point that they lose near term prestige if AI research is considered evil. But I doubt either of them are losing sleep over this because they're making a calculation about how likely it is they'll be immortal; that's the sort of high-concept tradeoff that people simply don't organize their entire politics around, and just end up finding cartoon-villanous.

Based on personal experience, you are definitely not the only one thinking about that Statement.

The statement seems like it's assuming:

1. we know roughly how to build AGI

2. we decide when to do that

3. we use the time between now and then to increase chance of successful alignment

4. if we succeed in alignment early enough, you and your loved ones won't die

I don't think any of these are necessarily true, and I think the ways they are false is asymmetric in a manner that favors caution

I think most humans agree with this statement in an "I emotionally want this" sort of way. The want has been sublimated via religion or other "immortality projects" (see The Denial of Death). The question is, why is it taboo, and it is taboo in the sense you say? (a signal of low status)

I think these elements are at play most in peoples mind, from lay people to rationalists:

1. It's too weird to think about: Considering the possibility of a strange AI-powered world where either complete extinction or immortality are possible feels "unreal". Our instinct that everything that happens in the world is within an order of magnitude of "normal" directly opposes being able to believe in this. As a result, x/s-risk discussions, either due to personal imagination or optics reasons, are limited to natural extrapolations of things that have occurred in history (eg. biological attacks, disinformation, weapons systems, etc.). It's too bizarre to even reckon that there is a non-zero chance immortality via any conduit is possible. This also plays into the low status factor: weird, outlandish opinions on the future not validated by a high-status figure are almost always met with resistance
2. The fear of "missing out" leads to people not even wanting to think about it seriously at all: People don't want give higher credence to hypotheticals that increase the scale of their losses. If we think death is the end for everyone, it doesn't seem so bad to imagine. If we think that we will be the ones to maybe die and others won't, or that recent or past loved ones are truly gone forever in a way not unique to humankind, it feels unfair/insulting by the universe.
3. Taking it seriously would massively change one's priorities in life and upset the equilibrium of their current value structures: As in: one would do everything they could to minimize the risk of early death. if they believe immortality could be possible in 20 years or less, their need for long term planning is reduced, as immortality would also imply post-scarcity, so their assiduous saving, sacrifices for their children, future are worthless. That cognitive dissonance does not sit well in the mind and hinders one's individual agentic efficiency.

I have expressed versions of this Statement, or at least parts of it, to those close to me. They are not in the EA or LW communities and so I have to rephrase a lot. Mostly along the lines of, "One way or another, the last generation to die of old age has most likely already been born. I'm not quite sure which one, but if we manage to survive the century as a species, then I expect it's my own, and that my nieces and nephew will be able to live for millennia. I hope I'm wrong about which generation it is."

For me, both AI and medicine are far outside my areas of expertise, so I'm focusing on other problems that I have a better chance of influencing, hence the passive framing. This is even more true for my family members.

Off-topic, but somewhat related. I want to know if there is anyone reading these words who is willing to admit that he or she is kind of hoping humanity will go extinct because humanity has been unfair to him or her or because (for some other reason) humanity is bad or unworthy.

Honestly, I haven’t seen much about individual biological immortality, or even significant life-extension, in the last few years.

I suspect progress on computational consciousness-like mechanisms has fully eclipsed the idea that biological brains in the current iteration are the way of the future. And there’s been roughly no progress on upload, so the topic of immortality for currently-existing humans has mostly fallen away.

Also, if/when AI is vastly more effective than biological intelligence, it takes a lot of the ego-drive away for the losers.

Who said anything about slowly and painfully?? I don't think a fully reflective you would endorse that.

(FWIW I'm not whoever downvoted your comments, though these definitely aren't to my taste and could be interpreted as threatening)

I'm very much of an egoist but I don't really agree with this statement. Even if the last smidge of my altruism were deleted, I still wouldn't rob banks or run a fraudulent asset exchange because I have a very cushy position in society as it is and cooperating with humanity in fact makes my life much better than not. I similarly wouldn't do existentially crazy things to try to reach the singularity for me personally. Even if normal people wouldn't understand how I'm burning the commons, you guys would.

I'm like way more likely to do norm-breaking stuff (like sell unlicensed securities) for altruistic reasons, though it is still usually a bad idea.

--

I really do feel it's a bummer that I don't get to be part of the Future. It really does suck. The way I cope is by trying to make the present a little bit more future (in a way that isn't existentially threatening). I'll get to see a few cool things this way at least.

I think a lot of people haven't fully considered how amazing immortality is. Your 7th observation is probably very accurate too. Socially sophisticated people want others to believe they're fighting for the greater good rather than their own self interest. That doesn't mean they're necessarily lying.

I fully agree with the Statement but also support Pause/Stop. The reason is that I'm highly confident that everyone dies if we achieve AGI in the near future, so since I'm still relatively young I prefer getting to live some more years instead.

Not much to add content wise but just to be one data point: I strongly agree and have talked about it with exactly one person. I have no plans to advance the position openly and proactively, largely for reasons stated, but would quietly and strongly support efforts to that end should I become aware of them.

As I've put it to my friend: the worst day for a transcendent AI to emerge is the week after I would die and the best day is the week before. We then argue over where "tomorrow" fits on that spectrum

I'm very glad that you're raising this topic for discussion. I've recently been thinking about the same thing myself. If I could delay the advent of AGI/TAI by, say, 100 years, would I? There are at least two relevant considerations here: 1) the commonly held (but not, to me, obviously true) assumption that delaying AGI/TAI increases the probability of it being safely deployed and 2) the costs of delaying AGI/TAI. 

(2) motivates thinking really hard about whether (1) is true. General concern about AI safety also motivates thinking hard about whether (1) is true, since if delaying AGI/TAI does not increase the probability of a safe deployment, then we should think about what would increase it.

You should consider cross-posting to EAF. Would be interesting to see the poll differences, but might be fiddly to set up.

If I take the statement face value as written, s-risks are very relevant to disagreement - a life lived longer without solving sources of suffering tends to result in accruing damage, and saving my life immediately before I would die is one of the more likely ways to force me to live with the maximum possible damage dealt by (what just nearly killed me, and cumulative health troubles in general).

I notice that "don't die" is rarely a terminal value. Most people who say they want immortality are making some assumptions about the quality of life it would entail -- both in an s-risk sense and in a "technically people who are permanently comatose on life support aren't actually dead" sense as well.

I wouldn't call "don't die" a terminal value in the cryonics case either, because in that case you do die for awhile but then you get better. Eventual reincarnation after a delay seems meaningfully distinct from immortality, because immortality implies continuity of life.

To be naively utilitarian, this question is reducible to following expression:

shouldBeAMoronAndAccelerate = ( pda * (ud + gud / sf) + pia * ui + (1 - pda - pia) * ud > pds * (ud + gud / sf) + pis * ui + (1 - pds - pis) * ud )

where:

pda: Probability of doom if you ignore safety and accelerate

pds: Probability of doom if you pay attention to safety

pia: Probability of achieving immortality if you accelerate

pis: Probability of achieving immortality of you play safe

ud: "Selfish" utility gain if you die (very negative)

gud: "Common good" utility gain if everyone dies (even more negative)

ui: "Selfish" utility gain if you achieve immortality (positive)

sf: "Selfish factor" - how much "Common good" utility you're willing to sacrifice per unit "Selfish" utility.

(you can also add terms for living happily ever after without immortality and stuff like that, but utility gain in this cases is negligible in comparison)

The main problem I see is that pia < pis.

In other words, no matter how high your "Selfish factor" is - accelerating is equivalent to shooting yourself in a leg. Also you die. Also everyone dies.