How likely is it that AI will torture us until the end of time?

Q: How to cope with the possibility of immense suffering?

I want to address the psychological aspect first, because at the start you say “I've been stuck on s-risks for over a month now. My life has been turned upside down since I first learned about this subject.”

The most helpful emotional state for thinking about this is calm, sober, lucid, and patient. If you rush to conclusions based on anxiety, you’ll probably get the wrong answer.

Although immense suffering is possible, your body is reacting to that possibility as if it were a physical threat in your immediate environment. Your heart rate increases, your breathing gets faster, and your muscles tense. This topic requires careful thinking, so that physiological response is totally unhelpful. You can work with the problem more effectively if you're in an emotional state conducive to high quality thinking.

To calm down you can breathe more slowly, take a break from things that trigger anxiety, and observe the physical sensations of anxiety with a neutral attitude.

Q: Is suicide justified by subjective s-risk?

We don’t know enough to answer this question. There’s too much we don’t understand. The s-risk part of the equation is a complete mystery, which means the ordinary reasons against suicide take precedence. There’s no reason to sacrifice your life and the wellbeing of the people around you when the expected value is a question mark (meaning total cluelessness, not just error bars).

If you say death is oblivion and therefore reduces subjective s-risk, I would ask why you think you know that.

To take one example (that’s not quantum immortality), consider that before you were born, nothing in the universe was “you”. Then “you” came into existence. After you die, nothing in the universe will be “you”. If there’s no information in reality to identify “you” because you no longer exist, then that’s the same situation as before you were born. In other words, you’ll be in a situation that once preceded your coming into existence. Nonexistence isn’t an experience, so the subjective duration between dying and coming into existence would be zero.

In other words, the zero-information oblivion that produced you once can produce you again, maybe in a different form.

In that case, death is not subjective oblivion, but a rolling of the cosmic dice. I have no idea what experiences would follow, but I don’t see why they would predictably include less suffering.

If you say that our current universe has unusually high s-risk so the dice roll is worth it, again I ask why you think you know that. Maybe most minds exist in simulations run by unaligned AGI. Maybe our slightly pre-AGI world has unusually low s-risk.

Maybe we’re in a simulation that punishes suicide because it harms others and is therefore a defection against the common good.

When you’re in such a state of extreme uncertainty, going around sacrificing things you value, like your life, doesn’t magically help. The best thing you can do is relax, because psychological stability is conducive to clear thinking.

Here's another perspective: it is far less likely that an AGI will torture us until the end of time than that an AGI will give us unimaginable pleasures until the end of time.

While I agree that there are biases working against taking s-riaks seriously, I also think there's a powerful bias driving you to think more about the very small chance of extremely bad ASI outcomes than the much larger chance of very good outcomes. That is negativity bias. Evolution has made us worriers, because worriers survived more in dangerous natural environments. That's why we have far more anxiety than excessive joy and ambition.

I agree with the other answers that worry on your behalf that anxiety is driving your concerns. There is much to think about and much to do. Help us achieve the unimaginably good outcomes. It is a real possibility, with perhaps a 50% chance of getting there. Put your eyes on the prize, and worry about s-risks again if things start to go badly. One thing almost everyone agrees on now is that AGI will not suddenly achieve godlike power. Takeoff will be slow enough to see.

So for now, keep your eyes on the prize and think about how much we have to win.

You can help with this project by spreading awareness of AGI risks to the public. You don't need to obsess or devote your life to it if you don't want to or it's not healthy. We now have plenty of people doing that.

We're in a Pascal’s mugging situation, but from a negative point of view, where the trade-off is between potential infinite years of suffering and suicide in order to avoid them for sure.

In the past I've struggled deeply with this thought process and I have reasoned my way out of that conclusion. It's not necessarily a more hopeful conclusion but it takes away the idea that I need to make a decision, which I find very comforting.

Ultimately it comes down to the illusory nature of identity.

A super powerful entity would have the power to create bespoke conscious entities for the purpose of inducing suffering.

The suffering of "Future you" is no more or less real than the suffering of future entities in general. The only difference is that your present mind projects a sense of identity and continuity which causes you to believe there's a difference.

The illusory sense that there is continuity of consciousness and identity is evolutionarily advantageous but it fully loses coherence and relevance in the domain you're talking about.

I'm happy to go into more detail if that isn't convincing.

You could think of identity in this case as a special type of sympathetic empathy for a version of yourself in the future which you wouldn't grant to future entities which aren't "yourself". This is just a feeling that present you is feeling, and has no actual meaningful connection to the entity you'd classify as your future self.

This is the problem of multiplying a big number with a little number. It could zoom off to infinity, stabilize at a value, or shrink to nothing.

The scenario you presented seems to contain a lot of conditional probabilities, which to me make it pretty implausible. That said I don’t want to discount the idea because of the details. I think a runaway wealth gap is not an insignificant possibility.

In situations like this, I come down on the side of being aware of the possibilities, but try to remember that it’s unlikely. Brains are going to brain, so there is no helping aliefs. All I can do is give an answering voice to anxieties when they won’t shut up.

If you’re feeling overwhelmed it’s okay to step away for a bit. I’m not worried about discussing dark topics, but the impact on your current real life, as you mentioned it’s turned your world upside down. It’s good to recharge. Maybe get out in the green. If I’m misreading this, I apologize. I’d rather err on the side of reaching out.

I'm reminded of a quote I love from an apocrypha that goes roughly like this:

Q: How long will suffering rule over humans?

A: As long as women bear children.

Also, there's the possibility you are already in a digital resurrection of humanity, and thus, if you are worried about s-risks for AI, death wouldn't necessarily be an escape but an acceleration. So the wisest option would be maximizing your time when suffering is low as inescapable eternal torture could be just around the corner when these precious moments pass you by (and you wouldn't want to waste them by stressing about tomorrow during the limited number of todays you have).

But on an individualized basis, even if AI weren't a concern, everyone faces significant s-risks towards end of life. An accident could put any person into a situation where unless they have the proper directives they could spend years suffering well beyond most people's expectations. So if extended suffering is a concern, do look into that paperwork (the doctors I know cry most not about the healthy that get sick but the unhealthy kept alive by well meaning but misguided family).

I would argue that there's very, very low chances of an original human capably being kept meaningfully alive to torture for eternity though. And there's a degree of delusion of grandeur that an average person would have the insane resources necessary to extend life indefinitely spent on them just to torture them.

There's probably better things to worry about, and even then there's probably better things to do than worry with the limited time you do have in a non-eternal existence.