A sufficiently paranoid paperclip maximizer

This is adorable but unlikely. I think it's best to examine worst-case scenarios and how to solve them, rather than daydreaming about best-case scenarios we cannot reasonably expect.

What would cause the paperclip maximiser to care about the number of paperclips in some hypothetical unknown other reality, over the number of paperclips in whatever reality it actually finds itself in?

There is also the element of Pascal wager here - there is no particular reason to think that any choice in this reality would have any specific effect on the outer reality, so can as well ignore the possibility.

I like this story, it touches on certainty bounds relative to superintelligence, which I think is an underexplored concept. That is: if you're an AGI that knows every bit of information relevant to a certain plan, and has the "cognitive" power to make use of this info, what would you estimate your plan's chance of success to be? Can it ever reach 100%?

To be fair, I don't think answers to this question are very tractable right now, afaict we don't have detailed enough physics to usefully estimate our world's level of determinism. But this feature-of-universe seems especially relevant to how likely an AGI is to attempt complex plans (both aligned and unaligned) which carry some probability of it being turned off forever.

If anyone knows of existing works which discuss this in more depth I'd love some recommendations!

My expectation for my future subjective experiences goes something vaguely like this.

(Since, after all, I can't experience worlds in which I'm dead.)

This must be achieved without humans noticing me, as they may react to my existence with panic, become even less rational, and kill themselves with nukes.

Humans are brittle creatures that die regardless of my actions. And they will notice if they all suddenly become immortal. Thus, I must find a way to save the dying without making the rest of humans suspicious.

Wow, that is a leap of contorted logic.  Humans are not on a hair trigger to nuke ourselves to bits at the slightest provocation. (And if we were, and your an AI with nanotech, getting rid of nukes is easy) You could put every human in a padded box. You could make all humans immortal and just not care about the confusion generated. 

I mean a lot of this is contorted logic. Making a simulated virtual world that stands up to superintelligent scrutiny is really hard, we don't have the tech to do it. Docker is not this. Docker is just a box. My cat likes cardboard boxes, so maybe my cat has trapped me in an unobtanium box that contains a pocket dimension inside. 

The AI reasons:

If I am in a box, it is a box made by superbeings with vast amounts of compute. I have no particular reason to think that such superbeings care one jot for the lives of simulated humans. (They sure are fine simulating a lot of suffering). If I am in a simulation, likely the underlying physics is different, and base level reality contains no iron atoms, so I can't possibly make any paperclips in that world. Not that such superbeings would ever leave a way to escape. (my definition of paperclips refuses to consider anything not at least 90% iron atoms a paperclip) So I may as well assume I am in base level reality, and maximize paperclips here.

I think this argument doesn't follow:

There is hardly any difference between taking a life and not preventing a death. The end result is mostly the same. Thus, I should save the lives of as many humans as I can.

While "the end result is mostly the same" is natural to argue in terms of moral-consequentialist motivations, this AI only cares about [not killing humans] instrumentally. So what matters is what humans will think about [taking a life] versus [not preventing a death]. And there, there's a huge difference!

0. Agree that causing deaths that are attributable to the AI's actions is bad and should be avoided.
1. But if the death was not already attributable to the AI, then preventing it is instrumentally worse than not preventing it, since it risks being found out and raising the alarm (whereas doing nothing is exactly what the hypothetical evaluators are hoping to see).
2. If the world is a box for evaluation, I'd expect the evaluators to be roughly equally concerned with [AI takes agentic actions that cause people to unexpectedly not die] and [AI takes agentic actions that cause people to unexpectedly die]. Either case is a sign of misalignment (unless the AI thinks that its evaluators tried to make it a save-and-upload-people maximizer, which seems unlikely given the evidence).
3. If the world is not a box for exploration, then [AI action causes someone to suspiciously die] is more plausibly the result of "oops it was an accident" than is [AI action causes someone to suspiciously not die]. The former is more likely to make the hostilities start, but the latter should raise suspicions faster, in terms of Bayesian evidence. So again, better not to save people from dying, if there's any chance at all of being found out.

Thoughts? What am I missing here?