an ethical puzzle about brain emulation

Even in C,D, and E, the torture was still computed at some point. This will still feel like torture. It just happened before the part of the scenario you stated, where you load up the pre-computed torture. That part doesn't seem bad. See The Generalized Anti-Zombie Principle.

I would say that, by the time you get to C, there probably isn't any problem anymore. You're not actually computing the torture; or, rather, you already did that.

Scenario C is actually this:

You scan John Smith's brain, run a detailed simulation of his being tortured while streaming the intermediate stages to disk, and then stream the disk state back to memory (for no good reason).

There is torture there, to be sure; it's in the "detailed simulation" step. I find it hard to believe that streaming, without doing any serious computation, is sufficient to produce consciousness. Scenario D and E are the same. Now, if you manage to construct scenario B in a homomorphic encryption system, then I'd have to admit to some real uncertainty.

I've been thinking a lot about this issue (and the broader issue that this is a special case of) recently. My two cents:

Under most views, this isn't just an ethical problem. It can be reformulated as a problem about what we ought to expect. Suppose you are John Smith. Do you anticipate different experiences depending on how far down the sequence your enemies will go? This makes the problem more problematic, because while there is nothing wrong with valuing a system less and less as it gets less and less biological and more and more encrypted, there is something strange about thinking that a system is less and less... of a contributor to your expectations about the future? Perhaps this could be made to make sense, but it would take a bit of work. Alternatively, we could reject the notion of expectations and use some different model entirely. This "kicking away the ladder" approach raises worries of its own though.

I think the problem generalizes even further, actually. Like others have said, this is basically one facet of an issue that includes terms like "dust theory" and "computationalism."

Personally, I'm starting to seriously doubt the computationalist theory of mind I've held since high school. Not sure what else to believe though.

Could the relevant moral change happen going from B to C, perhaps? i.e. maybe a mind needs to actually be physically/causally computed in order to experience things. Then the torture would have occurred whenever John's mind was first simulated, but not for subsequent "replays," where you're just reloading data.

By the time we get to E, to a neutral observer it's just as likely we're writing the state of a happy brain rather than a sad one. See the waterfall argument, where we can map the motion of a waterfall to different computations, and thus a waterfall encodes every possible brain at once.

This probably reflects something about a simplicity or pattern-matching criterion in how we make ethical judgments.

Yeah, add me to the "If anything morally interesting is happening here, it is happening when the mind-states are computed. Copying those computed-and-recorded states into various media, including 'into memory,' doesn't have moral significance."

More generally: any interesting computational properties of a system are interesting only during computation; the stored results of those computations lack those interesting properties.

My instinct is that each mental state only happens once. If you redo the calculations, he doesn't experience it twice. It's similar to the idea that doubling the size of each neuron doesn't seem like it would change anything.

On the other hand, the Born rule suggests that the universes involved having twice the value makes them four times as likely to be experienced. That suggests that running a one-second calculation ten times would be ten times worse than just running a ten-second calculation.

In any case, I would definitely say that C isn't as bad as A. Moving written values around is not the same as running calculations.

Up-voted for the construction of this hierachy. As the other comments show this can be made finer. probably arbitrarily fine. And it shows that ethics is in the end a vague concept standing on the aggregate of our complex values. You can't find a hard border because there is no simple value to observe changing.

What if we fully homomorphically encrypt John Smith's brain simulation before we start torturing it? We can even throw away our copy of the private keys so that no one, ever, outside the simulation will be able to see the results of the torture even though we can calculate it perfectly for as long as we want.

Does encrypted John Smith have qualia? If so, then isn't the waterfall argument true? We could choose initial keys with enough entropy such that every possible state of the simulation has a corresponding key that would decrypt it. Our physics simulation just reads the encrypted state of every particle/wave in the simulation, applies the results of physics, writes the encrypted results back and repeats. Our choice of decryption key effectively picks a random universe whose evolution is fully determined by the single series of homomorphically encrypted computations, and it's no longer the computation that's creating specific qualia, but our choice of key revealing it. Another interpretation is that a single series of computations is producing all possible qualia. This seems to be a very strong case for the waterfall argument being true; the physical evolution of the universe is producing states that can be interpreted in many different ways, and the interpretation is what matters to us.

In complexity terms, fully homomorphic encryption is expensive but it is polynomial in the size of the key, and simulations are polynomial in the number of elements being simulated. If N is the number of elements and M is the number of bits necessary to represent the state of one element then the key would be NM bits. Current fully homomorphic ciphers encrypt/decrypt one bit at a time, meaning that each bit of the encrypted state would require NM bits of storage, and computations are implemented as boolean circuits using homomorphic operations on encrypted bits. Calculating one step of the simulation would require O( ( N + C(N) ) H(N M) ) time and O( ( N + C(N) ) (N M) ) space where C(x) is the boolean circuit size of a physics simulation for N elements and H(x) is the cost of homomorphic encryption operations on x bits. C(x) is polynomial if the simulation is polynomial in x. H(x) is polynomial. The result is polynomial and smaller than the Θ(M^N) universes that could be decrypted. That implies either that John Smith and his exponentially many parallel selves have qualia and we can create exponential qualia with only polynomial work, or encrypted encrypted John Smith does not have qualia. If only some of the John Smith's have qualia that would imply the existence of philosophical zombies. If we can create exponential qualia from polynomial work then I think the waterfall argument is basically true.

Nick Bostrom's paper on Duplication vs Unification claims that we should accept Duplication because otherwise we lose the ability to compare the morality of different actions. If the waterfall argument is true then we probably have to accept Unification. There may still be room for some kind of moral optimization under Unification by finding the simplest perspective/mapping such that we can observe the most good given our computational constraints, but if not then we are left with being morally neutral in cases A-F.

A is not bad because torturing a person and then restoring their initial state has precisely same consequences as forging own memory of torturing a person and restoring their initial state.

It seems the disconnect is between B & C for most people.

But why is the generative simulation (B) not morally equivalent to the replay simulation (C)?

Perhaps because the failure modes are different. Imagine the case of a system sensitive to cosmic rays. In the replay simulation, the everett bundle is locally stable; isolated blips are largely irrelevant. When each frame is causally determining the subsequent steps, the system exhibits a very different signature.

IMO, even E is problematic: where did the torture-information come from in the first place?

It all breaks done to the "nature of consciousness", which isn't fully understood yet, even we have a lot of insight.

In my view, the "nature of consciousness" lies somewhere in the computing, in the dynamics, in the current and neurotransmitters flowing into the brain (or it's simulation), more than in the actual configuration of neurons. So scenario A and B are unethical, because the process is there, so the consciousness is there, but scenario C to E aren't because there is no one to feel it.

Even without upload, if you using nanotech you synthetize many different set of brains, representing each "step" of a person being tortured, but each brain being frozen in time (kept at extra low temperature, or accelerated near the speed of light, or whatever), there is no one feeling the torture.

I take for granted that scenario A is bad: one oughtn't be inflicting pain, even if there's no permanent record or consequence of the pain.

Is this deontologically bad, virtue ethics bad or consequentialism bad? Why?

Just for reference, this has been discussed on less wrong before in my post Waterfall ethics.

Perhaps a way to consider the question would be "You're John Smith. If told in advance about this, before it happens, how much do you pay to avoid having A happen? How much do you pay to avoid having B happen? C, D, E, etc."

Of course, a great deal of this depends on context and not everyone has the same answers. Some people might say "But John Smith doesn't even know anything happened, and experiences no physical or mental consequences so he can't be a victim in any usual sense of the word. If I was John Smith I wouldn't pay a penny to avoid even A."

Which I guess brings up the question of 'Does anyone else know and how much do they know, and what does John Smith know about them?'

I mean, there are at least three potential levels of attention I'm envisioning:

1: Unknown entities are doing A-E and they never ask us about it and we can't tell when it happens in the slightest.

2: John Smith's buddy Bob says "John, I think unknown entities may have done something from A-E to you last night, but I'm not sure what. I haven't told anyone else, but I thought you should know, since they did it to you."

3: John Smith finds out anyone can download John_Smith_Pain.Sim online and that 1 million people have done so in the 24 hours since it happened, even though John Smith is personally unaware of anything that happened in John_Smith_Pain.Sim.

It seems reasonable that John Smith would be willing to pay different amounts of money to avoid 1-3 (and other knowledge combinations that aren't listed)

Of course, 3 presumably has mental consequences, which may mean that it was implied to be screened off from A.

The key used for encryption has been lost.

Did you mean the key used for decryption? And could you elaborate on the significance of this?

We can 'answer' this one from the recent Ethics of brain emulation post:

[...] the nature of the emulation does not matter: if it is cruel [in real life] the same cruel impulse is present in [the simulation]. It is like damaging an effigy: it is the intention behind doing damage that is morally bad, not the damage. Conversely, treating emulations well might be like treating dolls well: it might not be morally obligatory but its compassionate.

Note: This reasoning depends on your ethics (in this kind kantian).

Wikipedia tells me that the brain has ~120 trillion synapses. Most of the storage cost will be the per-timestep data, not the underlying topology. If we need one byte per synapse per timestep, that's 120TB/timestep.

I don't think there any reason to assume that one byte is enough to fully represent the state of a synapse.

This is just a special case of a larger question -- "dust theory."