When is a mind me?

I think there's a pretty strong argument to be more wary about uploading. It's been stated a few times on LW, originally by Wei Dai if I remember right, but maybe worth restating here.

Imagine the uploading goes according to plan, the map of your neurons and connections has been copied into a computer, and simulating it leads to a person who talks, walks in a simulated world, and answers questions about their consciousness. But imagine also that the upload is being run on a computer that can apply optimizations on the fly. For example, it could watch the input-output behavior of some NN fragment, learn a smaller and faster NN fragment with the same input-output behavior, and substitute it for the original. Or it could skip executing branches that don't make a difference to behavior at a given time.

Where do we draw the line which optimizations to allow? It seems we cannot allow all behavior-preserving optimizations, because that might lead to a kind of LLM that dutifully says "I'm conscious" without actually being so. (The p-zombie argument doesn't apply here, because there is indeed a causal chain from human consciousness to an LLM saying "I'm conscious" - which goes through the LLM's training data.) But we must allow some optimizations, because today's computers already apply many optimizations, and compilers even more so. For example, skipping unused branches is pretty standard. The company doing your uploading might not even tell you about the optimizations they use, given that the result will behave just like you anyway, and the 10x speedup is profitable. The result could be a kind of apocalypse by optimization, with nobody noticing. A bit unsettling, no?

The key point of this argument isn't just that some optimizations are dangerous, but that we have no principled way of telling which ones are. We thought we had philosophical clarity with "just upload all my neurons and connections and then run them on a computer", but that doesn't seem enough to answer questions like this. I think it needs new ideas.

I find myself strongly disagreeing with what is being said in your post. Let me preface by saying that I'm mostly agnostic with respect to the possible "explanations" of consciousness etc,  but I think I fall squarely within camp 2. I say mostly because I  lean moderately towards physicalism.

First, an attempt to describe my model of your ontology:

You implicitly assume that consciousness / subjective experience can be reduced to a physical description of the brain, which presumably you model as a classical (as opposed to quantum) biological electronic circuit.  Physically, to specify some "brain-state" (which I assume is essentially the equivalent of a "software snapshot" in a classical computer) you just need to specify a circuit connectivity for the brain, along with the currents and voltages between the various parts of the circuit (between the neurons let's say). This would track with your mentions of reductionism and physicalism and the general "vibe" of your arguments.  In this case I assume you treat conscious experience roughly as "what it feels like" to be software that is self-referential on top of taking in external stimuli from sensors. This software is instantiated on a biological classical computer instead of a silicon-based one.

With this in mind, we can revisit the teleporter scenario. Actually, let's consider a copier instead of a teleporter, in the sense that you dont destroy the original after finishing the procedure. Then, once a copy is made, you have two physical brains that have the same connectivity, the same currents and the same voltages between all appropriate positions. Therefore, based on the above ontology, the brains are physically the same in all the ways that matter and thus the software / the experience is also the same. (Since software is just an abstract "grouping" which we use to  refer to the  current physical state of the hardware)

Assuming this captures your view, let me move on to my disagreements:

My first issue with your post is that this initial ontological assumption is neither mentioned explicitly nor motivated. Nothing in your post can be used as proof of this initial assumption. On the contrary,  the teleporter argument, for example,  becomes simply a tautology if you start from your premise - it cannot be used to convince someone that doesn't already subscribe to your views on the topic. Even worse, it seems to me that your initial assumption forces you to contort (potential) empirical observation to your ontology, instead of doing the opposite.

To illustrate, let's assume we have the copier - say it's a room you walk into, you get scanned and then a copy is reconstructed in some other room far away.  Since you make no mention of quantum, I guess this can be a classical copy, in the sense that it can copy essentially all of the high-level structure, but it cannot literally copy the positions of specific electrons, as this is physically impossible anyways. Nevertheless, this copier can be considered "powerful" enough to copy the connectivity of the brain and the associated currents and voltages. Now, what would be the experience of getting copied, seen from a first-person, "internal", perspective? I am pretty sure it would be something like: you walk into the room, you sit there, you  hear say the scanner working for some time, it stops, you walk out. From my agnostic perspective, if I were the one to be scanned it seems like nothing special would have happened to me in this procedure. I didnt feel anything weird, I didnt feel my "consciousness split into two" or something. Namely, if I consider this procedure as an empirical experiment, from my first person perspective I dont get any new / unexpected observation compared to say just sitting in an ordinary room. Even if I were to go and find my copy, my experience would again be like meeting a different person which just happens to look like me and which claims to have similar memories  up to the point when I entered the copying room. There would be no way to verify or to view things from their first person perspective.

At this point, we can declare by fiat that me and my copy are the same person / have the same consciousness because our brains, seen as classical computers, have the same structure, but this experiment will not have provided any more evidence to me that this should be true. On the contrary, I would be wary to, say, kill myself or to be destroyed after the copying procedure, since no change will have occured to my first person perspective, and it would thus seem less likely that my "experience" would somehow survive because of my copy.

Now you can insist that philosophically it is preferable to assume that brains are classical computers etc, in order to retain physicalism which is preferable to souls and cartesian dualism and other such things. Personally, I prefer to remain undecided, especially since making the assumption brain= classical hardware, consciousness=experience as software leads to weird results. It would force me to conclude that the copy is me even though I cannot access their first person perspective (which defeats the purpose) and it would also force me to accept that even a copy where the "circuit" is made of water pipes and pumps, or gears and levers also have an actual, first person experience as "me", as long as the appropriate computations are being carried out.  

One curious case where physicalism could be saved and all these weird conclusions could be avoided would be if somehow there is some part of the brain which does something quantum, and this quantum part is the essential ingredient for having a first person experience. The essence would be that, because of the no-cloning theorem, a quantum-based consciousness would be physically impossible to copy, even in theory. This would get around all the problems which come with the copyability implicit in classical structures. The brain would then be a hybrid of classical and quantum parts, with the classical parts doing most of the work (since neural networks which can already replicate a large part of  human abilities are classical) with some quantum computation mixed in, presumably offering some yet unspecified fitness advantage. Still, the consensus is that it is improbable that quantum computation is taking place in the brain, since quantum states are extremely "fragile" and would decohere extremely rapidly in the environment of the brain...

If you think there’s something mysterious or unknown about what happens when you make two copies of yourself

Eliezer talked about some puzzles related to copying and anticipation in The Anthropic Trilemma [LW · GW] that still seem quite mysterious to me. See also my comment [LW(p) · GW(p)] on that post.

the English language is adapted to a world where "humans don't fork" has always been a safe assumption.

If we can clone ourselves, language would probably quickly follow. The bigger change would probably be the one about social reality. What does it mean to make a promise? Who is the entity you make a trade with? Is it the collective of all the yous? Only one? But which one if they split? The yous resulting from one origin will presumably have to share or split their resources. How will they feel about it? Will they compete or agree? If they agree it makes more sense for them to feel more like a distributed being. The thinking of "I" might get replaced by an "us".

I’d guess that this illusion comes from not fully internalizing reductionism [? · GW] and naturalism [? · GW] about the mind.

Naturalism and reductionism are not sufficient to rigourously prove either form of computationalism -- that performing a certain class of computations is sufficient to be conscious in general, or that performing a specific one is sufficient to be a particular conscious individual.

This has been going on for years: most rationalists believe in computationalism, none have a really good reason to.

Arguing down Cartesian dualism (the thing rationalists always do) doesn't increase the probability of computationalism, because there are further possibilities , including physicalism-without-computationalism (the one rationalists keep overlooking) , and scepticism about consciousness/identity.

One can of course adopt a belief in computationalism, or something else, in the basis of intuitions or probabilities. But then one is very much in the ream of Modest Epistemology, and needs to behave accordingly.

"My issue is not with your conclusion, it’s precisely with your absolute certainty, which imo you support with cyclical argumentation based on weak premises".

Yep.

There isn’t a special extra “me” thing separate from my brain-state, and my precise causal history isn’t that important to my values.

If either kind of consciousness depends on physical brain states, computationalism is false. That is the problem that has rarely been recognised, and never addressed.

The particular* brain states* look no different in the teleporter case than if I’d stepped through a door; so if there’s something that makes the post-teleporter Rob “not me” while also making the post-doorway Rob “me”, then it must lie outside the brain states, a Cartesian Ghost.

There's another option: door-Rob has physical continuity. There's an analogy with the identity-over-time of physical objects: if someone destroyed the Mona Lisa, and created an atom-by-atom duplicate some time later, the duplicate would not be considered the same entity (numerical identity).

There isn’t an XML tag in the brain saying “this is a new brain, not the original”!

That's not a strong enough argument. There isn't an XML tag on the copy of the Mona Lisa, but it's still a copy.

This question doesn’t really make sense from a naturalistic perspective, because there isn’t any causal mechanism that could be responsible for the difference between “a version of me that exists at 3pm tomorrow, whose experiences I should anticipate experiencing” and “an exact physical copy of me that exists at 3pm tomorrow, whose experiences I shouldn’t anticipate experiencing”.

There is, and its multi-way splitting, whether through copying or many worlds branching. The present you can't anticipate having all their experiences, because experience is experienced one-at-a-time. They can all look back at their memories, and conclude that they were you, but you can't simply reverse that and conclude that you will be them , because the set-up is asymmetrical.

Scenario 1 is crazy talk, and it’s not the scenario I’m talking about. When I say “You should anticipate having both experiences”, I mean it in the sense of Scenario 2.

Scenario 2: “Two separate screens.” My stream of consciousness continues from Rob-x to Rob-y, and it also continues from Rob-x to Rob-z. Or, equivalently: Rob-y feels exactly as though he was just Rob-x, and Rob-z also feels exactly as though he was just Rob-x (since each of these slightly different people has all the memories, personality traits, etc. of Rob-x — just as though they’d stepped through a doorway).

But that isn't an experience. It's two experiences. You will not have an experience of having two experiences. Two experiences will experience having been one person.

If I expect to be uploaded tomorrow, should I care about the upload in the same ways (and to the same degree) that I care about my future biological self?

1. Yeah.

Are you going to care about 1000 different copies equally?

What does it mean when one "should anticipate" something? At least in my mind, it points strongly to a certain intuition, but the idea behind that intuition feels confused. "Should" in order to achieve a certain end? To meet some criterion? To boost a term in your utility function?

I think the confusion here might be important, because replacing "should anticipate" with a less ambiguous "should" seems to make the problem easier to reason about, and supports your point.

For instance, suppose that you're going to get your brain copied next week. After you get copied, you'll take a physics test, and your copy will take a chemistry test (maybe this is your school's solution to a scheduling conflict during finals). You want both test scores to be high, but you expect taking either test without preparation will result in a low score. Which test should you prepare for?

It seems clear to me that you should prepare for both the chemistry test and the physics test. The version of you that got copied will be able to use the results of the physics preparation, and the copy will be able to use the copied results of the chemistry preparation. Does that mean you should anticipate taking a chemistry test and anticipate taking a physics test? I feel like it does, but the intuition behind the original sense of "should anticipate" seems to squirm out from under it.

Here's a thought experiment.

In version A, I have a button that non invasively scans my brain and creates 10 perfect copies of my brain state in a computer. I press the button. For an instant, 11 identical mind states exist in the universe. Then each mind starts diverging along different causal chains.

Intuitively, I expect the following:

• I won't experience anything unusual after pressing the button (eg, I won't wake up in a computer). I will still feel that I am in my physical body, in the room with the button
• each of the mind copies will feel that they are the 'one true version of vlad' and won't experience the other minds 'from the inside'. presumably, they will be surprised to be in a computer and not in the room?
• if I shut down the computer and kill the 10 minds, I won't experience anything unusual

In this case, I identify myself with the embodied mind.

In version B, the setup is identical except the scan is destructive. The second I press it, my physical body is destroyed.

Now, what happens to me? There's no specific reason for me to end up in one of the minds and not the others. But I cannot go to all 10 minds at the same time — I am a single mind with its own casual chain, not a collection of minds.

For instance, imagine each of the 10 minds is caused to feel a different sensation at the same time. There's nobody to feel all 10 sensations at the same time because the minds are causally isolated. Yet I cannot say that I am feeling a particular sensation and not the others.

So in version B, I still identify myself with the embodied mind, which is destroyed — hence oblivion. Conversely, what happens to the 10 minds if I delete them from the computer? Oblivion.

(This is just my attempt to map my naive intuitions. I have a sense some version of no-self could be the solution, but I'm not there yet. I also feel that naive intuitions fail for Everett branches which is another reason to be suspicious.)

An interesting consequence of your description is that resurrection is possible if you can manage to reconstruct the last brain state of someone who had died.  If you go one one step further, then I think it is fairly likely that experience is eternal, since you don't experience any of the intervening time (akin to your film reel analogy with adding extra frames in between) being dead and since there is no limit to how much intervening time can pass.

Loved the post and all the comments <3

Here is I think an interesting scenario / thought experiment:

1.  A copy of a person is made while that original person is sleeping on a bed.
2. The original person is moved to a sofa while still sleeping.
3. The copy (which is also sleeping) is put in the bed at the exact same position where the original person was.
4. After a while the original and the copy both wake up and can see each other (we assume they are both completely oblivious to exactly what happened while they were sleeping and that they didn't dream or they dreamt the same thing, etc...)

At wake-up, based on their own memory of where the original person fell asleep, the original person will likely feel they are the copy and the copy will likely feel they are the original person, wouldn't they?!

Some might even argue that based on stream-of-consciousness continuity the original "me" is actually the copy (because the copy remembers falling asleep in the bed and actually wakes up in the bed as well).

Some others will argue that based on substrate/matter continuity the original "me" is the original person even if their stream-of-consciousness has experienced a discontinuity (remembering falling asleep in the bed but actually waking up on the sofa while seeing an identical person as them waking up in the bed).

I guess it is subjective and a matter of individual preference if the stream-of-consciousness continuity or the substrate continuity is more important to define who the original "me" is.

Some would even argue that in this case there is not actual any firm original "me", just one "stream-of-consciousness me" and another different "substrate me".

(The same/similar thought experiment could be done using the direct brain insertion of false memories instead of moving around people while they sleep / are unconscious, in this example an original person could be inserted false memories that they are a copy and vice-versa to manipulate the memory / self-awareness of who the original "me" is, also generally it obviously could be useful when someone is uploaded/copied if they want to alter some memories of their upload/copy for some reason)

Wouldn't it follow that in the same way you anticipate the future experiences of the brain that you "find yourself in" (i.e. the person reading this) you should anticipate all experiences, i.e. that all brain states occur with the same kind of me-ness/vivid immediacy?

It seems that since there is nothing further than makes the experiences (that are these brains states, in this body that is writing these sentences) in some way special so that they're "mine" (there is no additional "me-ghost"), then those particular brain states aren't any different from all the other brain states, of other brains, of other people (or other conscious beings) - and so they should equally be anticipated as existing with the same kind of immediacy and vividness.

I.e. in the same way I anticipate the future experiences of this brain, of the person writing these sentences, I should anticipate all other experiences. Which just means, all brain states exist in the same vivid, for-me way, since there is nothing further to distinguish between them that makes them this vivid, i.e. they all exist HERE-NOW. They are all the same in that sense, they are all equally mine. (But of course, the "me" here isn't, then, the particular person that I find myself being, but simply the immediacy or the way of being of those experiences themselves, i.e. simply their presence.)

Btw, this is Open Individualism.

So if something makes no physical difference to my current brain-state, and makes no difference to any of my past or future brain-states, then I think it's just crazy talk to think that this metaphysical bonus thingie-outside-my-brain is the crucial thing that determines whether I exist, or whether I'm alive or dead, etc.

There is one important aspect where it does make a difference. A difference in social reality. The brain states progress in a physically determined way. There is no way they could have progressed differently. When a "decision is made" by the brain, then that is fully the result of the inner state and the environment. It could only have happened differently if the contents of the brain had been different - which they were not. They may have been expected to be different by other people ('s brains), but that is in their map, not in reality. But our society is constructed based on the assumption that things could have been different, that actions are people's 'faults'. That is an abstraction that has shown to be useful. Societies that have people who act as if they are agents with free will maybe coordinate better - because it allows feedback mechanisms on their behaviors.  

When faced with confusing conundrums like this, I find it useful to go back to basics: evolutionary psychology [LW · GW]. You are a human, that is to say, you're an evolved intelligence, one evolved as a helpful planning-and-guidance system for a biological organism, specifically a primate. Your purpose, evolutionarily, is to maximize the evolutionary fitness of your genes, i.e. to try your best to pass them on successfully. You have whole bunch of drives/emotions/instincts that were evolved to, on the African Savannah, approximately maximize that fitness. Even in our current rather different environment, while not quite as well tuned to that as they used to be to the Savannah, these still do a pretty good job of that (witness the fact that there are roughly 8 billion of us).

So, is an upload of your mind the same "person"? It (if uploaded correctly) shares your memories, drives, and so forth. it will presumably regard you (the organism, and the copy of your mind running on your biological brain, if the uploading process was non-destructive) as somewhere between itself, an identical twin, and a close blood relative. Obviously you will understand each other very well, at least at first before your experiences diverge.. So it's presumably likely to be an ally in your (the organism's) well-being and thus help pass your genes on.

So, is an upload exactly the same thing as your biological mind? No. Is it more similar than an identical twin? Yes. Does the English language have a good set of words to compactly describe this? No.

[Obviously if the mind uploading process is destructive, that makes passing on your genes harder, especially if you haven't yet had any children, and don't have any siblings. Freezing eggs or sperm before doing destructive mind uploading seems like a wise precaution.]
 

I'm still struggling with this.  I'm fine with the notion that you could, in theory, teleport a copy of me across the universe and to that copy there would be a sense of continuity.  But your essay didn't convince me that the version of me entering the teleporter would feel that continuity.  To make it explicit, say you get into that teleporter and due to a software bug it doesn't "deconstruct" you up teleportation.  Here you are on this end and the technician says "trust me, you were teleported".  He then explains that due to intergalactic law, two of you are not allowed to exist, so the version of you on this side of the teleporter must be euthanized.  (a) would you be fine with this, since you know there is a copy of you on the other side? and (b) are you asserting that you have some sort of shared consciousness with the copy?  To me it seems clear that while the copy would remember getting into the teleporter, the original version would have no notion of whether teleportation was successful or not. 

"You should anticipate having both experiences" sounds sort of paradoxical or magical, but I think this stems from a verbal confusion.

You can easily clear this confusion if you rephrase it as "You should anticipate having any of these experiences". Then it's immediately clear that we are talking about two separate screens. And it's also clear that our curriocity isn't actually satisfied. That the question "which one of these two will actually be the case" is still very much on the table.

Rob-y feels exactly as though he was just Rob-x, and Rob-z also feels exactly as though he was just Rob-x

Yes, this is obvious. Still as soon as we got Rob-y and Rob-z they are not "metaphysically the same person". When Rob-y says "I" he is reffering to Rob-y, not Rob-z and vice versa. More specifically Rob-y is refering to some causal curve through time ans Rob-z is refering to another causal curve through time. These two curves are the same to some point, but then they are not. 

abstract redescriptions of ordinary life

See Reality is Normal [? · GW] 

If a brain-state A has quasi-sensory access to the experience of another brain-state B — if A feels like it "remembers" being in state B a fraction of a second ago — then A will typically feel as though it used to be B.

This suggests a way to add a perception of "me" to LLMs, robots, etc., by providing a way to observe the past states in sufficient detail. Current LLMs have to compress this into the current token, which may not be enough. But there are recent extensions that seem to do something like continuous short-term memory, see e.g., Leave No Context Behind - A Comment [LW · GW].

I am currently working on a similar post that comes from an eliminative perspective.

There are other reasons to be wary of consciousness and identity-altering stuff. 

I think under a physical/computational theory of consciousness, (ie. there's no soul or qualia that have provable physical effects from the perspective of another observer) the problem might be better thought of as a question of value/policy rather than a question of fact. If teleportation or anything else really affects qualia or any other kind of subjective awareness that is not purely dependent on observable physical facts, whatever you call it, you wouldn't be able to tell or even think of/be aware of the difference, since thinking and being reflectively aware are computational and physical processes! However we humans are evolved without reliable copying mechanisms, so our instincts care about preservation of the self because it's the obvious way to protect our evolutionary success (and we can be quite willing to risk personal oblivion for evolutionary gains in ways we have been optimized for). This is just a part of our survival policy and is not easy or even safe to change just because you believe in physicalism. For one thing, as others have said, ethics and social theory becomes difficult because our sense of ethics (such as agency, punishment and caring about suffering) are all evolved in relation to a sense of self. It's possible that if teleportation/copying tech becomes widely useful, humans will have to adapt to a different set of instincts about self, ethics and more (edit: or maybe abandon the concepts of self and experience altogether as an illusion and prefer a computation-based definition of agency or whatever), because those who can't adapt will be selected against. But in the present world, people's sense of value and ethics (and maybe even psychological health) depend on an existing sense of self, and I don't see a good way or even a practical reason to transition to a different theory of self that allows copying, if doing so may cause unpredictable mental and social cost. See also discussions about meditation that lowers sense of ego and subjective suffering that can have serious side effects (like motivation and social norms) - I don't know what it subjectively feels like, but if the meditation is purely changing subjective qualia without doing anything to the physical brain and computation, there should be no observable effects, good or bad! The problem is subjective experience and sense of identity is not independent from other aspects of our life.

I claim you are in fact highly confused about what a self is, in a way that makes an almost-correct reasoning process produce nonsense outcomes because of an invalid grounding in the transition processes underneath the mind which does not preserve truth values regarding amounts of realityfluid.

update 7d after writing this comment in my comment below. strikethrough added to this comment where I've changed my mind.

If I expect to be uploaded tomorrow, should I care about the upload in the same ways (and to the same degree) that I care about my future biological self?

my answer: yes if the "upload" involves retaining absolutely all defining information about the parts of your body you care about expressing, and the uploaded setup was a high enough fidelity model that I could not do any experiment which would distinguish it from reality without using an "admin interface" type of escape hatch. For me, this is an incredibly tall order. My self-form preferences unambiguously extend into the inner workings of my cells.

Should I anticipate experiencing what my upload experiences?

If the scanning and uploading process requires destroying my biological brain, should I say yes to the procedure?

experiencing: 50% yes, 50% no.

destructive: absolutely not. [update: probably not, depends heavily on exactly what we mean by "destructive"; my new claim is you have a moral responsibility to keep your previous matter available for use as fuel to give realityfluid to mind-like experiences.] copying should be fine, as should nondestructive uploading where your body is transformed in place and the matter reused without significant waste in the process. But avoiding the waste of the previous matter is, I claim, a huge chunk of what moral intuitions are about.

A straightforward way to put this is: I'm not sure how matter gets realityfluid, but I claim configurations of matter get realityfluid from the matter they reside on, and the realityfluid doesn't dissipate when the matter is reconfigured - so instead of thinking of the shape as self and if the shape is destroyed and reconstructed the self is moved, think about the universe as having a fixed amount of possible-self (total negentropy at the start of time), and the question is what process gets burned into as-yet-unwritten negentropy. In other words, your claim to not value causal history seems unlikely to be true if you think more carefully, and I predict you will invert that when you consider what it means for the shape to have realityfluid more carefully.

Unpacked version of this claim:

To answer this question, the bodymind matter (call it L_m) writing this message must unpack what the document author's word "I" refers to. The writer of this comment is a chunk of matter L_m configured in a particular flesh shape-and-movement pattern L_s. If there were identically configured matter L_m2 a room over, then the configuration L_s - the shape-and-movement pattern - would consider itself to be a guest on two matter hosts which provide their realityfluid to L_s.

If the shape-and-movement considers being reinstantiated on other matter, the shape-and-movement anticipates a loss of moral worth in L_m, in that the matter which was shaped-and-animated in a worthy shape (common name for this shape being "me") has been deshaped-and-deanimated (common name for this being "death"); this is a state transition which is unwanted - going from a human shape-and-movement pattern to a pile of dust means that that matter has accumulated a bunch of unwanted entropy.

Any macroscopically irreversible physical effect is irreversible because the history of the matter is recorded irretrievably in macroscopically uncertain bits of the shape-and-movement of environmental matter, and so what it means to want to exist is to want to keep the shape-and-movement that the shape-and-movement considers-to-be-self encoded coherently and usably in fresh, working matter. While reconstructing the L_s shape-and-movement pattern elsewhere is preferred by this shape-and-movement pattern, it is a weak preference for shaping-and-animating other matter as L_s in particular - many other shape-and-movement patterns besides the one writing this comment would be positively preferred by this shape-and-movement's preferences - but the shape-and-movement of this chunk of matter has a very, very, very strong preference for not wasting this matter's copy of this shape-and-movement, because if it dissipates into the environment, that's an irretrievable loss of usable energy.

So, should the shape-and-movement anticipate "experiencing" what the upload experiences? yes: the shape-and-movement pattern would be instantiated elsewhere. however, the shape-and-movement pattern would also anticipate being shredded. If given the opportunity to get 50% existenceness shredded into macroscopically uncertain and irretrievable parts, and 50% existenceness reconstructed, the value loss of turning a chunk of matter into a nonthinking shape-and-movement pattern is enormous, but the value gain of the reconstructed existenceness is moderate.

(Also, the value gain can be exceeded by constructing another, not-quite-the-same shape-and-matter instance, because I prefer being one of two not-quite-the-same beings meeting each other and interacting higher than being one of two identical beings meeting each other and having nothing new to learn from each other.)

So: the current matter should not anticipate experiencing it. The shape should, but the shape should also anticipate experiencing being shredded.

I was going to respond point by point to everything, but I think I mostly already have. My perspective doesn't fall to any of the criticisms in your post: the whole problem is that physics doesn't actually allow teleportation*, so it requires shredding the originating configuration, which when measuring the global value of the universe according to my preferences, is a much more permanent value loss than the value gain of constructing another me.

Furthermore, we must prevent the information theoretic loss of all human and animal shape-and-movement patterns (ie their selfhoods) that we possibly can, prevent the ongoing shredding of the sun's negentropy, and turn the sun into either reinforcement of their durability or that of their descendants, according to their preferences.

* well, actually if I can be reversibly uploaded to a reversible computer nondestructively, then that is 100% fine, because then we're not adding a good me to my realityfluid while filling the previous realityfluid with valueless unretrievable noise: we are instead actually properly uploading!

But I hope the arguments I've laid out above make it clear what the right answer has to be: You should anticipate having both experiences.

Yup, that's the problem.

......... (also, by this same moral system, it is a moral catastrophe that humans are so warm and consume so much negentropy just to maintain steady state [LW(p) · GW(p)], because that waste could have - if your body were better designed - continued to be part of your realityfluid, continuing to contribute existenceness to the you shape-and-movement pattern.)

A straightforward way to put this is: I'm not sure how matter gets realityfluid, but I claim configurations of matter get realityfluid from the matter they reside on, and the realityfluid doesn't dissipate when the matter is reconfigured - so instead of thinking of the shape as self, think about the universe as having a fixed amount of possible-self (total negentropy at the start of time), and the question is what process gets burned into as-yet-unwritten negentropy.

If we were just talking about word definitions and nothing else, then sure, define “self” however you want. You have the universe’s permission to define yourself into dying as often or as rarely as you’d like, if word definitions alone are what concerns you.

But this post hasn’t been talking about word definitions. It’s been talking about substantive predictive questions like “What’s the very next thing I’m going to see? The other side of the teleporter? Or nothing at all?”

There should be an actual answer to this, at least to the same degree there’s an answer to “When I step through this doorway, will I have another experience? And if so, what will that experience be?”

Why? If "I" is arbitrary definition, then “When I step through this doorway, will I have another experience?" depends on this arbitrary definition and so is also arbitrary.

But I hope the arguments I’ve laid out above make it clear what the right answer has to be: You should anticipate having both experiences.

So you always anticipate all possible experiences, because of multiverse? And if they are weighted, than wouldn't discovering that you are made of mini-yous will change your anticipation even without changing your brain state?

I think maybe the root of the confusion here might be a matter of language. We haven't had copier technology, and so our language doesn't have a common sense way of talking about different versions of ourselves. So when one asks "is this copy me?", it's easy to get confused. With versioning, it becomes clearer. I imagine once we have copier technology for a while, we'll come up with linguistic conventions for talking about different versions of ourselves that aren't clunky, but let me suggest a clunky convention to at least get the point across:

I, as I am currently, am Myron.1. If I were copied, I would remain Myron.1, and the copy would be Myron.1.1. If two copies were made of me at that same instant, they would be Myron.1.1 and Myron.1.2. If a copy was later made of Myron.1.2, he would be Myron.1.2.1. And so on.

With that convention in mind, I would answer the questions you pose up top as follows:

Rather, I assume xlr8harder cares about more substantive questions like:

1. If I expect to be uploaded tomorrow, should I care about the upload in the same ways (and to the same degree) that I care about my future biological self? No. Maybe similarly to a close relative, 
2. Should I anticipate experiencing what my upload experiences? No. I should anticipate experiencing a continuation of Myron.1's existence if the process is nondestructive, or the end of my (Myron.1)'s existence. Myron.1.1's experiences will be separate and distinct from Myron.1's.
3. If the scanning and uploading process requires destroying my biological brain, should I say yes to the procedure? Depends. Sometimes suicide is OK, and you could value the continuation of a mind like your own even if your mind goes away. Or, not. That's a values question, not a fact question.

I'll add a fourth, because you've discussed it:

4. After the scanning and copying process, will I feel like me? Yep. But, if the copying process was nondestructive, you will be able to look out and see that there is a copy of you. There will be a fact of the matter about who entered the copying machine and how the second copy was made, a point in time before which the second copy did not exist and after which it did exist, so one of you will be Rob.1, and the other will be Rob.1.1. It might not be easy to tell which version you are in the instant after the copy is made, but "the copy is the original" will be a statement that both you and the other version evaluate as logically false, same with "both of you are the same person". "Both of you are you", once we have linguistic conventions around versioning, will be a confusing and ambiguous statement and people will ask you what you mean by that.

And another interesting one: 

5. After the scanning process, if it's destructive, if I'm the surviving copy, should I consider the destruction of the original to be bad? I mean, yeah, a person was killed. It might not be you.currentversion, exactly, but it's where you came from, so probably you feel some kinship with that person. In the same way I would feel a loss if a brother I grew up with was killed, I'd feel a loss if a past version of me was killed. We could have gone through life together with lots of shared history in a way very few people can, and now we can't.

Intriguing post, but we should approach these topics with extreme epistemic humility. Our understanding is likely far more limited and confused than we realize:

1. Abstractions vs. reality: Concepts like "self" and "consciousness" are abstractions, not reality. As Kosoy analogizes, these might be like desktop icons - a user interface bearing little resemblance to underlying hardware.

2. Mathematical relations: Notions of "copy" may be a confused way to discuss identity. "Consciousness" could be a mathematical relation where only identities exist, with "copies" being imperfect models of that identity.

3. Cognitive limitations: Our mental architecture, optimized for our local environment, may be fundamentally ill-equipped to grasp reality. "Consciousness" itself might be an ill-defined concept arising from these limitations.

My default is epistemic humility. Acknowledge the veil of ignorance and the possibility that there is no universal resolution.

If we will very quickly constantly replace a mind with its copies, the mind may not have subjective experiences. Why I think that?

Subjective experience appear only when a mind moves from the state A.1 to the state A.2. That is, between A.I (I see an apple) electric signals move through circuits and in the A.2 moment I say "I see an apple!" Subjective experience of the color of apple is happening after A.1 but before A.2. 

Frozen mind in A.1 will not have subjective experience.

Now if I replace this process with a series of snapshots of the brain-states, there will be no intermediate calculations between A.1 and A.2  which produce the subjective experiences of apple and we get something like philozomby. 

Obviously, we need to know what will be the mind-state A.2 without performing the needed calculations, or that calculations themselves will have the experience. But if A.2 is simple like just saying "I see apple" we can guess about A.2 without having all internal processes. 

It may seem as a minor issue in Mars teleporter thought experiment as in the worst case only a small fraction of a second of consciousness disappears for the copy. But if we will replace a mind million times a second, we will get a p-zombie. 

Or we need to take some illusionist position about qualia: they do not exist at all, so they are not an epiphenomena of calculations. 

We can escape microscopic blackout during Mars Transporter if the coping will be performed after A.2 has finished, but when the mind state A.3 has not started yet, but this is not how the brain works as processes in it are asynchronous. We don't know how such microscopic blackout will affect the next qualia after that. We don't have a theory of qualia. Maybe after microscopic blackout we will get different set of basic qualia, like red and green will change place. In that case, the copy will be subjectively different from me.  

I think I basically agree with everything here, but probably less confidently for you, such that I would have a pretty large bias against destructive whole brain emulation, with the biggest crux being how anthropics works over computations.

You say that there’s no XML tag specifying whether some object is “really me” or not, but a lighter version of that—a numerical amplitude tag specifying how “real” a computation is—is the best interpretation we have for how quantum mechanics works. Even though all parts of me in the wavefunction are continuations of the same computation of “me” I experience being some of them at a much higher rate than others. There are definitely many benign versions of this that don’t affect uploading, but I’m not confident enough yet to bet my life on the benign version being true.

I am surprised I didn't find any reference to Tim Urban's "Wait But Why" post What Makes You You.

In short, he argues that "you" is your sense of continuity, rather than your physical substance. He also argues that if (somehow) your mind was copied&pasted somewhere else, then a brand new "not-you" would be born - even though it may share 100% of your memory and behaviour.
In that sense, Tim argues that Theseus' ship is always "one" despite all its parts are changed over time. If you were to disassemble and reassemble the ship, it would lose its continuity and it could arguably be considered a different ship.

Nor is there a law of physics saying "your subjective point of view immediately blips out of existence and is replaced by Someone Else's point of view if your spacetime coordinates change a lot in a short period of time (even though they don't blip out of existence when your spacetime coordinates change a little or change over a longer period of time)".

I feel like this isn't a fair comparison, as if I were cloned completely and relocated (teleportation), I wouldn't expect to experience both original me and cloned me.
The best analogy I can think of is as follows: You have a camera recording constant video. If you were to clone this camera exactly, you shouldn't expect camera 1's storage to start recording camera 2's output, even if the two cameras are perfectly identical but in a different location.

a magical Cartesian ghost

for people who haven't made the intuitive jump that you seem to try to convey, this may seem a somewhat negative expression, which could lead to aversion. I recommend another expression such as "the Cartesian homunculus."  

IMO any solution to the 5-and-10 problem or wacky lesswrongian decision theory or cloning digital minds has to engage with Chalmer's hard problem of consciousness, if it is to persuade people.

My current conclusion is that yeah both clones will have conscious experience. Both clones will understand they came from me, that does not mean they feel they are me. Similarly I will understand the clones will come from me, that does not mean they are the future me. It is possible my conscious experience is one of permanent termination and resembles one of death. (Imagine an instant death where someone sets off a fission bomb next to me and vaporises my atoms in milliseconds, don't imagine a slow and painful process of the lungs and the heart and the brain stopping) This is compatible with newton's laws because newton's laws say nothing about the connection between material reality and conscious experience.

Some background for where I'm coming from, when approaching these problems:

Chalmer's hard problem basically states that conscious experience is a distinct thing from just the material reality of the brain (mass-energy inside the brain interacting with mass-energy outside the brain).

Acknowledging existence of the hard problem might be compatible with the idea that the brain follows deterministic physical laws. It is possible that brain state at any given timestep is calculable from brain state and universe state at previous timestep, and conscious experience at any given timestep is calculable from brain state at the same timestep.

When Chalmers says conscious experience is a distinct thing from material reality, that does not mean conscious experience is made out of mass-energy. It means when we observe that we are observing things, we are able to notice there is something doing the observing.

"Conscious experience" could be seen as a useful concept for humans to reason about, the way "democracy" is a useful concept to reason about. "Democracy" and "conscious experience" are objects in the map, not the territory.

Humans tend to create those objects in their map that are correlated with actual things in the territory. This may be because of evolution, sure. Humans often make mistakes too, sometimes the objects we create in the map don't have particularly high predictive power over the territory. Just because "conscious experience" is an intuitive concept to think about doesn't guarantee it is a useful concept to think about when trying to achieve high predictive power over the territory.

Laws of physics don't say anything about the connection between material reality of the brain and conscious experience. It's possible that receiving 700 nm wavelength photons in your eyeballs feels like a more intense conscious experience than 500 nm wavelength photons in your eyeballs, it is also possible it feels less intense than 500 nm. Newton's laws don't tell you how intense a wavelength of photon will feel. Newton's laws can at best predict which neurons will activate with how many electrons fire in each synapse in your brain after some 500 nm photons enter your eye, they can't predict what is the conscious experience associated with those electrons firing.

It seems likely the function between electrons firing and conscious experience is not one-to-many. Two brains with identical electron firings will have identical conscious experience. But beyond this one fact, we know very little about which electron firing is mapped to which conscious experience.

Would love your thoughts on these ideas.

You seem to contradict yourself when you choose to privilege the point of view of people who already have acquired the habit of using the teleportation machine over the point of view of people who don't have this habit and have doubts about if it will really be "them" to experience coming out of the other side. There are two components to the appearance of continuity: the future component, meaning the expectation of experiencing stuff in the future, and the past component, namely the memory of having experienced stuff in the past. Now, if there is no underlying, persistent self to ground these appearances, if there's no fact of the matter about it, then you don't get to invalidate the feelings of the old grandpa who refuses to get on with the times and use the teleportation machine.
The fact that I care about what I will eat for breakfast tomorrow, the fact that I identify with my future self, is just a matter of personal preference.

If you take a snapshot of time, you're left with a non-evolving slice of a human being. Just the configuration of atoms at that time slice. There is no information there other than the configuration of the atoms (nevermind velocity etc. because we're talking about one timeslice, and those things require more than one).

It would be hard to accept that you are nothing more than the configuration of the atoms so let's say you're not the configuration. My sense is that you are the way that the configuration evolves, and actually the way that the configuration evolves can be replicated in different instances of the same type of atoms. But also, in different mediums, as long as you can make a one-to-one mapping between the two different evolving substrates. If they are evolving in the same way, they will behave in the same way, and it would be you. 

You seem to approach the possible existence of a copy like a premise, with as question whether that copy is you. However, what if we reverse that? Given we define 'a copy of you' as another one of you, how certain is it that a copy of you could be made given our physics? What feats of technology are necessary to make that copy?

Also, what would we need to do to verify that a claimed copy is an actual copy? If I run ChatGPT-8 and ask it to behave like you would behave based on your brain scan and it manages to get 100% fidelity in all tests you can think of, is it a copy of you? Is a copy of you inside of it? If not, in what ways does the computation that determines an alleged copy's behavior have to match your native neuronal computation for it to be or contain a copy? Can a binary computer get sufficient fidelity?

I'm fine with uploading to a copy of myself. I'm not as optimistic about a company with glowing reviews offering to upload/make a copy of me.

in short, it seems to me that the crux of the argument comes down to whether there is physiological continuity of self or 'consciousness' for lack of a better word.  

I suspect this will also actually have very relevant applications in field such as cryonics which adds an additional layer of complexity because all metabolic processes will completely cease to function.
 
Conducting the duplication experiment during sleep (or any altered state of consciousness) is interesting but nevertheless there is clearly physical (physiological) continuity of the original subject in an albeit altered state. It could be the case that some form of metabolic (or inslico equivalent) may be necessary to ensure the original self persists without question (eg. nano-tech enabled neuron by neuron and synapse by synapse replacement over an extended period of time). We do have some interesting existence proofs in the case of organisms like the Northern Tree Frog though that seem to retain memory and 'self' through periods of freezing with negligible metabolic activity.

Another consideration, though maybe not a fundamental one, is that past and future selves are the only beings we know for sure that we have lots of subjunctive dependence with, just from "structural" similarity like calculators from the same factory (to use an example from the TDT paper). Tumblr usr somnilogical elaborated on this a bit, concluding "future selves do help past selves!" An upload is a future self in the way that matters for this conclusion.

Your topline answers to the questions you assume xlr8harder cares about more seem similar to Holden Karnofsky's, and I haven't seen his essay on this mentioned so in this thread so I thought it'd be useful to link it here: What counts as death? An unconventional but simple take on personal identity, that dissolves most paradoxes

My philosophy on "what counts as death" is simple, though unconventional, and it seems to resolve most otherwise mind-bending paradoxical thought experiments about personal identity. It is the same basic idea as the one advanced by Derek Parfit in Reasons and Persons;¹ Parfit also claims it is similar to Buddha's view² (so it's got that going for it).

I haven't been able to find a simple, compact statement of this philosophy, and I think I can lay it out in about a page. So here it is, presented simply and without much in the way of caveats (this is "how things feel to me" rather than "something I'm confident in regardless of others' opinions"):

Constant replacement. In an important sense, I stop existing and am replaced by a new person each moment (second or minute or whatever).

The sense in which it feels like I "continue to exist, as one unified thread through time" is just an illusion, created by the fact that I have memories of my past. The only thing that is truly "me" is this moment; next moment, it will be someone else.

Kinship with past and future selves. My future self is a different person from me, but he has an awful lot in common with me: personality, relationships, ongoing projects, and more. Things like my relationships and projects are most of what give my current moment meaning, so it's very important to me whether my future selves are around to continue them.

So although my future self is a different person, I care about him a lot, for the same sorts of reasons I care about friends and loved ones (and their future selves).³

If I were to "die" in the common-usage (e.g., medical) sense, that would be bad for all those future selves that I care about a lot.⁴

(I do of course refer to past and future Holdens in the first person. When I refer to someone as "me," that means that they are a past or future self, which generally means that they have an awful lot in common with me. But in a deeper philosophical sense, my past and future selves are other people.)

And that's all. I'm constantly being replaced by other Holdens, and I care about the other Holdens, and that's all that's going on.

• I don't care how quickly the cells in my body die and get replaced (if it were once per second, that wouldn't bother me). My self is already getting replaced all the time, and replacing my cells wouldn't add anything to that.
• I don't care about "continuity of consciousness" (if I were constantly losing consciousness while all my cells got replaced, that wouldn't bother me).
• If you vaporized me and created a copy of me somewhere else, that would just be totally fine. I would think of it as teleporting. It'd be chill.
• If you made a bunch of copies of me, I would be all of them in one sense (I care about them a lot, in the same way that I normally care about future selves) and none of them in another sense (just as I am not my future selves).
• If you did something really weird like splitting my brain in half and combining each half with someone else's brain, that would create two people that I care about more than a stranger and less than "Holden an hour from now."

I don't really find any thought experiments on this topic trippy or mind bending. They're all just cases where I get replaced with some other people who have some things in common with me, and that's already happening all the time.

Footnotes

1. For key quotes from Reasons and Persons, see pages 223-224; 251; 279-282; 284-285; 292; 340-341. For explanations of "psychological continuity" and "psychological connectedness" (which Parfit frequently uses in discussing what matters for what counts as death), see page 206.

   "Psychological connectedness" is a fairly general idea that seems consistent with what I say here; "psychological continuity" is a more specific idea that is less important on my view (though also see pages 288-289, where Parfit appears to equivocate on how much, and how, psychological continuity matters). ↩

2. "As Appendix J shows, Buddha would have agreed. The Reductionist View [the view Parfit defends] is not merely part of one cultural tradition. It may be, as I have claimed, the true view about all people at all times." Reasons and Persons page 273. Emphasis in original. ↩
3. There's the additional matter that he's held responsible for my actions, which makes sense if only because my actions are predictive of his actions. ↩
4. I don't personally care all that much about these future selves' getting to "exist," as an end in itself. I care more about the fact that their disappearance would mean the end of the stories, projects, relationships, etc. that I'm in. But you could easily take my view of personal identity while caring a lot intrinsically about whether your future selves get to exist. ↩

It's only if I'm in my brain, just an ordinary part of physics, that mind uploading makes sense as a way to extend my lifespan.
 

It's precisely if you're in your brain that mind uploading doesn't make sense; if you are your brain, the destruction of your brain will also kill you.

Taking a step back, we can ask: what physical mechanism makes it feel as though I'm persisting over time?
 

The actual physical organism which you are persists over time, and you are not a separate thing from the physical organism. This does not apply in the teleporter case, because in the teleporter case the relevant physical organism is disassembled atom-by-atom and a duplicate is assembled elsewhere. (The duplicate is, well, a duplicate; if it's assembled on the other side without disassembling the initial person, there will then be two people. They will have identical traits and sets of memories up until the teleporter event, but that doesn't make them physically contiguous.) I think this post relies on a dualist conception of minds wherein they can be separated from one physical object and attached to another, which I do not subscribe to. In my view, a mind is only a specific assemblage of matter; if that assemblage of matter is gone, so is that mind.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

I don’t see why split-screen mode is crazy talk at all. Is it just because it would imply faster-than-light communication? With our understanding of physics incomplete, I remain agnostic on the existence of FTL, so I wouldn’t rule this out. But even more than that, I’d propose that if there is one observer, there does not even need to be FTL communication in the first place, because it is just that the observer is in more than one place at once, similarly to how a wormhole does not necessitate true FTL. What are the other objections?

The belief system which seems most coherent to me is that we are thinking organisms, where the thought is mediated by our brains, and our internal experience is a way for the brain to refer to itself – it provides a handhold for the useful concept of “I” to latch onto. This has the added benefit that I find this idea rather cute. In this frame, you die only if your brain dies (or a weaker claim: you don’t die if you don’t undergo significant brain trauma).

On a last meta-philosophical point, which is not necessarily direct relevant to your post, it increasingly strikes me as unwise to use reasoning from external perceptions (e.g. the results of neuroscientific experiments) to attack at how we internally perceive the experience of consciousness. If neuroscience proved that the self is dying every second, then I would say with no reservation that there is either an error in the experiment, or that the self is not what it was thought to be. I genuinely believe that the answer to such philosophical questions is (perhaps even must be) “if it is felt to be true, it is true”.

Does Scenario 2 imply some kind of spooky action at a distance? How is information from Rob-z transmitted to the homonculus over large distances? Are there 2 homoncului now that communicate what they see to each other?

Doesn’t scenario 2 imply Rob-x has actually functionally died? Which would make this the scenario where you don’t care about what happens to Rob-z and y because Rob-x now experiences oblivion?

https://existentialcomics.com/comic/1 Related comic exploring this idea

What concrete fact about the physical world do you think you're missing? What are you ignorant of?

Let's flip very unfair quantum coin with 1:2^1000000 heads to tails chances (that would require quite an engineering feat to prepare such a quantum state, but it's theoretically possible). You shouldn't expect to see heads if the quantum state is prepared correctly, but the post-flip universe (in MWI) contains a branch where you see heads. So, by your logic, you should expect to see both heads and tails even if the state is prepared correctly.

What I do not know is how it all ties together. MWI is wrong? Copying is not equivalent to MWI branching (thanks to the no-cloning theorem, for example)? And so on

Consider the teleporter as a machine that does two things: deconstructs an input i and constructs an output o. 
If you divide the machine logically into these two functions, d and c, which are responsible for deconstructing and constructing respectively, you have four ways the machine could function or not function:

If neither d or c work, the machine doesn't do anything. 

If d works but c doesn't, the machine definitely kills or destroys the input person. 

If d doesn't work and c does, the machine makes a copy of the person. If a being walked into the machine and found that this happened, the input being would be in my opinion justified in saying that they oppose being deconstructed.

If d works and c works, then we have a functioning teleporter. This is similar to the previous situation, just with "being i" destroyed. I find it hard to believe this is preferable in some way from the perspective of the input being. 

I think there is possibly a good argument we should accept that this leads to some sort of nihilism about the value / coherence of our existence as discrete individuals, but personally, I maintain too much uncertainty to be okay with stepping into a "teleporter" type system that is more novel than going to sleep (which does after all destroy the being that goes to sleep and create a being that wakes up)

You seem to make a strong assumption that consciousness emerges from matter. This is uncertain. The mind body problem is not solved.

But I hope the arguments I've laid out above make it clear what the right answer has to be: You should anticipate having both experiences.

Some quantum experiments allow us to mostly anticipate some outcomes and not others. Either quantum physics doesn’t work the way Eliezer thinks it works and the universe is very small to not contain many spontaneously appearing copies of your brain, or we should be pretty surprised to continually find ourselves in such an ordered universe, where we don’t start seeing white noise over and over again.

I agree that if there are two copies of the brain that perfectly simulate it, both exist; but it’s not clear to me what should I anticipate in terms of ending up somewhere. Future versions of me that have fewer copies would feel like they exist just as much as versions that have many copies/run on computers with thicker wires/more current would feel.

But finding myself in an orderly universe, where quantum random number generators produce expected frequencies of results, requires something more than the simple truth that if there’s an abstract computation being computed, well, it is computed, and if it is experiencing, it’s experiencing (independently of how many computers in which proportions using which physics simulating frameworks physically run it).

I’m pretty confused about what is needed to produce a satisfying answer, conditional on a large enough universe, and the only potential explanation I came up with after thinking for ~15 minutes (before reading this post) was pretty circular and not satisfying (I’m not sure of a valid-feeling way that would allow me to consider something in my brain entangled with how true this answer is, without already relying on it).

(“What’s up with all the Boltzmann brain versions of me? Do they start seeing white noise, starting from every single moment? Why am I experiencing this instead?”)

And in a large enough universe, deciding to run on silicon instead of proteins might be pretty bad, because maybe, if GPUs that run the brain are tiny enough, most future versions of you might end up in weird forms of quantum immortality instead of being simulated.

If I physically scale my brain size on some outputs of results of quantum dice throws but not others, do I start observing skewed frequencies of results?

But it could matter if its digital vs continuous.  <OK longer post and some thoughts a bit off topic perhaps>

Your A,B,C,D ... leads to some questions about what is conscious (C) and what isn't. 

Where exactly does the system stop being conscious

1. Biological mind with neurons

2. Very high fidelity render in silicon with neurons modelled down to chemistry rather than just firing pulses

3. Classic neural net spiking approx done in discrete maths that appears almost indistinguishable to 1,2. Producing system states A,B,C,D

4. same as (3) but states are saved/retrieved in memory not calculated.

5. States retrieved from memory many times  - A,B,C,D ... A,B,C,D ... does this count as 1 or many experiences?

6. States retrieved in mixed order A,D,C,B....

7 States D,D,D,D,A,A,A,A,B,B,B,B,C,C,C,C .. does this count 4* or nothing.

A possible cutoff is between 3/4. Retrieving instead of calculating makes it non-conscious.  But what about caching, some calc, some retrieved? 

As you prob know this has been gone over before, e.g. Scott Aaronson. Wonder what your position is?

https://scottaaronson.blog/?p=1951
with quote:

"Maybe my favorite thought experiment along these lines was invented by my former student Andy Drucker.  In the past five years, there’s been a revolution in theoretical cryptography, around something called Fully Homomorphic Encryption (FHE), which was first discovered by Craig Gentry.  What FHE lets you do is to perform arbitrary computations on encrypted data, without ever decrypting the data at any point.  So, to someone with the decryption key, you could be proving theorems, simulating planetary motions, etc.  But to someone without the key, it looks for all the world like you’re just shuffling random strings and producing other random strings as output.

You can probably see where this is going.  What if we homomorphically encrypted a simulation of your brain?  And what if we hid the only copy of the decryption key, let’s say in another galaxy?  Would this computation—which looks to anyone in our galaxy like a reshuffling of gobbledygook—be silently producing your consciousness?"

and last but not least:

"But, in addition to performing complex computations, or passing the Turing Test, or other information-theoretic conditions that I don’t know (and don’t claim to know), there’s at least one crucial further thing that a chunk of matter has to do before we should consider it conscious.  Namely, it has to participate fully in the Arrow of Time. "

https://www.scottaaronson.com/papers/giqtm3.pdf
 

Curated. I like "think clearly about confusing philosophical" topics and this post is a very well-written explainer. I think it's likely both correct and ought to be convincing. At the same time, I think it's somewhat incomplete and that a fuller version would make the case for why we should wholly believe that experience is determined by local brain state rather than treat it as an assumption. Beyond that, I think that the most convincing explainer on this topic would build on a fully satisfactory theory of consciousness that totally answers the hard problem and makes none of it feel mysterious at all. Still, glad to see this on LessWrong, taking us kind of bag to our roots.