LessWrong 2.0 Reader

COMMENT THREAD

If you comment anywhere other than here, Nathan will delete your comment.

I think you're imagining that we modify the shrink-and-reposition functions each iteration, lowering their scope? I. e., that if we picked the topmost triangle for the first iteration, then in iteration two we pick one of the three sub-triangles making up the topmost triangle, rather than choosing one of the "highest-level" sub-triangles?

Something like this:

If we did it this way, then yes, we'd eventually end up jumping around an infinitesimally small area. But that's not how it works, we always pick one of the highest-level sub-triangles:

Note also that we take in the "global" coordinates of the point we shrink-and-reposition (i. e., its position within the whole triangle), rather than its "local" coordinates (i. e., position within the sub-triangle to which it was copied).

Here's a (slightly botched?) video explanation.

I suppose you’re right.

Maybe ‘death’ was a poor example as it inherently leads us to a state of relief from discomfort. If we instead take the example of ‘extreme torture’, then it makes more sense to compare the two.

The ‘discomfort’ I was referring to was more from a ‘physical sensation’ perspective rather than any second-order effects.

Imagine these experiences occur in a closed system with no influence on the outside world. Each person has been brought into existence by some higher power specifically for the purposes of this experiment. They have no family, no friends, and are genetically identical.

Imagine 1,000,000,000 participants with a single rational observer. The observer is forced by the higher power to make a choice – so some method of comparison is required.

Would it make more sense for the observer to choose for every single one of the participants to be burdened with a speck of dust in their eye, or for one single participant to be subjected to ‘extreme torture’?

Is there any point where increasing the number changes your mind?

For me it doesn’t matter how many participants there are – the option of torture should never be taken.

The ‘logarithmic stacking theory’ allows this to work mathematically, while a linear model does not.

Two notes: 

1. I think the title is a somewhat obscure pun referencing the old saying that Stanford was the "Hardvard of the West". If one is not familiar with that saying, I guess some of the nuance is lost in the choice of term. (I personally had never heard that saying before recently, and I'm not even quite sure I'm referencing the right "X of the West" pun)
2. habryka did have a call with Nick Bostrom a few weeks back, to discuss his idea for an "FHI of the West", and I'm quite confident he referred to it with that phrase on the call, too. Far as I'm aware Nick didn't particularly react to it with more than a bit humor. 

Thanks for pointing this out, I missed a word. I have added it now.

I don't think this phenomenon is just related to the training data alone because in RLLMv3, the " Leilan" glitch mode persisted while " petertodd" became entirely unrelated to bitcoin. It's like some glitch tokens can be affected by the amount of re-training and some aren't. I believe that there is something much deeper is happening here, an architectural flaw that might be related to the token selection/construction process.

Guys, social reality is one if not the cause of the self:

Robin Hanson:

And the part of our minds we most fear losing control of is: our deep values.

PubMed: The essential moral self

folk notions of personal identity are largely informed by the mental faculties affecting social relationships, with a particularly keen focus on moral traits.

The solution is here [AF · GW]. In a nutshell, naive MWI is wrong, not all Everett branches coexist, but a lot of Everett branches do coexist s.t. with high probability all of them display expected frequencies.

My model is that the concept of "morality" is a fiction which has 4 generators that are real:

• People have empathy, which means they intrinsically care about other people (and sufficiently person-like entities), but, mostly about those in their social vicinity [LW(p) · GW(p)]. Also, different people have different strength of empathy, a minority might have virtually none.
• Superrational cooperation is something that people understand intuitively to some degree. Obviously, a minority of people understand it on System 2 level as well.
• There is something virtue-ethics-like which I find in my own preferences, along the lines of "some things I would prefer not to do, not because of their consequences, but because I don't want to be the kind of person who would do that". However, I expect different people to differ in this regard.
• The cultural standards of morality, which it might be selfishly beneficial to go along with, including lying to yourself that you're doing it for non-selfish reasons. Which, as you say, becomes irrelevant once you secure enough power. This is a sort of self-deception which people are intuitively skilled at.

If I expect to be uploaded tomorrow, should I care about the upload in the same ways (and to the same degree) that I care about my future biological self?

my answer: yes if the "upload" involves retaining absolutely all defining information about the parts of your body you care about expressing, and the uploaded setup was a high enough fidelity model that I could not do any experiment which would distinguish it from reality without using an "admin interface" type of escape hatch. For me, this is an incredibly tall order. My self-form preferences unambiguously extend into the inner workings of my cells.

Should I anticipate experiencing what my upload experiences?

If the scanning and uploading process requires destroying my biological brain, should I say yes to the procedure?

To answer this question, the bodymind matter writing this message must unpack what the document author's word "I" refers to. The writer of this comment is a chunk of matter configured in a particular flesh shape. If there were identically configured matter a room over, then the configuration - the shape - would consider itself to be a guest on two hosts. If the shape-and-movement considers being reinstantiated elsewhere, the shape-and-movement anticipates a loss of moral worth, in that the matter which was shaped-and-animated in a worthy shape (common name for this shape being "me") has been deshaped-and-deanimated; this is a state transition which is unwanted - going from a human shape-and-movement pattern to a pile of dust means that that matter has accumulated a bunch of unwanted entropy. Any macroscopically irreversible physical effect is irreversible because the history of the matter is recorded irretrievably in macroscopically uncertain bits of the shape-and-movement of environmental matter, and so what it means to want to exist is to want to keep the shape-and-movement that the shape-and-movement considers self encoded coherently and usably in fresh, working matter. While reconstructing that shape-and-movement pattern elsewhere is preferred by this shape-and-movement pattern, it is a weak preference for shaping-and-animating other matter - many other shape-and-movement patterns besides the one writing this comment would be positively preferred by this shape-and-movement's preferences - but the shape-and-movement of this chunk of matter has a very, very, very strong preference for not wasting this matter's copy of this shape-and-movement.

So, should the shape-and-movement anticipate "experiencing" what the upload experiences? yes: the shape-and-movement pattern would be instantiated elsewhere. however, the shape-and-movement pattern would also anticipate being shredded. If given the opportunity to get 50% existenceness shredded into macroscopically uncertain and irretrievable parts, and 50% existenceness reconstructed, the value loss of turning a chunk of matter into a nonthinking shape-and-movement pattern is enormous, but the value gain of the reconstructed existenceness is moderate.

Also, the value gain can be exceeded by constructing another, not-quite-the-same shape-and-matter instance.

So: the current matter should not anticipate experiencing it. The shape should, but the shape should also anticipate experiencing being shredded.

Furthermore, we must prevent the loss of all humans, prevent the ongoing shredding of the sun's negentropy, and turn the sun into their descendants.

I was going to respond point by point to everything, but I think I mostly already have. My perspective doesn't fall to any of the criticisms in your post: the whole problem is that physics doesn't actually allow teleportation\*, so it requires shredding the originating configuration, which when measuring the global value of the universe according to my preferences, is a much more permanent value loss than the value gain of constructing another me. and this line:

\* well, actually if I can be reversibly uploaded to a reversible computer nondestructively, then that is 100% fine, because then we're not adding a good me to my realityfluid while filling the previous realityfluid with valueless unretrievable noise: we are instead actually properly uploading!

But I hope the arguments I've laid out above make it clear what the right answer has to be: You should anticipate having both experiences.

Yup, that's the problem.

......... (also, by this same moral system, it is a moral catastrophe that humans are so warm and consume so much negentropy just to maintain steady state [LW(p) · GW(p)], because that waste could have - if your body were better designed - continued to be part of your realityfluid, continuing to contribute existenceness to the you shape-and-movement pattern.)