Human Minds are Fragile

That post discusses two big ideas, one is putting a human in a box and building a model of their input/output behavior as "the simplest model consistent with the observed input/output behavior." Nick Bostrom calls this the "crypt," which is not a very flattering name but I have no alternative. I think it has been mostly superseded by this kind of thing (and more explicitly, here, but realistically the box part was never necessary).

The other part is probably more important, but less colorful; extrapolate by actually seeing what a person would do in a particular "favorable" environment. I have been calling this "explicit" extrapolation.

I'm sorry to never name this. I think I can be (partly) defended because the actual details have changed so much, and it's not clear exactly what you would want to refer to.

I'm skeptical about the relevance of the fragility of minds here. Yes, if you mess up your simulation slightly it will start to make bad predictions. But that seems to make it easier to specify a person precisely, not harder. The differences in observations allow someone to quickly rule out alternative models by observations of people. Indeed, the way in which a human brain is physically represented makes little difference to the kind of predictions someone would make. As another commenter pointed out, if you randomly flip some bits in a computer it will not do anything good. But that has little relevance to your predictions of a computer, unless you expect some bits to get randomly flipped.

(A general point is also that you shouldn't be specifying values either through a definition of a human mind, or through language. You should be using these weak descriptions to give an indirect definition of value, via an edict like "do whatever it is that people want" and recommending actions like "when the time comes, ask people." (This indirection may be done in hypothetical simulation, for the explicit extrapolation approach.) These targets are much less fragile. So I tend to think that the fragility of value doesn't bear much on the feasibility of these approaches.

Just because a lot of people with amygdala damage are serial killers doesn't mean that everyone or most of the people with amygdala damage are.

I don't think the evidence to which you point proves a claim as strong as the one you make in the post.

Magnetic though?

But, but... people who want to do things that they feel are morally reprehensible are not reflectively consistent, and certainly people who imagine things like imminent end of the world. I thought CEV-type approaches only rely on reflectively consistent people to learn from.

I wanted to upvote, but you're currently an appropriate +5 :)