Mahatma Armstrong: CEVed to death.

I show the sequence to the AI and say, "CEV shouldn't work like this - this is a negative example of CEV."

"Example registered," says the young AI. "Supplementary query: Identify first forbidden transition, state general rule prohibiting it?"

Wow, Yudkowsky's CEV runs into Yudkowsky's Löb problem! That's cute :-)

Another angle is that a lot of your values are based in experience, or at least I hope so, and there's a limit to how much you can extrapolate about experience you haven't had.

This is based in the idea that business plans are of limited value because businesses find out about new opportunities because of the work the business is doing.

I think that the hope is that by bolstering your intelligence with each successive iteration (honoring the full letter of CEV) you would be thinking more precisely, and more easily spot any errors or fuzziness in your own reasoning. For example, Armstrong 5's reasoning is incoherent in several ways, most glaringly that he would "value all worthwhile things" which could be written without distortion as "value all valuable things" which is obviously circular. I doubt that Armstrong 1 would make this mistake, so Armstrong 5 should be even more likely to spot it.

Also, I thought that implicit in the "Coherent" part of CEV was the idea that Armstrong 1 would have to in some sense sign off on Armstrong 2 before going further. Maybe give Armstrong 2 a chance to plead his case to Armstrong 1; if his arguments for moral revision are accepted, proceed to Armstrong 3; at no point are you required to delete Armstrong 1, nor should you, because he is the only possible basis for maintaining coherence.

If my opinion on animal rights differs from my extrapolated opinion, I tend to think it's because the person I want to be knows something that I do not. Perhaps he has a better understanding of what sentience is.

If my extrapolated opinion is just chaotic value drift, then my problem is that I'm extrapolating it wrong. Under that idea, you're giving insight into a likely error mode of extrapolation.

I don't want my current values to be implemented, insomuch as my current values are based on my current understanding. Then again, if I want my extrapolated values to be implemented, isn't that just another way of saying that my extrapolated values are my current values?

When reading the post, I immediately pictured a white guy in the 18th century saying something like "Of course we shouldn't give blacks rights based on the reasons you suggested, that would imply that the next thing would be giving rights to women!" (and the rejection of the latter being motivated by a false map of the world).

I see no reason why Armstrong-1 needs to like or at least accept the conclusions of Armstrong-n, assuming he is not yet aware of all the inferential steps. The premise of CEV, as I understand it, is that each step by itself is sound and in accordance with the terminal value(s) of Armstrong-1. Whether this works and whether we can check if it worked is a different question.

Even if you think changing one's stated (as opposed to terminal) utility function is always done based on comparing outcomes for intuitiveness and consistency, a position I'm sympathetic towards, it would be irrational for Armstrong-1 to reject Armstrong-n's values just because they lead to very counterintuitive conclusions. The reason being that Armstrong-1 may not be aware, due to ignorance or mistakes in reasoning, that his own values imply conclusions that are even more absurd.

In general, I think people on LW are way too quick to declare something to be their terminal value (which is of course somewhat of a self-fulfilling prophecy).

Of course the sequence presented seems ridiculous, and it were the actual output of CEV, I would be virtually certain that something went wrong. However, that is based on the reasoning outlined at each step, not due to the final conclusion. All I'm saying is that Armstrong-1 has no vote on the final output before actually having gone through all the arguments. Not even if the outcome would be something counterintuitive like negative utilitarianism. (BTW, I have yet to hear a take on population ethics that doesn't include conclusions that are highly counterintuitive!)

I don't see how, because the barriers aren't clearly defined, they become irrelevant. There might not be a specific point where a mind is sentient or not, but that doesn't mean all living things are equally sentient (Fallacy of Grey).

I think Armstrong 4, rather than make his consideration for all living things uniform, would make himself smarter and try to find an alternate method to determine how much each living creature should be valued in his utility function.

I want to be more altruistic

This kind of moral tailbiting strikes me as a moral perversion. Wanting to be "more moral" is the more general perversion.

One may have altruistic wants. You see people, you hear of people who could use your help, and you want to help them. You're not wanting to "be more altruistic", you want to make a change to their situation for the better. The evaluation isn't about you; it's about them and their situation.

One may even evaluate yourself as the cause of the improved situation, and thereby feel pride.

But a morality of "being more moral" becomes a content free solipsism evaluating how much you're preoccupied with your evaluation of how moral you are. How much moral preoccupation did I exhibit today? Lots? Good boy!

There's a deeper question here: ideally, we would like our CEV to make choices for us that aren't our choices. We would like our CEV to give us the potential for growth, and not to burden us with a powerful optimization engine driven by our childish foolishness.

One obvious way to solve the problem you raise is to treat 'modifying your current value approximation'' as an object-level action by the AI, and one that requires it to compute your current EV - meaning that, if the logical consequences of the change (including all the future changes that the AI predicts will result from that change) don't look palatable to you, the AI won't make the first change. In other words, the AI will never assign you a value set that you find objectionable right now. This is safe in some sense, but not ideal. The profoundly racist will never accept a version of their values which, because of its exposure to more data and fewer cognitive biases, isn't racist. Ditto for the devoutly religious. This model of CEV doesn't offer the opportunity for growth.

It might be wise to compromise by locking the maximum number of edges in the graph between you and your EV to some small number, like two or three - a small enough number that value drift can't take you somewhere horrifying, but not so tightly bound up that things can never change. If your CEV says it's okay under this schema, then you can increase or decrease that number later.

I'm with you up to 6. Having a terminal value on everything does not mean that the final consistent evaluation is uniform over everything, because instrumental values come into play -- some values cancel out and some add up. But it does mean that you have justifications to make before you start destroying stuff.

My problem with CEV is the arbitrariness of what it means to "know more". My brain cannot hold all the knowledge about the universe, so the AI has to somehow choose what information to impart and in what order, and this would significantly influence the outcome. E.g. maybe hearing 100 heartwarming stories would make me care more about others, while hearing 100 stories about people being bastards to each other would make me care less, hearing all evidence supporting some political theory would sway me towards it, et cetera.

The difference between EV and reflexive equilibria seems to be that in the former, lower level desires are changed to match higher order desires, whereas the latter involves changing both higher and lower order desires till they are consistent. As such, it involves much less changing of lower order desires, and much less overall changing of desires, as you have more degrees of freedom to work with.

There is still some risk - as you say, maybe there are few attractors - but overall RE seems a much more conservative approach.

It's not clear to me what you mean by value. To say that something has value is to say that is more valuable than other things. This is why at the end of your progression valuing everything becomes equivalent to valuing nothing.

This is true for all definitions. If there is nothing that is not valuable, then the term "value" becomes semantically empty.

This has nothing inherently to do with altruism. Every agent makes value judgments, and value rather than being treated as a binary, is typically treated as a real number. The agent is thus free to choose between any number of futures and the infinitude of real numbers assures that those futures remain distinct, so Armstrong 7 should never be in turmoil. Additionally, this is more or less how human operate now. A pretty rock or an animal may be valuable, but no one is confused as to whether or not that means they are equivalent in worth to a human.

Interestingly, you use, and then deconstruct the binaries of sentient/nonsentient, living/nonliving, etc, but you don't apply that same tool to the dichotomy of altruistic toward/not altruistic toward.

I think the "coherent" in CEV can apply to different versions of your extrapolated self, so that the only things in your CEV are things that you pretty much always want under any "reasonable" extrapolation. E.g., getting rid of involuntary human death is almost definitely in my CEV, but it gets fuzzier as we go down the food chain (or rather the brain chain).

This seems analogous to the problem of marketplace investments. If you have good evidence that investment X is going to be worth twice as much tomorrow, your preference becomes to own as much as possible. But if your investment is high enough to impact the scarcity level, it becomes twice as expensive today, which could occlude or even negate the reasons it was going to be worth twice as much tomorrow in the first place. With that information in mind, your preference regarding how much to invest is different.

An interesting problem with CEV is demonstrated in chapter 5 "On the Rationality of Preferences" of Hilary Putnam "The Collapse of the Fact/Value Dichotomy and Other Essays". The problem is that a person might assign value to that a choice of a preference, underdetermined at a given time, being of her own free will.

This seems related to the problem that human preferences are contradictory. As I suggested here, this may be kept in place by evolution to keep us working and enhancing our genetic fitness - meaning that human preferences would not only be incoherent, but would need to be incoherent to capture human value.

So, how would one approach creating a mostly-usable extrapolation of inconsistent starting values?

He decides that he should solve this by valuing all worthwhile things

How exactly does one go about deciding a tautology?

In this context, what does "more altruistic" (as in 1) mean? Does it mean that you want to change your beliefs about what is right to do, or that given your current beliefs, you want to give more to charity (but, for whatever reason, find it difficult to do so)? If it's the former, it seems contradictory - it's saying "it's right for me to be more altruistic than I currently am, but I don't believe it". If it's the latter, the transition between 1 and 2 wouldn't happen, because your belief about the optimum level of altruism either wouldn't change (if you are currently correct about what the optimal amount of altruism is for you) or it would change in a way that would be appropriate based on new information (maybe giving more to charity is easier once you get started). I can see your estimation of your optimum level of altruism changing based on new information, but I don't see how it would lead to a transition such as that between 1 and 2. Even if charity is very easy and very enjoyable, it doesn't follow that you should value all humans equally.

I don't see how destroying all life follows logically from valuing all things. It is true that life destroys some things. However, it seems to me that the process of life - evolution and the production of novel genetic diversity - is a valuable thing in and of itself, definitely worth preserving. Not just for romantic notions of peace with nature, but for a very rational reason: the enormous amount of hard-obtained information present in genes that would be irreversibly lost if life were destroyed. By 'irreversibly' I mean it would take billions of years to evolve that information all over again.

So it makes much more sense to contain life (i.e. confine it to the planet of origin and prevent it from spreading, minimizing damage to other things) rather than destroying it outright. Ultimately, a superintelligence will understand that everything is a tradeoff and that you can't have your cake and it eat too.

You're making a conclusion based on the false assumption that so many sci-fi writers have relished in: that human morals trump superintelligence i.e. that a superintelligence would be stupid. In reality, a superintelligence will probably make a far better choice than you or I can, given the circumstances.