Value Deathism

Anyone who predicts that some decision may result in the world being optimized according to something other than their own values, and is okay with that, is probably not thinking about terminal values. More likely, they're thinking that humanity (or its successor) will clarify its terminal values and/or get better at reasoning from them to instrumental values to concrete decisions, and that their understanding of their own values will follow that. Of course, when people are considering whether it's a good idea to create a certain kind of mind, that kind of thinking probably means they're presuming that Friendliness comes mostly automatically. It's hard for the idea of an agent with different terminal values to really sink in; I've had a little bit of experience with trying to explain to people the idea of minds with really fundamentally different values, and they still often try to understand it in terms of justifications that are compelling (or at least comprehensible) to them personally. Like, imagining that a paperclip maximizer is just like a quirky highly-intelligent human who happens to love paperclips, or is under the mistaken impression that maximizing paperclips is the right thing could do and could be talked out of it by the right arguments. I think Ben and Robin ("Human value ... will continue to [evolve]", "I think it is ok (if not ideal) if our descendants' values deviate from ours") are thinking as though AI-aided value loss would be similar to the gradual refinement of instrumental values that takes place within societies consisting of largely-similar human brains (the kind of refinement that we can anticipate in advance and expect we'll be okay with), rather than something that could result in powerful minds that actually don't care about morality.

And I feel like anyone who really has internalized the idea that minds are allowed to fundamentally care about completely different things that we do, and still thinks they're okay with that actually happening, probably just haven't taken five minutes to think creatively about what kinds of terrible worlds or non-worlds could be created as a result of powerfully optimizing for a value system based on our present muddled values plus just a little bit of drift.

I suppose what remains are the people who don't buy the generalized idea of optimization processes as a superset of person-like minds in the first place, with really powerful optimization processes being another subset. Would Ben be in that group? Some of his statements (e.g. "It's possible that with sufficient real-world intelligence tends to come a sense of connectedness with the universe that militates against squashing other sentiences. But I'm not terribly certain of this, any more than I'm terribly certain of its opposite." (still implying a privileged 50-50 credence in this unsupported idea)) do suggest that he is expecting AGIs to automatically be people in some sense.

(I do think that "Value Deathism" gives the wrong impression of what this post is about. Something like "Value Loss Escapism" might be better; the analogy to deathism seems too much like a surface analogy of minor relevance. I'm not convinced that the tendency to believe that value loss is illusory or desirable is caused by the same thought processes that cause those beliefs about death. More likely, most people who try to think about AI ethics are going to be genuinely really confused about it for a while or forever, whereas "is death okay/good?" is not a confusing question.)

Change in values of the future agents, however sudden of gradual, means that the Future (the whole freackin' Future!) won't be optimized according to our values, won't be anywhere as good as it could've been otherwise.

That really depends of what you mean by "our values":

1) The values of modern, western, educated humans? (as opposed to those of the ancient Greek, or of Confucius, or of medieval Islam), or

2) The "core" human values common to all human civilizations so far? ("stabbing someone who just saved your life is a bit of a dick move", "It would be a shame if humanity was exterminated in order to pave the universe with paperclips", etc.)

Both of those are quite fuzzy and I would find it hard to describe either of them precisely enough that even a computer could understand them.

When Eliezer talks of Friendly AI having human value, I think he's mostly talking about the second set (in The Psychological Unity of Mankind. But when Ben or Robin talk about how it isn't such a big deal if values change, because they've already changed in the past, they seem to be referring to the first kind of value.

I would agree with Ben and Robin that it isn't a big deal if our descendents (or Ems or AIs) have values that are at odds with our current, western, values (because they might be "wrong", some might be instrumental values we confuse for terminal values, etc.); but I wouldn't extend that to changes in "fundamental human values".

So I don't think "Ben and Robin are OK with a future without our values" is a good way of phrasing it. The question is more whether there is such a thing as fundamental human values (or is everything cultural?), whether it's easy to hit those in mind-space, etc.

Counterpoints: The Psychological Diversity of Mankind, Human values differ as much as values can differ.

How about ruler-of-the-universe deathism? Wouldn't it be great if I were sore undisputed ruler of the universe? And yet thinking that rather unlikely, I don't even try to achieve it. I even think trying to achieve it would be counter-productive. How freackin' defeatist is that?

Voted up. However, I disagree with "it's not OK". Everything is always OK. OK is a feature of the map. From a psychological perspective, that's important. If an OK state of the map can't be generated by changing the territory, it WILL be generated by cheating and directly manipulating the map.

That said, we have preferences, rank orderings of outcomes. The value of futures with our values is high.

OK, fine, literally speaking, value drift is bad.

But if I live to see the Future, then my values will predictably be updated based on future events, and it is part of my current value system that they do so. I affirmatively value future decisions being made by entities that have taken a look at what the future is actually like and reflected on the data they gain.

Why should this change if it turns out that I don't live to see the future? I would like future-me to be one of the entities that help make future decisions, but failing that, my second-best option is to have future-others make those decisions. I don't want present-me's long dead hand to go around making stupid decisions for other people.

Even if future people's values seem bizarre to present-me, that just reflects the fact that it takes a while to process the data that leads to shifts in value. Presumably, if you show me the values and the cultural/demographic landscape of 2050 now, then by 2020 I'll endorse many of them and be chewing on most of the rest. Why arbitrarily privilege my initial disgust over my considered reaction?

Saying a certain future is "not ok", and saying gradual value drift is "business as usual", are both value judgments. I don't understand why you dismiss one of them but not the other, and call it "courageous and rational".

And it is difficult to make it so that the future is optimized: to stop uncontrolled "evolution" of value (value drift) or recover more of astronomical waste.

I still find it shocking and terrifying every time someone compares the morphing of human values with the death of the universe. Even though I saw another FAI-inspired person do it yesterday.

If all intelligent life held your view about the importance of their own values, then life in the universe would be doomed. The outcome of that view is that intelligent life greatly increases its acceptable ratio of (risk of destroying all life) / (chance of preserving its value system). (This is especially a problem when there are multiple intelligent beings with value systems that differ, as there already are.) The fragility of life in the long term means we can't afford that luxury. It will be hard enough to avoid the death of the universe even if we all cooperate.

Publicly stating the view that you cannot value the existence of anything but agents implementing your own values (even after your death), makes cooperation very difficult. It's easier to cooperate with someone who is willing to compromise.

Someone will complain that I can't value anything but my own values; that's a logical impossibility. But notice that I never said I value anything but my own values! The trick is that there's a difference between acting to maximize your values, versus going meta, and saying that you must value the presence of your values in others. There is no law of nature saying that a utility function U must place a high value on propagating U to other agents. In fact, there are many known utility functions U that would place a negative value on propagating their values to other agents! And it would be interesting if human utility functions are represented in a manner that is even capable of self-reference.

(The notion that your utility function says you must propagate your utility function only makes sense if you assume you will become a singleton.)

Even if you insist that you must propagate your utility function (or you plan on becoming a singleton), you should be able to reach a reflective equilibrium, and realize that attempting to force your values on the universe will result in a dead universe; and accept a compromise with other value systems. Avoiding that compromise, not saving humanity, is, I think, the most plausible reason for the FAI+CEV program.

But, falling short of reaching that equilibrium, I would like it if all the CEVers would at least stop talking about Human Values as if they were a single atomic package. If you model values as being like genes, and say that you want to transmit your values in the same way that an organism wants to transmit its genes (which it doesn't, by the way; and it is in exactly this way that the Yudkowskian attachment to values makes no sense - insisting that you must propagate your values because that's what your values want is exactly like insisting you must propagate your genes because that's what your genes want; it is forgetting that you are a computational system, and positing a value homunculus who looks at you with your own eyes to figure out what you should do); then that would at least allow for the possibility of value-altruism in a way isomorphic to kin selection (sometimes sacrificing your complete value package in order to preserve a set of related value packages).

I wish I could save up all my downvotes for the year, and apply them all to this post (if I could without thus having to apply them to its author - I don't want to get personal about it); for this is the single most dangerous idea in the LessWrong memespace; the one thing rotten at the core of the entire FAI/CEV project as conceived of here.

Goertzel: Human value has evolved and morphed over time and will continue to do so. It already takes multiple different forms. It will likely evolve in future in coordination with AGI and other technology.

Agree, but the multiple different current forms of human values are the source of much conflict.

Hanson: Like Ben, I think it is ok (if not ideal) if our descendants' values deviate from ours, as ours have from our ancestors.

Agree again. And in honor of Robin's profession, I will point out that the multiple current forms of human values are the driving force causing trade, and almost all other economic activity.

Nesov: Change in values of the future agents, however sudden or gradual, means that the Future (the whole freackin' Future!) won't be optimized according to our values, won't be anywhere as good as it could've been otherwise. ... Regardless of difficulty of the challenge, it's NOT OK to lose the Future.

Strongly disagree. The future is not ours to lose. A growing population of enfranchised agents is going to be sharing that future with us. We need to discount our own interest in that future for all kinds of reasons in order to achieve some kind of economic sanity. We need to discount because:

• We really do care more about the short-term future than the distant future.
• We have better control over the short-term future than the distant future.
• We expect our values to change. Change can be good. It would be insane to attempt to determine the distant future now. Better to defer decisions about the distant future until later, when that future eventually becomes the short-term future. We will then have a better idea what we want and a better idea how to achieve it.
• As mentioned, an increasing immortal population means that our "rights" over the distant future must be fairly dilute.
• If we don't discount the future, we run into mathematical difficulties. The first rule of utilitarianism ought to be KIFS - Keep It Finite, Stupid.

Hmm. I wonder if it helps with gathering energy to fight the views of others if you label their views as being "deathist".

I agree.

Direct question. I cannot infer answer from you posts. If human values do not exist in closed form (i.e. do include updates on future observations including observations which in fact aren't possible in our universe), then is it better to have FAI operating on some closed form of values instead?

"But of course the claims are separate, and shouldn't influence each other."

No, they are not separate, and they should influence each other.

Suppose your terminal value is squaring the circle using Euclidean geometry. When you find out that this is impossible, you should stop trying. You should go and do something else. You should even stop wanting to square the circle with Euclidean geometry.

What is possible, directly influences what you ought to do, and what you ought to desire.

Change in values of the future agents, however sudden of gradual, means that the Future (the whole freackin' Future!) won't be optimized according to our values, won't be anywhere as good as it could've been otherwise.

Even at the personal level our values change with time, somewhat dramatically as we grow in maturity from children to adolescents to adults, and more generally just as a process of learning modifying our belief networks.

Hanson's analogy to new generations of humans is especially important - value drift occurs naturally. A fully general argument against any form of value drift/evolution is equivalent to a fully general argument against begetting new generations of humans. Thus it is not value drift that equates to deathism, but the lack of value drift.

Out of the space of all trajectories value evolution could take, some are going to be much worse or much better from the perspective of our current value system.

All we have to do (and in fact the best we possibly can do) is guide that trajectory in a favorable direction from our current vantage point.

Ignore AGI for a moment and imagine a much more gradual future where human intelligence enhancement continues but asymptotically reaches a limit while human population somehow continues to explode exponentially (via wormholes for example). Run that millions of years forward and you get a future that is unimaginably different than our present, with future beings that are probably quite unlike humans.

A runaway Singularity is more or less equivalent to a portal into that future. It just compresses or accelerates it.

I would be that many of those futures, probably even most, will be considerably better overall than our past from the perspective of our current value system, and I would give good odds that they would be better than our present from the perspective of our current value system.

We don't need to suddenly halt all value drift in order to create a good future, and even if we could (questionable) it would probably be counterproductive or outright disasterous.

Azathoth already killed you once, at puberty. Are you significantly worse off now that you value sex? Enough to eliminate that value?

The future is lost when I cease to exist.

Is it more important to you that people of the future share your values, or that your values are actually fulfilled? Do you want to share your values, so that other (future?) people can make the world better, or are you going to roll up your sleeves and do it yourself? After all, if everyone relies on other people to get work done, nothing will happen. It's not Pareto efficient.

I think your deathism metaphor is flawed, but in your terms: Why do you assume "living for as long as I want" has a positive utility in my values system? It's not Pareto efficient to me: if everyone had the option of dying whenever they wanted, it would have consequences that I consider too negative in comparison to the benefits. (Bear in mind that this is a subjective assessment; even if I gave my reasoning you might still disagree, which is fine by me.)

Have the courage and rationality to admit that the loss is real, even if it's too great for mere human emotions to express.

I'm cowardly, irrational, and shallow, and I'm not afraid to admit it!

The problem with this logic is that my values are better than those of my ancestors. Of course I would say that, but it's not just a matter of subjective judgment; I have better information on which to base my values. For example, my ancestors disapproved of lending money at interest, but if they could see how well loans work in the modern economy, I believe they'd change their minds.

It's easy to see how concepts like MWI or cognitive computationalism affect one's values when accepted. It's likely bordering on certain that transhumans will have more insights of similar significance, so I hope that human values continue to change.

I suspect that both quoted authors are closer to that position than to endorsing or accepting random value drift.