Not necessarily. If A was a Nash equilibrium while B was a Pareto improvement from that but the second society couldn't coordinate to achieve it, then they could gaze wistfully into the past, say they had fallen, and be right to do so.

Presumably you are assuming that societies judge their values by their values, always coming to the answer "we're good". But societies can do better and worse at realising their values. Movoer, socieites can judge moral values by non moral values, for instance by consistency. (Yudkowsky's habit, aparently copied by Bostrom, of refusing to distingusih moral value from non-moral value, causes problems, inasmuch as making the distinction solves problems).

I am not sure putting your values into practice counts as a moral value.

There are two kinds of inconsistency, are both dealt with in CEV?

There is internal inconsistency of an individual's (each individual's?) morality. Things like pushing the fat guy onto the trolley tracks to save 5 skinny guys.

There is also (possibly) inconsistency between individual humans. A smart good friend of mine over the last 40 years has very different politics from mine, suggesting a different set of values. Sure we agree you shouldn't kill random people in the city and on like that. But it seems we disagree on the kinds of things that justify forced collective action (taxation, laws). As a simple and frustrating example, he would like to see flag-burning illegal, that is nuts to me.

Is there a plan to have CEV handle the differences in values between different humans? And where do we draw the line at human: a sociopath is pretty obviously human, must CEV be consistent with both my values and a sociopath's values? If not, are we just picking a subset of humanity, defining a "we" and a "they" and developing "our" CEV?

People talk as if inconsistencies and contradictions in our value systems mean the whole enterprise of emulating human morality is worthless. Of course human value systems are contradictory; you can still implement a contradictory value system if you're willing to accept the occasional mis-calculation.

Ot you could start a project to research whether the morally relevant subset of value is also a non contradictory subset of value. Just a thought.

One interesting question is, when deciding to CEV people, from which era to extract the people. Saying that all eras would result in the same CEV is equivalent to saying that there is a fundamental correlation between the course of history and coherence which has but one final telos. An unlikely hypothesis, due to all sorts of things from evolutionary drift - as opposed to convergence - to orthogonality in ethics, to reference class tennis.

So researching how to distribute the allocation of CEV among individuals and groups would be a fascinating area to delve into.

I think the two most-important decisions are:

1. Build a single AI and give it ultimate power, or build a stable ecosystem / balance of power between AIs?

2. Try to pass on specific values of ours, or try to ensure that life continues operating under parameters that produce some beings that have values something like that?

Each of these decisions suggests research questions.

1a. How can we extend our models of competition to hierarchical agents--agents that are composed of other agents? Is most of the competition at the top level, or at the lower levels? (For starters, is there some natural distribution of number of agents of different sizes / levels / timescales, like there is for cities of different sizes?) The purpose is to ask whether we can maintain useful competition within a singleton.

1b. For some set of competing hierarchical AIs, what circumstances make it more likely for one to conquer and subsume the others? Under what circumstances might a singleton AI split up into multiple AIs? The purpose is to estimate whether it's possible to indefinitely avoid permanent collapse into a singleton.

2a. Try to find a candidate set of human values. Find how each is implemented neurally. The purpose is to see whether such things exist, what sorts of things they are, and whether they're the sort of things that can be implemented in a logic.

2b. List the behaviors of a wide variety of animals. Find values/preferences/behaviors of interest, and for each, find the conditions that tend to lead animals to have / not have those behaviors, as I did for boredom in this comment. The purpose is to see what fraction of the space of behaviors is acceptable to us, and to discover the evolutionary conditions that lead to that fraction of that space. That will give us an idea of how tightly we can constrain future values by controlling the gross parameters of the ecosystem.

Researching CEV as a foregone conclusiom, or researching whether it is a good idea?

If human values are not coherent, is that not a problem for any plans we might have for the future, rather than just CEV?

It is a problem if your particular values require you to drag all of humanity along with you into whatever glorious future you perceive. If that is your value, to drag all of whatever you decide qualifies as human with you on your journey, then you will probably be defeated by groups who are not limited by that value. And the absolute mass of human history shows groups of humans who are more than happy to build a future only for their in-group, with significantly less regard if not total lack of regard for the interests of all the humans not in the in-group.

Certainly we have seen the size of in-groups rise through human history. And many of us have extrapolated from this that the in-group is heading towards all humanity, and then made the illogical leap (or succumbed to the sentiment) that therefore the in-group SHOULD be all of humanity. But at the same time, plenty of us, even those in groups measured in the 100s of millions, are still happy to support the in-group specially, to deport people who were not born in the US but happened to group up there since childhood, to pass laws giving all the good jobs and government benefits to citizens, to place it as a positive value to value the income of a lazy alcoholic in Ohio over the income of a peasant in Mexico or China who might work three times as hard for a tenth as much to support his family.

One would imagine from the success and power of nations numbering in the 100s of millions that widening the definition of "us" has made humans more productive, made the larger-grouped humans more fit in an evolutionary sense. But just as an elephant's size advantages do not lead to elephants as large as mountains, there is evidence that making the ingroup much larger than a billion, at this point in our evolution and technical expertise, does NOT provide an advantage over groups of 100s of millions.

I would expect the future to belong to groups numbered in the 100s of millions, perhaps leading towards a few billion as technology improves, that are happy to prioritize themselves much higher than the outgroup, that do not waste energy and resources on a bunch of humans who are not with the program and who are going to be more effort than they are worth trying to get with the program, or even worse are working against it.

It might not be a problem if we decide to work on the meta-level, and, rather than trying to optimize the universe according to some extrapolation of human values, tried to make sure the universe kept on having conditions that would produce some things like humans.

Nah, we can just ignore the evil fraction of humanity's wishes when designing the Friendly AI's utility function.

Split the cake into parts and provide it to different extrapolated volitions that didn't cohere, after allocating some defense army patrol keeping the borders from future war?

Perhaps the notion that we're obligated not just to care about the future, but to want it to have our values. There's a step skipped in going from "I value X" to "I value other agents valuing X."

This step is glossed over by saying that I have a utility function that values X. But my utility function is full of quasi-indexicals--a fancy way of saying that it has terms like "me" in it, as in, "I value my happiness," or, "I value the happiness of my people". If you copy that utility function into another agent's mind, the my will now refer to that agent.

If we look to the real human world we see immediately that people don't always want other agents to share their utility function. Kings like being king, and they want everyone else not to want to be king. Social parasites want other people not to be social parasites. Etc.

We also see that, while people profess to care about people on the other side of the world, they don't. There is a decay of our concern for people with geographic distance, and with distance in time going forward. We care a lot more about people existing today than about people to exist 100 years from now. I would argue that it is impossible for our evolved utility functions to say anything about what goes on outside our "influence cone" (parts of spacetime that can influence us or our descendants, or that we or our descendants can influence), and any care we feel about them is us following some abstract model we've built about ourselves, which will be the first thing to go if we ever do find real "human values".

I'd like the future to have nice things in it: life, consciousness, love. But to have my values? That's... kind of boring. I want the future to tell an interesting story. That probably requires having a lot of people who don't share my values.

I know somebody's going to say, "Well, then that's your utility function!" Yeah, sorta... but it's not the sort of thing that "human values" suggests. It's one or two levels of abstraction above "love your neighbors" or "music based on a 12-tone scale". It's not the sort of thing that needs to be controlled tightly to satisfy. It's not the sort of thing I'd try to optimize rather than satisfice.

The way CEV is phrased, it sounds more like a father trying to raise his kids to be just like him than a father trying to help them grow up.

I guess I disagree with the premise that we will have superintelligent successors who will think circles around us, and yet we get to specify in detail what ethical values they will have, and it will stick. Forever. So let's debate what values to specify.

A parent would be crazy to think this way about a daughter, optimizing in detail the order of priorities that he intends to implant into her, and expecting them to stick. But if your daughter is a superintelligence, it's even crazier.

The older I get and the more I think of the AI issues the more I realize how perfectly our universe is designed! I think about the process of growing up: I cherish the time I spent in each stage of life, unaware of what's to come later, because there are things to be learned that can only derive from that particular segment's challenges. Each stage has its own level of "foolishness", but that is absolutely necessary for those lessons to be learned! So too I think of the catastrophes I have endured that I would not have chosen, but that I would not trade for anything now due to the wisdom they provided me. I cannot see any way around the difficult life as the most supreme and loving teacher. This I think most parents would recognize as they wish for their kids: a life not too painful but not too easy, either.

CEV assumes that there is an arrival point that is more valuable than the dynamic process we undergo daily. Much as we delight in imagining a utopia, a truly good future is one that we STRUGGLE to achieve, balancing one hard-won value against another, is it not? I have not yet heard a single concept that arrives at wisdom without a difficult journey. Even the idea of a SI that dictates our behavior so that all act within its accordance has destroyed free will, much like a God who has revoked human volition. This leads me to a seemingly inevitable conclusion that no universe is preferable to the one we inhabit (though I have yet to see the value of horrible events in my future that I still try like the devil to avoid!) But despite this 'perfection' we're seemingly unable to stop ourselves from destroying it.

It seems Kurzweil's view of AI is different from this groups. This group seems to concentrate on autonomous AI on the one hand, against which humanity as it exists now must be protected. It seems to me Kurzweil, on the other hand, sees AI as enhancements to the human. That the human would be able to install hardware which would augment her brain (and her body, actually.)

Extrapolating my view of Kurzweil's view, I suppose that over time, the humans become less and less human as the enhancements they can adopt become an increasing fraction of the "human" which is designing and adopting the next generation of enhanced enhancements. Maybe the Kurzweilian view takes the universe to the same place that lesswrongian autonomous UAI's take the universe: with mechanical intelligence taking over primary spot from biological. But the transition seems more comfortable to contemplate, since at every step something that at least used to be human is taking the next step.

In some sense, I see CEV and attempts to insure FAI over UAI as a modern equivalent of Chasidic Judaism. Preserving a "better" past against the encroachment of modern "sin" without actually succeeding in doing that. In some sense, scientifically educated rational atheists are the UAI threat to the orthodox believers.

In some sense, I think I'd rather be extinct than to live in a community guarded from the harsh realities by FAI, living out a cartoon version of a life which is centuries out of date.

I suppose I think of my in-group as "dominant intelligences" instead of "humanity." And so I feel more kinship with the dominant intelligences, even when they are no longer human.

I read the book several times already and it makes me more and more pessimistic. Even if we make SI to follow CEV, at some point it might decide to drop it. Its SI above all, it can find ways to do anything. Yet we can't survive without SI. So SEV proposal is as good and as bad as any other proposal. My only hope is that moral values could be as fundamental as laws of nature. So a very superintelligent AI will be very moral. Then we'll be saved. If not, then it will create Hell for all people and keep them there for eternity (meaning that even death could be a better way out yet SI will not let people die). What should we do?

Is this different from the question "Can groups cooperate rationally?"

Could groups commit to CEV somehow, so that it could better prevent conflict?

I have a sense that this is what people who join the intelligence services do. The intelligence services lie both to keep secrets and to manipulate people into doing things they don't really want to do. The group of people in those services who believe in what they are doing believe they are doing right, that the things they are preserving are more valuable than the lesser values of telling the truth and not killing "innocent" people. Within the intelligence services it would appear conflict is largely avoided, with rare but spectacular exceptions like Snowden.

Indeed, this group value I am noticing would appear to apply to the police and the military. Anywhere where getting things done relies upon contravening the simple values of not using physical force against other people.

Is this the kind of thing you mean to be asking about?

It depends on how many groups and how many people in each. The Tobin tax would have been great to avoid some international financial catastrophes. The Kyoto protocol and all other sorts of tragedy of the common problems indicate that this type of commitment is hard to make and harder to keep. An interesting book mixing this with transhumanism and bioethics is "Unfit for the future" by Julian Savulescu.

I might not mind locking in my current values, but I sure don't want to lock in your current values.

No, more serious: Yes, it would be bad. As I wrote in "The human problem",

Pretty soon your humans will tile your universe with variations on themselves. And the universe you worked so hard over, that you had such high hopes for, will be taken up entirely with creatures that, although they become increasingly computationally powerful, have an emotional repertoire so impoverished that they rarely have any complex positive qualia beyond pleasure, discovery, joy, love, and vellen. What was to be your masterpiece becomes instead an entire universe devoid of fleem.

Consider that homo sapiens is in some sense a creation of our primate ancestors. Would it have been bad if those primate ancestors had managed to put limits on the evolution that lead to homo sapiens such that the evolution could never create a species which might supplant the ancestor? That would successfully force the any evolved knew species to serve the old one, perhaps to use its technology to cure the diseases of the primate ancestor, to tile the world in bananas and coconuts, to devote itself to the preservation and enhancement of primate ancestor culture?

I guess it would be bad for homo sapiens, but not so bad for the primate ancestors?

Would our primate ancestors be open to a charge that they were creating a race of slaves by limiting their evolution thusly? Of course, it seems more than likely that creating a race of slaves would not be ruled out by our primate ancestors CEV.