The Bedrock of Morality: Arbitrary?

I'm fine with a galaxy without humor, music, or art. I'd sacrifice all of that reproductive fitness signalling (or whatever it is) to maximize my persistence odds as a subjective conscious entity, if that "dilemma" was presented to me.

I kind of think humor, music and art are pretty cool, myself.

Hopefully: But would you replace those with anything else? I'd want persistance, but I'd want growth and, well, fun! :)

I think that your use of the word arbitrary differs from mine. My mind labels statements such as "we should preserve human laughter for ever and ever" with the "roko-arbitrary" label. Not that I don't enjoy laughter, but there are plenty of things that I presently enjoy that, if I had the choice, I would modify myself to enjoy less. Activities such as enjoying making fun of other people, eating sweet foods, etc. It strikes me that the dividing line between "things I like but wish I didn't like" and "things I like and want to keep liking" should be made in some non-roko-arbitrary way. One might incorporate my position with eliezer's by saying that my concept of "rightness" relies heavily on my concept of arbitrariness, and that my concept of arbitrariness is clearly different to eliezer's.

Eliezer: "I am not playing games by redefining 'good' or 'arbitrary' [...]"

I imagine the counterargument would be that while you're not playing e-games by e-redefining the terms, you are playing games by redefining the terms.

Upon a preview it looks like Roko beat me to it.

It also worries me quite a lot that eliezer's post is entirely symmetric under the action of replacing his chosen notions with the pebble-sorter's notions. This property qualifies as "moral relativism" in my book, though there is no point in arguing about the meanings of words.

My posts on universal instrumental values are not symmetric under replacing UIVs with some other set of goals that an agent might have. UIVs are the unique set of values X such that in order to achieve any other value Y, you first have to do X. Maybe I find this satisfying because I have always been more at home with category theory than logic; I have defined a set of values by requiring them to satisfy a universal property.

But laughter, I suspect, may be rarer by far than mercy.

Curious why you suspect this. Is it particularly mammalian in some respect? I confess I could be naive, but it seems to me that any sufficiently intelligent being/agent would be just as likely as we humans are to have humor. I suppose that raises the question of how likely it is, and are we just incredibly lucky to have inherited such a trait. Still, it's such a core aspect of so much of our species -- even more than mercy, I think! -- that I'm curious why you think that.

Eliezer, you claim that there is no necessity we should accept Dennis's claim he should get the whole pie as fair. I agree.

There is also no necessity he should accept our alternative claim as fair.

There is no abstract notion that is inherently fair. What there is, is that when people do reach agreement that something is fair, then they have a little bit more of a society. And when they can't agree about what's fair they have a little less of a society. There is nothing that says ahead of time that they must have that society. There is nothing that says ahead of time what it is that they must agree is fair. (Except that some kinds of fairness may aid the survival of the participants, or the society.)

Concepts of fairness aren't inherent in the universe, they're an emergent property that comes from people finding ways to coexist and to find mutual aid. If they agree that it's fair for them to hunt down and kill and eat each other because each victim has just as much right and almost as much opportunity to turn the tables, this does not lead to a society that's real useful to its participants and it does not lead them to be particularly useful to each other. It's a morality that in many circumstances will not be fully competitive. But this is a matter of natural selection among ideas, there isn't anything less fair about this concept than other concepts of fairness. It's only less competitive, which is an entirely different thing.

It's an achievement to reach agreement about proper behavior. The default is no agreement. We make an effort to reach agreement because that's the kind of people we are. The kind of people who've survived best so far. When Dennis feels he deserves something different from what we think, we often feel we should try to understand his point of view and see if we can come to a common understanding.

And we have to accept that sometimes we cannot come to any common understanding, that's just how it works. We have to accept that sometimes somebody will feel that it isn't fair, that he's been mistreated, and we have to live with whatever consequences come from that. Society isn't an all-or-none thing. We walk together, we stumble, we fall down, we get back up and try some more.

Why would anybody think that there is a single perfect morality, and if everybody could only see it then we'd all live in peace and harmony?

You might as well imagine there's a single perfect language and if we all spoke it we'd understand each other completely and everything we said would be true.

Though Eliezer does not say it explicitly today, the totality of his public pronouncements on laughter leads me to believe that he considers laughter an instrinsic good of very high order. I hope he does not expect me to accept the highness of the probability of the rareness of humor in the universe as evidence for humor's intrinsic goodness. After all, spines are probably very rare in the universe, too. At least spines with 32 (or however many humans have) vertebrae are.

Eliezer does not explicitly say today that happiness is an intrinsic good, but he does contrast pebble sorting with "the human vision of a galaxy in which agents are running around experiencing positive reinforcement."

I take it Eliezer does not wish to see the future light cone tiled with tiny computers running Matt Mahoney's Autobliss 1.0. Pray tell me, What is wrong with such a future that is not also wrong with a future in which the resources of the future light cone are devoted to helping humans run around and experience positive reinforcement? Eliezer answer might refer to the difference between the simplicity of Autobliss 1.0 and the complexity of a human. Well, my reply to that is that it is relatively easy to make Autobliss more complex. We can even employ an evolutionary algorithm to create the complexity, increasing the resemblance between Autobliss 2.0 and humans. Eliezer probably has a reply to that, too. But when does this dialog reach the point where it is obvious that the distinction that makes humans intrinsically valuable and Autobliss 1.0 not valuable is being chosen so as to have the desired consequence? And did we not have a sermon some day in the last couple of weeks about how it is bad to gather evidence for a desired conclusion while ignore evidence against the conclusion?

I find Eliezer's seemingly-completely-unsupported belief in the rightness of human benevolence, as opposed to sorting pebbles, pretty scary.

@J Thomas: "Why would anybody think that there is a single perfect morality, and if everybody could only see it then we'd all live in peace and harmony?"

Because they have a specific argument which leads them to believe that?

You know, there's no reason why one couldn't consider one language more efficient at communication than others, at least by human benchmarks, all else being equal (how well people know the language, etc.). Ditto for morality.

Thomas, you are running in to the same problem Eliezer is: you can't have a convincing argument about what is fair, versus what is not fair, if you don't explicitly define "fair" in the first place. It's more than a little surprising that this isn't very obvious.

Eliezer can reply that moral conclusions are different, so the sermon does not apply. Well, I think it should apply, in certain case, such as when you are contemplating the launch of the seed of a superintelligence, which is an occasion that IMO demands a complete reevaluation of one's terminal values and the terminal values of one's society.

I find Eliezer's seemingly-completely-unsupported belief in the rightness of human benevolence, as opposed to sorting pebbles, pretty scary.

...

Kip and Wiseman slipped in. Busy tonight.

Richard: "Eliezer answer might refer to the difference between the simplicity of Autobliss 1.0 and the complexity of a human."

I'm pretty sure he wouldn't say that. Rather, the claim (if I'm reading him correctly) is that the true referent of good really is a really complicated bundle of human values. In a material universe, you can't cash out "intrinsic goodness" in the intuitive way.

I'm really having trouble understanding how this isn't tantamount to moral relativism -- or indeed moral nihilism. The whole point of "morality" is that it's supposed to provide a way of arbitrating between beings, or groups, with different interests -- such as ourselves and Pebblesorters. Once you give up on that idea, you're reduced, as in this post, to the tribalist position of arguing that we humans should pursue our own interests, and the Pebblesorters be damned. When a conflict arises (as it inevitably will), the winner will then be whoever has the bigger guns, or builds AI first.

Mind you, I don't disagree that this is the situation in which we in fact find ourselves. But we should be honest about the implications. The concept of "morality" is entirely population-specific: when groups of individuals with common interests come into contact, "morality" is the label they give to their common interests. So for us humans, "morality" is art, music, science, compassion, etc. in short, all the things that we humans (as opposed to Pebblesorters) like. This is what I understand Eliezer to be arguing. But if this is your position, you may as well come out and admit that you're a moral relativist, because this is the position that the people who are scared of moral relativism are in fact scared of. What they dread is a world in which Dennis could go on saying that Dennis-morality is what really matters, the rest of us disagree, war breaks out, Dennis kills us all, eats the whole pie, and is not spanked by any cosmic force. But this is indeed the world we live in.

Roko,

"UIVs are the unique set of values X such that in order to achieve any other value Y, you first have to do X." Roko,

You know that all of the so-called 'UIVs' that have been postulated only apply for some Y under some conditions (the presence of other powerful agents and game theoretic considerations or manipulation, self-referential utility functions, preferences over mathematical truths, and many other considerations make so-called UIVs useless or sources of terminal disvalue for an infinite number of cases), and an agent could have the terminal value Y1, where Y1 is not valuing anything in X, so X is an empty set.

Why are 'often instrumental values,' or OIVs, non-arbitrary terminal values as well?

Kip, can you expound?

There has never been, so far as I able to determine, any force so unfriendly to humans as humans. Yet we read day after day about one very smart man's philosophizing about the essence of humanity, supposedly so that it can be included in the essence of fAI. Wouldn't it be incredible if tomorrow, or sometime in the near future, someone who has been working and actually come up with some designs for fAI or AGI produces a real product, and it makes all the hubris of these responses irrelevant? What is the purpose of an intelligence that is able to take all the unkind things mankind has been able to do, and do them faster and more efficiently? Paper clips may be the answer; certainly humans cannot use their record to debate it. Finally, the fact that one man, no matter how gifted, thinks that he is the only possible answer makes one shudder at the potential attitude of the superhuman intelligence he would create. It will not only have the attitude, "I am unlikely to to take your advice, or even to take it seriously, so stop wasting your time", as E. Y. said to one poster, but it will have that attitude toward it's programmers as well, at the level of superhuman effectiveness. I want fAI as much as anyone. All this public rumination is not the approach.

Clarification: in the first paragraph of the above comment, when I wrote "The whole point of 'morality' is..." what I meant was "The whole point of non-relativist 'morality' is...".

"I find Eliezer's seemingly-completely-unsupported belief in the rightness of human benevolence, as opposed to sorting pebbles, pretty scary."

Kip,

Given Eliezer's definition of rightness (which is different from current object-level views), if there is a sufficiently cogent and convincing argument for pebblesorting, then pebblesorting is both right and p-right. Do you think that there is a significant chance you would ever view pebblesorting as k-right with expanded intelligence and study?

It looks like fairness can be said to be f-morality, built from current morality so that it is known to be sufficiently stable under reflection (that is, (meta)*-f-moral), and as moral as possible. While we travel the road of moral progress, avoiding getting trapped in the simplistic ditches of fake moralities, we need a solid target for agreement, and this is what a particular fairness is. Morality unfolds in a moral way, while casting a shadow of unfolded fairness.

Yes, Z.M., human happiness is not what Eliezer plans to use the superintelligence to maximize. Good to make that clear. But it might be worthwhile to question the intrinsic goodness of human happiness, as a warm-up to questioning the coherent extrapolated volition (CEV) of the humans.

But most of all - why on Earth would any human being think that one ought to optimize inclusive genetic fitness, rather than what is good? What is even the appeal of this, morally or otherwise? At all?

I don't think you ought to try to optimise fitness. Your opinion about fitness might be quite wrong, even if you accept the goal of optimising fitness. Say you sacrifice trying to optimise fitness and then it turns out you failed. Like, you try to optimise for intelligence just before a plague hits that kills 3/4 of the public. You should have optimised for plague resistance. What a loser.

And what would you do to optimise genetic fitness anyway? Carefully choose who to have children with?

Perhaps you would want to change the environment so that it will be good for humans, or for your kind of human being. That makes a kind of sense to me, but again it's hard to do. Not only do you have the problem of actually changing the world. You also have the problem of ecological succession. Very often, species that take over an ecosystem change it in ways that leave something else better able to grow than that species' own children. Some places, grasses provide a good environment for pine seedlings that then shade out the grass. But the pines in turn create an environment where hardwood saplings can grow better than pine saplings. Etc. If you like human beings or your own kind of human beings then it makes some sense to create an environment where they will thrive. But do you know how to do that?

If you knew all about how to design ecosystems to get the results you want, that might provide some of the tools you'd need to design human societies. I don't think those tools exist yet.

On a different level, I feel like it's important to avoid minimising mimetic fitness. If you have ideas that you believe are true or good or beautiful, and those ideas seem to kill off the people who hold them faster than they can spread the ideas, that's a sign that something is wrong. It should not be that the good, true, or beautiful ideas die out. Either there's something wrong with the ideas, or else there should be some way to modify the environment so they spread easier, or at least some way to modify the environment so the bearers of the ideas don't die off so fast. I can't say what it is that's wrong, but there's something wrong when the things that look good tend to disappear.

If they're good then there ought to be a way for them to persist until they can mutate or recombine into something better. They don't need to take over the world but they shouldn't just disappear.

I don't like it when the things I like go extinct.

So I don't want to maximise the fitness of things I like, but I sure do want that fitness to be adequate. When it isn't adequate then something is wrong and I want to look carefully at what's wrong. Maybe it's the ideas. Maybe something else.

Similarly, if you run a business you don't need to maximise profits. But if you run at a loss on average then you have a problem that needs to be fixed.

"Why would anybody think that there is a single perfect morality, and if everybody could only see it then we'd all live in peace and harmony?"

Because they have a specific argument which leads them to believe that?

Sure, but have you ever seen such an argument that wasn't obviously fallacious? I have not seen one yet. It's been utterly obvious every time.

Thomas, you are running in to the same problem Eliezer is: you can't have a convincing argument about what is fair, versus what is not fair, if you don't explicitly define "fair" in the first place. It's more than a little surprising that this isn't very obvious.

I gave a simple obvious definition. You might disagree with it, but how is it unclear?

Re: why on Earth would any human being think that one ought to optimize inclusive genetic fitness

"Ought" is a word that only makes sense in the context of an existing optimisation strategy. As far as biologists can reasonably tell, the optimisation strategy of organisms involves maximising their inclusive genetic fitness. So the short answer to this is: because nature built them that way.

The bigger puzzle is not why organisms act to maximise their inclusive genetic fitness, but why they sometimes systematically fail to do so. What cognitive malfunction causes phenomena such as the western demographic shift - that are unlikely to be adaptive.

With humans, it's not such a puzzle - the answer is usually: humans exist in a bizarre environment, and this causes their genetic program to malfunction. The problem will get fixed eventually.

For evolution being wasteful, see: http://alife.co.uk/essays/evolution_is_good/

For evolution being stupid, see: http://alife.co.uk/essays/evolution_sees/

komponisto: "I'm really having trouble understanding how this isn't tantamount to moral relativism"

I think I see an element of confusion here in the definition of moral relativism. A moral relativist holds that "no universal standard exists by which to assess an ethical proposition's truth". However, the word universal in this context (moral philosophy) is only expected to apply to all possible humans, not all conceivable intelligent beings. (Of all the famous moral relativist philosophers, how many have addressed the morals of general non-human intelligences?)

So we can ask two different questions:

#1. Is there a standard by which we can assess an ethical proposition's truth that applies to all humans?

#2. Is there a standard by which we can assess an ethical proposition's truth that applies to all conceivable intelligent beings?

I expect that Eliezer would answer yes to #1 and no to #2.

If you interpret universal in the broader sense (#2), then Eliezer would indeed be a moral relativist, but I think that distorts the concept of moral relativism, since the philosophy was developed with only humans of different cultures in mind.

You can apply the standard of goodness to all intelligent beings, no problem. It's just that they won't apply it to themselves.

The content of "good" is an abstracted idealized dynamic, or as Steven put it, a rigid designator (albeit a self-modifying rigid designator). Thus what is good, or what is not good, is potentially as objective as whether a pile of pebbles is prime. It is just that not every possible optimization process, or every possible mind, does what is good. That's all.

I wish Arnie's character had made a long speech in the middle of the film explaining that Predator wasn't evil or even wrong, he was just working around a different optimization process.

@HA

I'm fine with a galaxy without humor, music, or art. I'd sacrifice all of that...to maximize my persistence odds as a subjective conscious entity....

So existing is a terminal value in and of itself for you HA. Wouldn't you get bored? Or would you try to excise your boredom circuits, along with your humour, music and art circuits? How about your compassion circuits? Do you strive for the condition of perfect, empty, value-less ghost in the machine, just for its own sake...?

...because if so, I can tell you that you're 100% b-wrong about that.

In the real world, everything worth having comes from someone's effort -- even wild fruit has to be picked, sorted, and cleaned and fish need to be caught, gutted etc. I think this universal fact of required effort is probably part of the data we get the concept of fairness from in the first place, so reasoning in a space where pies pop in to existence from nothing seems like whatever you conclude might not be applicable to the real world anyway.

Ben, you write "Do you strive for the condition of perfect, empty, value-less ghost in the machine, just for its own sake...?".

But my previous post clearly answered that question: "I'd sacrifice all of that reproductive fitness signalling (or whatever it is) to maximize my persistence odds as a subjective conscious entity, if that "dilemma" was presented to me."

It is pretty clever to suggest objective morality without specifying an actual moral code, as it is always the specifics that cause problems.

My issue would be how Eliezer appears to suggest that human morality and alien morality could be judged separately from the genetics of each. Would super intelligent alien bees have the same notions of fairness as we do, and could we simply transplant our morality onto them, annd judge them accordingly, with no adjustments made for biological differences? I think it is very likely that such a species would consider the most fair distribution of a found pie to be one that involved a sizeable portion going to the queen, and that a worker who disagreed would be acting immorally. Is this something that we can safely say is objectively wrong?

To be fair (cough), your argument that '5 people means the pie should be divided into 5 equal parts' assumes several things...

1) Each person, by virtue of merely being there, is entitled to pie.

2) Each person, by virtue of merely being there, is entitled to the same amount of pie as every other person.

While this division of the pie may be preferable for the health of the collective psyche, it is still a completely arbitrary (cough) way to divide the pie. There are several other meaningful, rational, logical ways to divide the pie. (I believe I suggested one in a previous post.) Choosing to divide the pie into 5 equal parts simply asserts the premise 'existence = equal right' as the dominate principle by which to guide the division of the pie.

You have to remove all other considerations (including hunger, health, and any existing social relationships such as parent-child) in order to allow the 'existence = equal right' principle to be an acceptable way to divide the pie. This doesn't make that principle the 'bedrock' of morality. Quite the contrary. It says that this principle only dominates when all other factors are ignored.

Eliezer: "I really don't consider myself a moral relativist - not even in the slightest!"

Meta-ethical relativism (wikipedia)

Meta-ethical relativists, in general, believe that the descriptive properties of terms such as "good", "bad", "right", and "wrong" do not stand subject to universal truth conditions, but only to societal convention and personal preference. Given the same set of verifiable facts, some societies or individuals will have a fundamental disagreement about what one ought to do based on societal or individual norms, and one cannot adjudicate these using some independent standard of evaluation. The latter standard will always be societal or personal and not universal, unlike, for example, the scientific standards for assessing temperature or for determining mathematical truths.

I think that this describes Eliezer's position. He can adjudicate a disagreement between the pebblesorters and the humans, but he does it in a rather trivial way: he uses the standards of the humans not an independent standard.

Lakshmi, Eliezer does have a point, though.

While there are many competing moral justifications for different ways to divide the pie, and while a moral relativist can say that no one of them is objectively correct, still many human beings will choose one. Not even a moral relativist is obligated to refrain from choosing moral standards. Indeed, someone who is intensely aware that he has chosen his standards may feel much more intensely that they are his than someone who believes they are a moral absolute that all honest and intelligent people are obligated to accept.

So, once you have made your moral choice, it is not fair to simply put it aside because somebody else disagrees. If he convinces you that he's right, then it's OK. But if you believe you know what's right and you agree to do wrong, you are doing wrong.

If all but two members of the group -- you and Aaron -- think it's right to do something that Aaron thinks is unfair to him, then it's wrong for you to violate your ethics and go along with the group. If everybody but you thinks it's right then it's still wrong for you to agree, when you believe it's wrong.

Unless, of course, you belong to a moral philosophy which says it's right to do that.

When Dennis says he deserves the whole pie and you disagree, and it violates your ethical code to say it's right when you think it's wrong, then you should not agree for Dennis to get the whole pie. It would be wrong.

I believe that what you ought to do in the case when there's no agreement, should still be somewhat undecided. If you have the power to impose your own choice on someone else or everybody else then that might be the most convenient thing to do. But it takes a special way of thinking to say it's fair to do that. Is it in general a fair thing to impose your standards on other people when you think you are right? I guess a whole lot of people think so. But I'm convinced they're wrong. It isn't fair. And yet it can be damn convenient....

As far up as you go, there's no level that calls for unconditional surrender.

Roko: What the heck does morality have to do with category theory at all?

Meta-ethical relativists, in general, believe that the descriptive properties of terms such as "good", "bad", "right", and "wrong" do not stand subject to universal truth conditions, but only to societal convention and personal preference. Given the same set of verifiable facts, some societies or individuals will have a fundamental disagreement about what one ought to do based on societal or individual norms, and one cannot adjudicate these using some independent standard of evaluation. The latter standard will always be societal or personal and not universal, unlike, for example, the scientific standards for assessing temperature or for determining mathematical truths.

And there you have it: I am not a meta-ethical relativist. Humans and Pebblesorters do not have a fundamental disagreement about what one ought to do; humans do what they ought to do and Pebblesorters do what they p-ought to do. You can't have a disagreement without some particular fact or computation or idealized abstract dynamic that you are both arguing about, different beliefs with the same referent. "That which an optimization process does" is not a belief that can be argued about, it is a property of that optimization process. What is right, on the other hand, or what is p-right, is something that can be argued about; but a human does not dispute which piles of pebbles are prime.

Two people who actually disagree about the same question, a common referent, can try to adjudicate it (even if they don't have a nice neat formal procedure which is readily computable and known to them). I'm not sure what a "universal truth condition" is, but statements about rightness have truth conditions just as much as p-rightness.

Furthermore, I believe that human beings are better than Pebblesorters. This is not written upon the very stars, it is written in Platonia as the objective answer to that question that we ask when we ask "Is it better?" and not "Is it more xyblz?"

@Eliezer: "what one ought to do" vs. "what one p-ought to do"

Suppose that the pebblesorter civilization and the human civilization meet, and (fairly predictably) engage in a violent and bitter war for control of the galaxy. Why can you not resolve this war by bringing the pebblesorters and the humans to a negotiating table and telling them "humans do what they ought to do and Pebblesorters do what they p-ought to do"?

You cannot play this trick because p-ought is grounded in what the pebblesorters actually do, which is in turn grounded in the state of the universe they aim for, which is the same universe that we live in. The humans and the pebblesorters seem to be disagreeing about something as they fight each other: the usual way that people would put this disagreement into words is by saying "they are disagreeing about what is right".

However, you are using the word "right" in a nonstandard way. You have changed the meaning of the entire ethical vocabulary in this same way, to represent a specific constant answer rather than a variable, so it becomes very hard to say what the humans and pebblesorters are disagreeing about. It seems a little odd to say that these hated enemies are in complete agreement, and it is certainly not the standard way that people use the ethical vocabulary. Perhaps it is a better way: I'm just taking some time getting used to it.

In fact in your new use of the English language, you probably are not a relativist, for the way you are using the ethical vocabulary it is in fact impossible to be a relativist: all ethical theories T describe some objective predicate, T-right, and any act is either T-right or it isn't. In your new language, it isn't possible to talk of "rightness" detached from any particular predicate.

But I think that in your new use of language, you will need a word for the idea of a justification for an ethical theory, for example Kant's arguments "from first principles" in favor of the categorical imperative. Perhaps you could call ethical theories with this property "first-principles justified theories"? You may argue that no such theory exists, but a lot of philosophers would disagree, so you should have a word for it. And your ethical theory doesn't even try for this property, it is unashamedly unjustified.

Eliezer said: "Furthermore, I believe that human beings are better than Pebblesorters.

In your new use of the ethical vocabulary, this is a vacuous applause light. Of course the humans are better than the pebblesorters: you defined "good" as "the predicate that describes the particular set of things that humans do".

You cannot play this trick because p-ought is grounded in what the pebblesorters actually do, which is in turn grounded in the state of the universe they aim for, which is the same universe that we live in. The humans and the pebblesorters seem to be disagreeing about something as they fight each other

Does a human being disagree with natural selection? About what, exactly? How would we argue natural selection into agreement with us?

Standard game theory talks about interactions between agents with different goals. It does not presume that all agents must theoretically be arguable into exactly equal utility functions. These interactions are not called "disagreements", they are called "games" or "problems".

You have changed the meaning of the entire ethical vocabulary in this same way, to represent a specific constant answer rather than a variable

Lemme get this straight: I'm talking about rightness as a constant, you're talking about rightness as a variable, and you accuse me of being an moral relativist?

the way you are using the ethical vocabulary it is in fact impossible to be a relativist: all ethical theories T describe some objective predicate, T-right, and any act is either T-right or it isn't. In your new language, it isn't possible to talk of "rightness" detached from any particular predicate.

You're correct that I think moral relativism is an incoherent position for the same reason that factual relativism is an incoherent position. If anything that anyone wanted were right, this itself would be an ethical theory and not a relative one. Just as if anything that anyone believed were true, this itself would be reality.

But I think that in your new use of language, you will need a word for the idea of a justification for an ethical theory, for example Kant's arguments "from first principles" in favor of the categorical imperative.

All attempts to justify an ethical theory take place against a background of what-constitutes-justification. You, for example, seem to think that calling something "universally instrumental" constitutes a justification for it being a terminal value, whereas for me this is a nonstarter. For every mind that thinks that terminal value Y follows from moral argument X, there will be an equal and opposite mind who thinks that terminal value not-Y follows from moral argument X. I do indeed have a word for theories that deny this: I call them "attempts to persuade an ideal philosopher of perfect emptiness".

My theory is unabashedly justified; it is justified by arguments on the level of morality-as-morality. It so happens that human beings are the sort of creatures that respond to such arguments, and Pebblesorters are not; but we are not trying to "be human" in responding to such arguments - the justification for doing so, is that they are right (not xyblz).

In your new use of the ethical vocabulary, this is a vacuous applause light. Of course the humans are better than the pebblesorters: you defined "good" as "the predicate that describes the particular set of things that humans do".

No, "good" is defined as that which leads to sentient beings living, to people being happy, to individuals having the freedom to control their own lives, to minds exploring new territory instead of falling into infinite loops, to the universe having a richness and complexity to it that goes beyond pebble heaps, etc.

It so happens that humans are the sort of beings who do good, and that Pebblesorters are not; but this is a mere happenstance of a moral miracle, not the justification for having fun instead of sorting pebbles.

Do the fox and the rabbit disagree? It seems reasonable so say that they do if they meet: the rabbit thinks it should be eating grass, and the fox thinks the rabbit should be in the fox's stomach. They may argue passionately about the rabbit's fate - and even stoop to violence.

Do the fox and the rabbit disagree? It seems reasonable so say that they do if they meet: the rabbit thinks it should be eating grass, and the fox thinks the rabbit should be in the fox's stomach. They may argue passionately about the rabbit's fate - and even stoop to violence.

Really? I would be interested in hearing their philosophical arguments then as for why the rabbit should be eating grass or the rabbit should be in the fox's stomach. I understand, of course, that the rabbit does eat grass and that the fox does hunt the rabbit, but I was not aware that these were persuasive moral arguments. A rock does roll downhill, but I wasn't aware that this had any particular correlation to whether it should roll downhill - if gravity pulls a rock, it will just as readily roll over a toddler.

It would seem that you are not distinguishing between what a system does and what it should do. The former is not necessarily a statement about the latter; a rock, in rolling over a toddler, is not offering evidence or even argument about the worthlessness of human life - it's just showing that gravity doesn't care. Neither does a fox or rabbit.

Eliezer: No, "good" is defined as that which leads to sentient beings living, to people being happy, to individuals having the freedom to control their own lives, to minds exploring new territory instead of falling into infinite loops, to the universe having a richness and complexity to it that goes beyond pebble heaps, etc.

• you're going to have to give me a definition that doesn't involve "etc" for me to critique this further. As I understand it, you've taken some averaged or CEV'd collection of things that humans like doing, or things that humans find valuable.

But the actual initial segment of the list that you've proceeded with "etc" above is nothing like what most people think is good or valuable in life. An honest answer would probably read more like:

"good" is defined as that which leads to me having positive and exciting emotions, to people who I meet and am close to being happy, to me having children, to me having sex, to me having higher status and prestige than those around me, to everyone in the world worshiping the one true religion, to rich and undeserving westerners (excluding me) being punished for their sins against the third world/the natural environment, to the rival tribe/country/political party being destroyed and humiliated, to human dignity and natural things, etc

My list is the current human notion of goodness to 5 decimal places. Your list seems a lot more reasonable, but that's probably because you made it up yourself and you are a lot more reasonable than most humans. Are you claiming that the result of CEV, applied to my list, will be your list?

I would be interested in hearing their philosophical arguments then as for why the rabbit should be eating grass or the rabbit should be in the fox's stomach. I understand, of course, that the rabbit does eat grass and that the fox does hunt the rabbit, but I was not aware that these were persuasive moral arguments.

They are to the parties in question:

The rabbit argues that if it is eaten by the fox, then it will die - and that should not happen.

The fox argues that if it doesn't eat rabbits, then it will die - and that should not happen.

Neither considers the death of the other to be of much conseqence: for rabbits, foxes are evil rabbit-eaters, while foxes see rabbits as mere dumb lifestock.

@Eli: All attempts to justify an ethical theory take place against a background of what-constitutes-justification. You, for example, seem to think that calling something "universally instrumental" constitutes a justification for it being a terminal value, whereas for me this is a nonstarter. For every mind that thinks that terminal value Y follows from moral argument X, there will be an equal and opposite mind who thinks that terminal value not-Y follows from moral argument X. I do indeed have a word for theories that deny this: I call them "attempts to persuade an ideal philosopher of perfect emptiness".

• I think that there are quite canonical justifications for particular axiological statements. I should make these on my own blog, because that's where they belong.

Your argument that "For every mind that thinks that terminal value Y follows from moral argument X, there will be an equal and opposite mind who thinks that terminal value not-Y follows from moral argument X" is true if you regard "mind" as an abstract turing machine, but false if you regard it as an embodied agent. For example, you will not find an agent who thinks that it should delete itself immediately, though it would be possible for a disembodied mind to think this. Reality itself breaks the symmetry of abstract computations.

There are substantial things we can say about what real world agents are likely to think or do, purely based on them being agents. We can also say things about what real world societies of agents are likely to think, purely based on them being societies.

I think that there are shades of grey between "An argument that is universally compelling", and "An argument that compels only those who already believe its conclusion". The former is clearly impossible, the latter is what you have given.

Eliezer, I think I kind-of understand by now why you don't call yourself a relativist. Would you say that it's the "psychological unity of mankind" that distinguishes you from relativists?

A relativist would stress that humans in different cultures all have different - though perhaps related - ideas about "good" and "right" and so on. I believe your position is that the bulk of human minds are similar enough that they would arrive at the same conclusions given enough time and access to enough facts; and therefore, that it's an objective matter of fact what the human concepts of "right" and "good" actually mean.

And since we are human, there's no problem in us continuing to use those words.

Am I understanding correctly?

It seems like your position would become more akin to relativism if the "psychological unity" turned out to be dubious, or if our galaxy turned out to be swarming with aliens, and people were forced to deal with genuinely different minds. In those cases, would there still be anything to separate you from actual relativists?

(In either case, it would still be an objective matter of fact what any given mind would call "good" if given enough time - but that would be a much less profound fact than it is for a species all alone and in a state of psychological unity.)

It would seem that you are not distinguishing between what a system does and what it should do.

In my book, there's not really any such thing as what a system should do.

Should only makes sense with respect to the morals of some agent.

If you don't specify an agent, should becomes an extremely vague and ambiguous term.

Should statements are not about what happens, but about the desirability of what might happen - according to the moral system of some agent.

Concerning the charge of relativism: it seems clear that Eliezer is a moral relativist in the way that the term is normally understood, but not as he understands it. There may be a legitimate dispute here, but as far as communication goes, we should not be having problems. In deference to common usage, I would reserve right for the moral realism of Roko et al. and use something like h-right for Eliezer's notion of humanity's abstracted idealized dynamic--but I don't think it really matters right now.

Roko writes: "My list is the current human notion of goodness to 5 decimal places. Your list seems a lot more reasonable, but that's probably because you made it up yourself and you are a lot more reasonable than most humans. Are you claiming that the result of CEV, applied to my list, will be your list?"

This, I think, is the interesting question. Eliezer has been leaning heavily on the psychological unity of humankind, but I don't think this is enough to carry his argument. The unity of which we speak is a (you will forgive me:) relative term. We can agree that complex functional adaptations are species-typical modulo sex, and that all humans are virtually alike compared to the space of all possible minds, but that doesn't mean that there's no room at all for variation in morality in that tiny dot of human minds--variation that cannot be waved away as trivial. Evopsych can only go so far; the SSSM might have been a mistake, but that doesn't necessarily mean cultural and individual differences don't matter at all. That would take a separate, stronger argument, at least. (Cf. Virge and myself in comments to "Moral Error and Moral Disagreement.")

So we are left with a difficult empirical question: to what extent do moral differences amongst humans wash out under CEV, and to what extent are different humans really in different moral reference frames? I fear that there is no way to resolve this issue without a tremendous amount of data. Even if you had all the data you needed, it might be easier just to build the AI!

Eliezer, I think you come closer to sharing my understanding of morality than anyone else I've ever met. Places where I disagree with you:

First, as a purely communicative matter, I think you'd be clearer if you replaced all instances of "right" and "good" with "E-right" and "E-good."

Second, as I commented a couple threads back, I think you grossly overestimate the psychological unity of humankind. Thus I think that, say, E-right is not at all the same as J-right (although they're much more similar than either is to p-right). The fact that our optimization processes are close enough in many cases that we can share conclusions and even arguments doesn't mean that they're the same optimization process, or that we won't disagree wildly in some cases.

Simple example: I don't care about the well-being of animals. There's no comparison in there, and there's no factual claim. I just don't care. When I read the famous ethics paper about "would it be okay to torture puppies to death to get a rare flavor compound," my response was something along the lines of, "dude, they're puppies. Who cares if they're tortured?" I think anyone who enjoys torturing for the sake of torturing is probably mentally unbalanced and extremely unvirtuous. But I don't care about the pain in the puppy at all. And the only way you could make me care is if you showed that causing puppies pain came back to affect human well-being somehow.

Third, I think you are a moral relativist, at least as that claim is generally understood. Moral absolutists typically claim that there is some morality demonstrably binding upon all conscious agents. You call this an "attempt to persuade an ideal philosopher of perfect emptiness" and claim that it's a hopeless and fundamentally stupid task. Thus you don't believe what moral absolutists believe; instead, you believe different beings embody different optimization processes (which is the name you give to what most people refer to as morality, at least in conscious beings). You're a moral relativist. Which is good, because it means you're right.

Excuse me. It means you're J-right.

Odd how, despite the psychological unity of mankind and the ease of 'extrapolating' human volition, these discussions always seem to end in establishing specialized words to refer to the perceptions and beliefs of specific individuals.

Why "ought" vs. "p-ought" instead of "h-ought" vs. "p-ought"?

Sure, it might just be terminology. But change

"So which of these two perspectives do I choose? The human one, of course; not because it is the human one, but because it is right."

to

"So which of these two perspectives do I choose? The human one, of course; not because it is the human one, but because it is h-right."

and the difference between "because it is the human one" and "because it is h-right" sounds a lot less convincing.

To say that Eliezer is a moral relativist because he realizes that a primality sorter might care about primality rather than morality, is equivalent to calling him a primality relativist because he realizes that a human might care about morality rather than primality.

Nominull, don't the primalists have a morality about heaps of stones?

They believe there are right ways and wrong ways to do it. They sometimes disagree about the details of which ways are right and they punish each other for doing it wrong.

How is that different from morality?

If you've ever taken a mathematics course in school, you yourself may have been introduced to a situation where it was believed that there were right and wrong ways to factor a number into primes. Unless you were an exceptionally good student, you may have disagreed with your teacher over the details of which way was right, and been punished for doing it wrong.

It strikes me as plainly apparent that math homework is not morality.

To say that Eliezer is a moral relativist because he realizes that a primality sorter might care about primality rather than morality, is equivalent to calling him a primality relativist because he realizes that a human might care about morality rather than primality.

Thank you, Nominull. I'm glad someone gets it, anyway.

If you've ever taken a mathematics course in school, you yourself may have been introduced to a situation where it was believed that there were right and wrong ways to factor a number into primes. Unless you were an exceptionally good student, you may have disagreed with your teacher over the details of which way was right, and been punished for doing it wrong.

My experience with math classes was much different from yours. When we had a disagreement, the teacher said, "How would we tell who's right? Do you have a proof? Do you have a counter-example?". And if somebody had a proof we'd listen to it. And if I jumped up and said "Wait, this proof is wrong!" then the teacher would say, "First you have to explain what he said up to the point you disagree, and see if he agrees that's what he means. Then you can tell us why it's wrong."

I never got punished for being wrong. If I didn't do homework correctly then I didn't get credit for it, but there was no punishment involved.

But Eliezer described people who disagreed about how many stones to put in a pile and who had something that looked very much like wars about it. That isn't like the math I experienced. But it's very much like the morality I've experienced.

To say that Eliezer is a moral relativist because he realizes that a primality sorter might care about primality rather than morality, is equivalent to calling him a primality relativist because he realizes that a human might care about morality rather than primality.

But by Eliezer's standards, it's impossible for anyone to be a relativist about anything.

Consider what Einstein means when he says time and space are relative. He doesn't mean you can just say whatever you want about them, he means that they're relative to a certain reference frame. An observer on Earth may think it's five years since a spaceship launched, and an observer on the spaceship may think it's only been one, and each of them is correct relative to their reference frame.

We could define "time" to mean "time as it passes on Earth, where the majority of humans live." Then an observer on Earth is objectively correct to believe that five years have passed since the launch. An observer on the spaceship who said "One year has passed" would be wrong; he'd really mean "One s-year has passed." Then we could say time and space weren't really relative at all, and people on the ground and on the spaceship were just comparing time to s-time. The real answer to "How much time has passed" would be "Five years."

Does that mean time isn't really relative? Or does it just mean there's a way to describe it that doesn't use the word "relative"?

Or to give a more clearly wrong-headed example: English is objectively the easiest language in the world, if we accept that because the word "easy" is an English word it should refer to ease as English-speakers see it. When Kyousuke says Japanese is easier for him, he really means it's mo wakariyasui translated as "j-easy", which is completely different. By this way of talking, the standard belief that different languages are easier, relative to which one you grew up speaking, is false. English is just plain the easiest language.

Again, it's just avoiding the word "relative" by talking in a confusing and unnatural way. And I don't see the difference between talking about "easy" vs. "j-easy" and talking about "right" vs. "p-right".

Thanks, Yvain. Comparing well-understood special relativity to things characterized as "subjective" helps to clarify the sense in which they are really "objective", but look differently for different minds and are meaningless without any mind at all. You need a reference frame, and phenomenon does look different in different reference frames, but there are strict and consistent rules for converting between reference frames.

Very well said, Yvain!

I think it's is more akin to saying that "easy" could just as well mean difficult in some alien language, and so words don't mean anything and language is a farce. That's the true linguistic relativist position.

Yvain, I don't see why I would care about this thing you would call "moral", or refer to it often enough to justify such a short name.

Or indeed why it's the same thing that people have traditionally meant by the word.

People keep using the term "moral relativism". I did a Google search of the site and got a variety of topics with the term dating from 2007 and 2008. Here's what it means to me.

Relative moral relativism means you affirm that to the best of your knowledge nobody has demonstrated any sort of absolute morality. That people differ in moralities, and if there's anything objective to say one is right and another is wrong that you haven't seen it. That very likely these different moralities are good for different purposes and different circumstances, and if a higher morality shows up it's likely to affirm that the different moralities you've heard of tend to each have its place.

This is analogous to being an agnostic about gods. You haven't seen evidence there's any such thing as an objectively absolute morality, so you do not assert that there is such a thing.

Absolute moral relativism accepts all this and takes two steps further. First, the claim is that there is no objective way to judge one morality better than another. Second, the claim is that without any objective absolute morality you should not have any.

This is analogous to being an atheist. You assert that there is no such thing and that people who think there is suffer from fallacious superstitions.

I can be a relative moral relativist and stil say "This is my morality. I chose it and it's mine. I don't need it to be objectively true for me to choose it. You can choose something else and maybe it will turn out we can get along or maybe not. We'll see."

Why should you need an absolute morality that's good all times and all places before you can practice any morality at all? Here I am, here's how I live. It works for me. If you want to politely tell me I'm all wrong then I'll listen politely as long as I feel like it.

Submitted humbly for consideration: Ayn Rand is to libertarianism as Greg Egan is to transhumanism as Eliezer Yudkowsky is to moral relativism?

I don't think Eliezer belongs in the same category as Ayn Rand and Greg Egan.

"But most of all - why on Earth would any human being think that one ought to optimize inclusive genetic fitness, rather than what is good?"

You are asking why anyone would choose life rather than what is good. Inclusive genetic fitness is just the long term form of life, as personal survival is the short-term form.

The answer is, of course, that one should not. By definition, one should always choose what is good. However, while there are times when it is right to give up one's life for a greater good, they are the exception. Most of the time, life is a subgoal of what is good, so there is no conflict.

A sidenote:

Eliezer: "It has something to do with natural selection never choosing a single act of mercy, of grace, even when it would cost its purpose nothing: not auto-anesthetizing a wounded and dying gazelle, when its pain no longer serves even the adaptive purpose that first created pain."

It always costs something; it is cheaper to build a gazelle that always feels pain than one that does so until some conditions are met. This is the related to the case of supposing a spaceship that has passed out of your lightcone still exists.

Natural selection isn't fair; it doesn't compute fairness at all. This has nothing to do with the question whether there are situations where it could be fair to no cost, even if perhaps humans are more easily made aware of this fact if presented with cases where natural selection is outright unfair. Such cases very probably exists (ponder gene fixation) but to find them is nigh to impossible: you have to prove an absence of costs.

Yvian: "So which of these two perspectives do I choose? The human one, of course; not because it is the human one, but because it is h-right."

- well said. Modulo Eliezer's lack of explicitness about his definition of "h-right", I fail to see how the human perspective could be anything other than h-right. This post is just an applause light for the values that we currently like, and I think that that is a bad sign.

If human values were so great, you wouldn't have to artificially make them look better by saying things like

"So which of these two perspectives do I choose? The human one, of course; not because it is the human one, but because it is right "

@Z.M. Davis: So we are left with a difficult empirical question: to what extent do moral differences amongst humans wash out under CEV, and to what extent are different humans really in different moral reference frames?

- yes, this is a good point. And I fear that the answer depends on the details of the CEV algorithm.

Z.M Davis: "Submitted humbly for consideration: Ayn Rand is to libertarianism as Greg Egan is to transhumanism as Eliezer Yudkowsky is to moral relativism?"

- not sure I get this... Rand abhorred libertarianism because she thought it was half-baked and amateurish, but actually she is a libertarian, Egan spoke out against transhumanism because, uuum, he thinks we're all crackpots, but actually he's a transhumanist, Yudkowsky speaks out against moral relativism but actually he's the canonical example of a relativist. Ah, yes, ok.

Spot on, Z.M, Seconded.

Michael Anissimov, August 14, 2008 at 10:14 PM asked me to expound.

Sure. I don't want to write smug little quips without explaining myself. Perhaps I'm wrong.

It's difficult to engage Eliezer in debate/argument, even in a constructive as opposed to adversarial way, because he writes so much material, and uses so many unfamiliar terms. So, my disagreement may just be based on an inadequate appreciation of his full writings (e.g. I don't read every word he posts on overcomingbias; although I think doing so would probably be good for my mind, and I eagerly look forward to reading any book he writes).

Let me just say that I'm a skeptic (or "anti-realist") about moral realism. I think there is no fact of the matter about what we should or should not do. In this tradition, I find the most agreement with Mackie (historically) and Joshua Greene at Harvard (today). I think Eliezer might benefit greatly from reading both of them. You can find Greene's Ph.d thesis here:

http://www.wjh.harvard.edu/~jgreene/GreeneWJH/Greene-Dissertation.pdf

It's worth reading in entirety.

Why am I a moral skeptic? Before I give good reasons, let me suggest some possibly bad ones: it's a "shocking" and unpopular position. And I certainly love to be a gadfly. So, if Eliezer and I do have a real disagreement here, it may be drawn along the same lines we have with the free will debate: Eliezer seems to have strong compatibilist leanings, and I'm more inclined towards non-realism about free will. Thus, Eliezer may be inclined to resist shocking or uncomfortable truths, or I may be overly eager to find them. That's one possible reason for my moral skepticism.

I certainly believe that any philosophical investigations which lead people to generally safe and comfortable positions, in which common sense is vindicated, should give us pause. And people who see their role as philosopher as vindicating common sense, and making cherished beliefs safe for the world, are dishonoring the history of philosophy, and doing a disservice to themselves and the world. To succeed in that project, fully at least, one must engage in the sort of rationalization Eliezer has condemned over and over.

Now let me give my good reasons:

P1. An essential aspect of what it means for something to be morally right is that the something is not just morally right because everyone agrees that the something is. Thus, everyone agrees that if giving to charity, or sorting pebbles, is morally right, it is not just right because everyone says that it is. It is right in some deeper sense.

P2. But, all we have to prove that giving to charity, etc., is right, is that everyone thinks it is (to the extent they do, which is not 100%).

You might say: well, giving to charity increases the sum amount of happiness in the world, or is more fair, or follows some Kantian rule. But, then again, we ask: why? And the only answer seems to be that everyone agrees that happiness should be maximized, or fairness maximized, or that rule followed. But, as we said when we started, the fact that everyone agreed wasn't a good enough reason.

So we're left with reasons which we already agree are not good enough. We can only get around this through fancy rationalization, and in particular by forgetting P1.

Eliezer offers his own reasons for believing something is right:

"The human one, of course; not because it is the human one, but because it is right. I do not know perfectly what is right, but neither can I plead entire ignorance."

What horribly circular logic is that? It's right because it's right?

The last few words present a link to another article. And there you find quotes like these:

"Why not accept that, ceteris paribus, joy is preferable to sorrow?"

"You might later find some ground within yourself or built upon yourself with which to criticize this - but why not accept it for now? Not just as a personal preference, mind you; but as something baked into the question you ask when you ask "What is truly right"?"

"Are you willing to relinquish your Socratean ignorance?"

This is special pleading. It is hand waving. It is the sort of insubstantial, waxing poetic that pastors use to captivate their audiences, and young men use to romance young women. It is a sweet nothing. It should make you feel like you're being dealt with by a used car salesman; that's how I feel when I read it.

The question isn't "why not prefer joy over sorrow?" That's a wild card that can justify anything (just flip it around: "why not prefer sorrow over joy?"). You might not find a decisive reason against preferring joy to sorry, but that's just because you're not going to find a decisive reason to believe anything is right or wrong. Any given thing might make the world happier, or follow a popular rule, but what makes that "right"? Nothing. The problem above, involving P1 and P2, does not go away.

The content of morality is not baked into the definitions of words in our moral vocabulary, either (as Eliezer implies when he writes: "you will have problems with the meaning of your words, not just their plausibility"---another link). Definitions are made by agreement and, remember, P1 says that something can't be moral just because everyone agrees that the something is moral. The language of morality just refers to what we should do. The words themselves, and their definitions, are silent about what the content of that morality is, what the things are that we should actually do.

So I seem to disagree with Eliezer quite substantially about morality, and in a similar way to how we disagree about free will.

Finally, I can answer the question: what scares me about Eliezer's view? Certainly not that he loves joy and abhors suffering so much. Believe me when I say, about his mission to make the universe one big orgasm: godspeed.

Rather, it's his apparent willingness to compromise his rationalist and critical thinking principles in the process. The same boy who rationalized a way into believing there was a chocolate cake in the asteroid belt, should know better than to rationalize himself into believing it is right to prefer joy over sorrow.

What he says sounds nice, and sexy, and appealing. No doubt many people would like for it to be true. As far as I can tell, it generally vindicates common sense. But at what cost?

Joy feels better than sorrow. We can promote joy instead of sorrow. We will feel much better for doing so. Nobody will be able to criticize us for doing the wrong thing. The world will be one big orgasm. Let's satisfy ourselves with that. Let's satisfy ourselves with the merely real.

Kip Werking: "P2. But, all we have to prove that giving to charity, etc., is right, is that everyone thinks it is"

You're stating that there exists no other way to prove that giving to charity is right. That's an omniscient claim.

Still, it's unlikely to be defeated in the space of a comment thread, simply because your sweeping generalization about the goodness of charity is far from being universally accepted. A very general claim like that, with no concrete scenario, no background information on where it is to be applied, makes relativism a foregone conclusion.

I'd like to hear your arguments for something a little more fundamental. Would you apply the same reasoning to the goodness of life? Would you be prepared to claim that
"all we have to prove that your life is better than your death, is that everyone thinks it is"?

And with regard to the joy/sorrow question, would you be prepared to claim that
"all we have to prove that your not suffering is better than your suffering, is that everyone thinks it is"?

I think that all the morality is is just to be sure that the persons close to me rank high in the prisioner dilemma game, and to assure others that I rank high too. Even higuer than I really am.

For this purpose is all that has been done by evolution, intellectual and religious thinking.

The same boy who rationalized a way into believing there was a chocolate cake in the asteroid belt, should know better than to rationalize himself into believing it is right to prefer joy over sorrow.

Obviously, he does know. So the next question is, why does he present material that he knows is wrong?

Professional mathematicians and scientists try not to do that because it makes them look bad. If you present a proof that's wrong then other mathematicians might embarrass you at parties. But maybe Eliezer is immune to that kind of embarrassment. Socrates presented lots of obvious nonsense and people don't think badly of him for it.

The usual reasons why not probably don't apply to him. I don't have any certainty why he does it, though.

I do apologize for coming late to the party; I've been reading, and really feel like I'm missing an inferential step that someone can point me towards.

I'll try to briefly summarize, knowing that I'll gloss over some details; hopefully, the details so glossed over will help anyone who wishes to help me find the missing step.

It seems to me that Eliezer's philosophy of morality (as presented in the metaethics sequence) is: morality is the computation which decides which action is right (or which of N actions is the most right) by determining which action maximizes a complex system of interrelated goals (eg. happiness, freedom, beauty, etc.). Each goal is assumed to be stated in such a way that "maximizes" is the appropriate word (ie. given a choice between "maximize X" and "minimize ~X", the former wording is chosen; "maximize happiness" rather than "minimize unhappiness").

Further, humanity must necessarily share the same fundamental morality (system of goals) due to evolutionary psychology, by analogy with natural selection's insistence that we share the same fundamental design.

One of Eliezer's primary examples is Alice and Bob's apparent disagreement over the morality of abortion, which must, it seems, come down to one of them having incomplete information (at least relative to the other). The other is the Pebblesorting People who have a completely different outlook on life, to the point where they don't recognize "right" but "p-right".

My first problem (which may well be a missed inferential step) is with the assumed universality, within humanity, of a system of goals. Most humans agree that freedom is a good thing, but a large minority doesn't believe it the most important thing (China comes to mind; my understanding is that a great many of China's citizens don't care that their government is censoring their own history). In point of of fact, it seems that "freedom", and especially individual freedom, is a relatively modern invention. But, let's visit Alice and Bob.

Alice believes that abortion is morally acceptable, while Bob disagrees. Eliezer's assertion seems to be that this disagreement means that either Alice or Bob has incomplete information (a missing fact, argument, or both, but information). Why is it not possible simply that Alice holds freedom as more important that life and Bob the reverse? A common argument against abortion holds that the future life of the fetus is more important than the restriction on the pregnant woman's freedom of choice. Eliezer's morality seems to imply that that statement must, a priori be either true or false; it cannot be an opinion like "walnuts taste better than almonds" or even "Christmas is more important than Easter" (without the former, the latter would not be possible; without the latter, the former would not be important).

It seems to be a priori true that 2+2=4. Why does it necessarily hold that "life is more (or less) important than choice"? And, for the latter, how would we decide between "more" and "less"? What experiment would tell us the difference; how would a "more" world differ from a "less" world?

I also have a question about the Pebble-people: how is it that humans have discovered "right" while the Pebble-people have discovered only "p-right"? Even if I grant the assertion that all humans are using the same fundamental morality, and Alice and Bob would necessarily agree if they had access to the same information, how is it that humans have discovered "right" and not "h-right"? H-morality values abstract concepts like beauty, art, love, and freedom; p-morality values concrete things like pleasingly prime piles of pebbles. P-morality doesn't prohibit beauty or art, but doesn't value them - it is apathetic towards them except so far as they further proper pebble piles. Similarly, h-morality is apathetic towards prime piles of pebbles, except so far as they are beautiful or artistic. Yet Eliezer asserts that h-morality is better, is closer to morality. I don't see the supporting evidence, though, except that humans invented h-morality, so h-morality matches what we expect to see when we look for morality, so h-morality must be closer to morality than p-morality is. This looks like a circular trip through Cultural Relativism and Rule Utilitarianism (albeit with complex rules) with strong Anthropic favoritism. I really feel like I'm missing something here.

A standard science fiction story involves two true aliens meeting for the first time, and trying to exist in the same universe without exterminating each other in the process. How would this philosophy of morality influence the aliens that humans should be allowed to flourish even if we may in principle become a threat to their existence (or vice-versa)? H-morality includes freedom of individual choice as a good thing; a-morality (alien, not "amoral") may not - perhaps they evolved something more akin to an ant colony where only a few have any real choice. How would h-morality (being closer to morality than a-morality, of course) influence them to stop using slave labor to power their star-ships or gladiatorial combat to entertain their citizens?

Fundamentally, though, I don't see what the difference is between Eliezer's philosophy of morality and simply saying "morality is the process by which one decided what is right, and 'what is right' is the answer one gets back from running their possible actions through morality". It doesn't seem to offer any insight into how to live, or how to choose between two mutually-exclusive actions, both of which are in classically morally gray areas (eg. might it be right to steal to feed one's hungry family?); as I understand it, Eliezer's morality simply says "do whatever the computation tells you to do" without offering any help on what that computation actually looks like (though, as I said, it looks suspiciously like Cultural [Personal?] Relativism blended with Rule Utilitarianism).

As I said, I really feel like I'm missing some small, key detail or inferential step. Please, take pity on this neophyte and help me find The Way.

OK... let me see if I'm following.

The idea, trying to rely on as few problematic words as possible, is:

• There exists a class of computations M which sort proposed actions/states/strategies into an order, and which among humans underly the inclination to label certain actions/states "good", "bad", "right", "wrong", "moral", etc.(4) For convenience I'll label the class of all labels like that "M-labels."

• If two beings B1 and B2 don't implement (1) a common M-instance, M-labels may not be meaningful, even in principle, in discussions between B1 and B2. For example, B1 and B2 may fundamentally not mean the same thing by "right" or "moral."

• If B1 and B2 implement (1) one and only one M-instance (Mb), then M-labels are in-principle meaningful in discussions between B1 and B2 (although this is no guarantee that B1 and B2 will actually understand one another, or even that they are capable of discussion in the first place).

• There exists (6) an M-instance at-least-partially implemented (1) by all humans (2). We label this the Coherent Extrapolated Volition (CEV).

• Two humans might implement other M-instances in addition to CEV, or might not, but either way all human M-instances are (by definition) consistent with CEV. In other words, all within-group moralities among humans can be treated as special cases of CEV, and implementing CEV will satisfy all of them. (3) Edit: Later, I conclude that this isn't what you're claiming. Rather, you're claiming that CEV is the intersection of all within-group moralities among humans. Implementing CEV won't necessarily fully satisfy all of them, nor even necessarily fully satisfy any of them, it simply won't violate any of them. That is, it may not do anything right, but it's guaranteed not to do anything wrong.

• We therefore want to ensure that any system X powerful enough to impose its preferences on us also implements CEV. This will ensure that it is at least meaningful for us to communicate with it using M-labels... e.g., talk about whether a given course of action is right or wrong. (5)

• Other optimization processes (like the Pebblesorters, or natural selection) might implement an M-instance that is inconsistent with CEV. There is no guarantee that implementing CEV will satisfy all within-group moralities among all sapient species, let alone among all optimization processes. Edit: It seems to be important to you that we not call M-instances within nonhuman species "moralities", though I haven't quite understood why.

• There might exist an M-instance at-least-partially implemented (1) by all sapient species. We could label this the Universal Coherent Extrapolated Volition (U-CEV) and desire to ensure that X also implements U-CEV. (7)

=== (1) Note that implementing a shared M doesn't necessarily mean B1 and B2 can apply M consistently to a specific situation, any more than knowing what a prime number is means I can always recognize or calculate one. It also doesn't mean B1 and B2 can articulate M. It doesn't even guarantee that any given M-label will be correctly or consistently used and understood when they converse. All of this means it may be difficult in practice to determine whether B1 and B2 share an M, or what that M might be.

(2) I'm not sure if this is quite what is being asserted... there may be humans excluded from this formulation, such as psychopaths who would refuse treatment.

(3) I haven't seen this actually being asserted, but it seems implicit. Otherwise, we shouldn't expect CEV to converge and include everybody's volition. (2) above seems relevant here.

(4) We're deliberately ignoring issues of language here and doing everything in English, but we expect that other human languages are isomorphic to English in relevant respects.

(5) There seems to also be an expectation that this is sufficient to avoid X doing bad things to us. I don't quite follow that leap, but never mind that for now.

(6) I don't actually see why I should believe that any such thing exists, though it would be nice if it did. Presumably arguments for this are coming. Edit: Given the "intersection not superset" correction above, then this definitely exists, and there's good reason to believe it's non-empty. Whether it's useful once we leave out all the stuff anyone disagrees with is still unclear to me.

(7) Although apparently we don't, judging from what I've seen so far... either we don't believe U-CEV exists, or we don't care. Presumably arguments for this are coming, as well.

So I really must deny the charges of moral relativism: I don't think that human morality is arbitrary at all, and I would expect any logically omniscient reasoner to agree with me on that. We are better than the Pebblesorters, because we care about sentient lives, and the Pebblesorters don't. Just as the Pebblesorters are p-better than us, because they care about pebble heaps, and we don't. Human morality is p-arbitrary, but who cares? P-arbitrariness is arbitrary.

Is the Logically Omniscient Reasoner agreeing that human morality is not h-arbitrary, or that it is not lor-arbitrary?

How do we know that The LOR (ha! walked into that one) isn't a Pebblesorter?

What p-bothers me (sorry couldn't resist!) about this approach is that "rightness" nowhere explicitly refers to "others", i.e. other conscious beings / consciousness-moments. Isn't there an interesting difference between a heap of eight pebbles (very p-bad) and a human getting tortured (very bad)? Concerning the latter, we can point to that human's first-person-perspective directly evaluating its current conscious state and concluding that the state is bad, i.e. that the person wants to get the hell out of it. This is a source of disvalue, an unfulfilled "want" -- for the "other" concerned -- which exists independent from what we, or the pebblesorters, might consider to be "moral". A heap of pebbles doesn't do anything like that, there is nothing which could be bad for it, and it seems puzzling why the mere existence of a heap of pebbles would be bad in any meaninful sense, all else being equal. The heap of pebbles is only bad for some being if the being is aware of the heap of pebbles and for whatever reason reacts aversively to it. Whereas being tortured, or suffering (defined as unfulfilled desires) is always bad for the being.

Maybe Eliezer's account already incorporates this implicitly, assuming that most humans terminally care about others. But as I said, it bothers me that this isn't made explicit. If it were, I think the quasi-relativist conclusion of this post would be less disturbing. If some heaps of pebbles are p-bad, that needn't bother us because the pebblesorters don't care about others, so they're egoists and not ethical, and even though they use analogous ways to label their terms like "p-ethical" doesn't imply that they compute their ethics according to the same content-criteria (others matter!) as we do.

You, the human, might say we really should pursue beauty and laughter and love (which is clearly very important), and that we p-should sort pebbles (but that doesn't really matter). And that our way of life is really better than the Pebblesorters, although their way of life has the utterly irrelevant property of being p-better.

But the Pebblesorters would say we h-should pursue beauty and laughter and love (boring!), and that we really should sort pebbles (which is the self-evident meaning of life). Further, they will say their way of life is really better than ours, even though ours has some stupid old h-betterness.

I side with you the human, of course, but perhaps it would be better (h-better and p-better) to say we are only h-right, not right without qualification. Of course, from the inside of our minds it feels like we are simply right. But the Pebblesorters feel the same way, and if they're as metaethically astute as us then it seems they are not more likely to be wrong than us.

For what it's worth, my ethic is "You should act on that set of motives which leads to the most overall value." (Very similar to Alonzo Fyfe's desirism, although I define value a bit differently.) On this view, we should pursue beauty and laughter and love, while the Pebblesorters should sort pebbles, on the same definition of "should."

EDIT: Upon reading "No License To Be Human" I am embarrassed to realize my attempted-coining of the term "h-should" in response to this is woefully unoriginal. Every time I think I have an original thought, someone else turns out to have thought of it years earlier!