Why didn't people (apparently?) understand the metaethics sequence?

Good [1] : The human consensus on morality, the human CEV, the contents of Friendly AI's utility function, "sugar is sweet, love is good". There is one correct definition of Good. "Pebblesorters do not care about good or evil, they care about grouping things into primes. Paperclippers do not care about good or evil, they care about paperclips".

Good[2] : An individual's morality, a special subset of an agent's utility function (especially the subset that pertains to how everyone aught to act). "I feel sugar is yummy, but I don't mind if you don't agree. However, I feel love is good, and if you don't agree we can't be friends."... "Pebblesorters think making prime numbered pebble piles is good. Paperclippers think making paperclips is good". (A pebblesorter might selfishly prefer maximize the number of pebble piles that they make themselves, but the same pebblesorter believes everyone aught to act to maximize the total number of pebble piles, rather than selfishly maximizing their own pebble piles. A perfectly good pebblesorter seeks only to maximize pebbles. Selfish pebblesorters hoard resources to maximize their own personal pebble creation. Evil pebblesorters knowingly make non-prime abominations.)

so I don't see where the confusion is.

Do you see what I mean by "semantic" confusion now? Eliezer (like most moral realists, universalists, etc) is using Good[1]. Those confused by his writing (who are accustomed to descriptive moral relativism, nihilism, etc) are using Good[2]. The maps are actually nearly identical in meaning, but because they are written in different languages it's difficult to see that the maps are nearly identical.

I'm suggesting that Good[1] and Good[2] are sufficiently different that people who talk about morality often aught to have different words for them. This is one of those "If a tree falls in the forest does it make a sound" debates, which are utterly useless because they center entirely around the definition of sound.

Eliezer views ethics the same way just about everyone intuitively view aesthetics -- as a body of facts that can be empirically studied and are not purely a matter of personal opinion or ad-hoc stipulation -- facts, though, that make ineliminable reference to the neurally encoded preferences of specific organisms, facts that are not written in the sky and do not possess a value causally independent of the minds in question.

Yup, I agree completely, that's exactly the correct way to think about it. The fact that you are able to give a definition of what ethics are while tabooing words like good and bad and moral, is the reason that you can simultaneously uphold Good[2] with your gustatory analogy and still understand that Eliezer doesn't disagree with you even though he uses Good[1].

Most people's thinking is too attached to words to do that, so they get confused. Being able to think about what things are without referencing any semantic labels is a skill.

I raise the analogy because it seems an obvious one to me, so I don't see where the confusion is.

Your analysis clearly describes some of my understanding of what EY says. I use "yummy" as a go to analogy for morality as well. But, EY also seems to be making a universalist argument, as least for "normal" humans. Because he talks about abstract computation, leaving particular brains behind, it's just unclear to me whether he's a subjectivist or a universalist.

The "no universally compelling argument" applies to Clippy versus us, but is there also no universally compelling argument with all of "us" as well?

What do you mean? Rational agents aught to converge upon what physics is.

By Eliezer's usage? I'd say aliens might have love and pleasure in the same way that aliens might have legs...they just as easily might not. Think "wolf" vs "snake" - one has legs and feels love while the other does not.

I'm ambiguous about it because I'm describing EY's usage of the word, and he's been ambiguous about it.

I typically adapt my usage to the person who I'm talking to, but the way that I typically define "good" in my own head is: "The subset of my preferences which do not in any way reference myself as a person"...or in other words, the behavior which I would prefer if I cared about everyone equally (If I was not selfish and didn't prefer my in-group).

Under my usage, different people can have different conceptions of good. "Good" is a function of the agent making the judgement.

A pebble-sorter might selfishly want to make every pebble pile themselves, but they also might think that increasing the total number of pebble piles in general is "good". Then, according to the Pebblesorters, a "good" pebble-sorter would put overall-prime-pebble-pile-maximization above their own personal -prime-pebble-pile-productivity. According to the Babyeaters, "good" baby-eater would eat babies indiscriminately, even if they selfishly might want to spare their own. According to humans, Pebble sorter values are alien and baby-eater values are evil.

Eliezer's "metaethics" sequence, despite its name, argues for his ethical theory

Yes; what else would you do in metaethics?
Isn't its job to point to ethical theories, while the job of ethics is to assume you have agreed on a theory (an often false assumption)?

I would be surprised if Eliezer believed (1) or (2), as distinct from believing that CEV[X] is the most viably actionable approximation of morality[X] (using your terminology) we've come up with thus far.

This reminds me somewhat of the difference between believing that 2013 cryonics technology reliably preserves the information content of a brain on the one hand, and on the other believing that 2013 cryonics technology has a higher chance of preserving the information than burial or cremation.

I agree that that he devotes a lot of time to arguing against (3), though I've always understood that as a reaction to the "but a superintelligent system would be smart enough to just figure out how to behave ethically and then do it!" crowd.

I'm not really sure what you mean by (4).

I don't think you're "confused" about what was meant. I think you understood exactly what was meant, and have identified a real (and, I believe, acknowledged?) problem with the moral realist definition of Good.

The assumption is that "if we knew more, thought faster, were more the people we wished we were, had grown up farther together” then a very large number of humans would converge onto moral agreement.

It's not the assumption that is good or bad, but the quality of argument provided for it.

Alternatively, we could bite the bullet and just say that some humans simply end up with alien values that are not "good",

Seeing as about 1% of the population are estimated to be psychopaths, not to mention pathological narcissists megalomaniacs etc, it seems hard to argue that there isn't a large (if statistically insignificant) portion of the population who are natural ethical egoists rather than altruists. You could try to weasel around it like Mr Yudkowski does, saying that they are not "neurologically intact," except that there is evidence that psychopathy at least is a stable evolutionary strategy rather than a malfunction of normal systems.

I'm usually not one to play the "evil psychopaths" card online, mainly because it's crass and diminishes the meaning of a useful medical term, but it's pretty applicable here. What exactly happens to all the psychopaths and people with psychopathic traits when you start extrapolating human values?

Yeah, I meant to remain ambiguous about how wide Eliezer means to cast the net around agents. Maybe it's psychologically normal humans, maybe it's wider or narrower than that.

I suppose 'The psychological unity of humankind' is sort of an argument that value convergence is likely at least among humans, though it's more like a hand-wave. In response, I'd hand-wave toward Sobel (1999); Prinz (2007); Doring & Steinhoff (2009); Doring & Andersen (2009); Robinson (2009); Sotala (2010); Plunkett (2010); Plakias (2011); Egan (2012), all of which argue for pessimism about value convergence. Smith (1994) is the only philosophical work I know of that argues for optimism about value convergence, but there are probably others I just don't know about.

No Universally Compelling Arguments has been put to me as a decisive refutation of Moral Realism, by somebody who thought the LW line was anti-realist. It isn't a decisive refutation because no (non strawman) realist thinks there are arguments that could compel an irrational person, an insane person, an very unintelligent person, and so on. Moral realists only need to argue that moral truths are independently discoverable by suitably motivated and equipped people, like mathematical truths (etc).

When you say "agents" here, did you mean to say "psychologically normal humans"? Because the general claim I think Eliezer would reject, based on what he says on No Universally Compelling Arguments.

Well, "No Universally Compelling Arguments" also applies to physics, but it is generally believed that all sufficiently intelligent agents would agree on the laws of physics.

EDIT: Here is the other highly relevant post I was thinking of.

I think what he is talking about there is lack of disagreement in the sense of incommensurability, or orthogonality as it is locally known. Lack of disagreement int he sense of convergence or consensus is a very different thing.

I find moral realism meaningful for each individual (you can evaluate choices according to my values applied with infinite information and infinite resources to think),

I can how that could be implemented. However, I don't see how that would count as morality. It amounts to Anything Goes, or Do What Thou Wilt. I don't see how a world in which that kind of "moral realism" holds would differ from one where moral subjectivism holds, or nihilism for that matter.

but I don't find it meaningful when applied to groups of people, all with their own values.

Where meaningful means implementable? Moral realism is not many things, and one of the things it is not is the claim that everyone gets to keep all their values and behaviour unaltered.

Getting round to metaethics should rank on Lukeprog's priorities: [pollid:573]

Creating a "balance Karma" post. Asking people use this for their conventional Karma for my above post, or to balance out upvotes/downvotes. This way my Karma will remain fair.

This is an excellent description of the argument.

Here is my question: Why bother with the middle man? No one can actually define good and everyone is constantly checking with 'human values' to see what it says! Assuming the universe runs on math and humans share attitudes about some things obviously there is some platonic entity which precisely describes human values (assuming there isn't too much contradiction) and can be called "good". But it doesn't seem especially parsimonious to reify that concept. Why add it to our ontology?

It's just semantics in a sense: but there is a reason we don't multiply entities unnecessarily.

Yes, sorry, I wasn't clear enough about that. No, let me go further; what I wrote was downright misleading. This is why I shouldn't write Less Wrong comments on a tablet where I am too strongly incentivized to make them brief :-). I endorse your description of Eliezer's position.

This is the definition you get if you say "let 'good' mean 'human values'". But the actual idea is meant to be more analogous to this: assuming for the sake of argument that humans value cake, define "good" to mean cake. Obviously, under that definition, "cake is always good regardless of what humans value" is true. In that case "good" is a rigid designator for cake.

Why is cake a referent of good?

The difference is that "good" and "human values" are not synonymous. But they refer to the same thing, when you fully dereference them, namely {happiness, fun and so forth}. This is the difference between sense and reference, and it's why it is necessary to understand rigid designators.

And what happened to the normativity of Good? Why does it appear to make sense to wonder if we are valuing the right things, when Good is just whatever we value?

ADDED:

The reason Eliezer's views are commonly mistaken for relativism in the manner you describe is because most people do not have a good grasp on the difference between sense and reference(a difference that, to be fair, doesn't seem to be well explained anywhere).

I don't see the S/R difference is relevant to relativism. If the referents of "good" vary with the mental contents of the person saying "good", that is relativism/subjectivism. (That the values referenced are ultimately physical does not affect that: relativism is an epistemological claim, not a metaphysical one).

Maybe Eliezer would say that he is a "subjective realist" or something like that. This is strictly different from moral relativism, where choice of morality is more or less arbitrary. As a subjective realist your morality is different than your pebblesorter friend, but it's not arbitrary.

Philosophers just call this position "moral subjectivism". Moral realism is usually defined to exclude it. "Relativism" at this point should be tabooed since no one uses it in the technical sense and the popular sense includes a half dozen variations which are very different from one another to the extent they have been defined at all.

cognitivist, which is often grouped against realism

Is it? That seems backwards to me: non-cognitivism is one of the main varieties of non realism. (The other being error theory.) What am I missing?

1- There are inevitable conflicts between practically any two creatures on this planet as to what preferences they would have as to the world. If you narrow these down to the area classified by humans as "moral" the picture can be greatly simplified, but there will still be a large amount of difference.

Sure, as I said at the end, the "universality" of the whole thing is an open problem.

I dispute that moral sentences ARE about the attitudes of people. Most people throughout history have had a concept of "Right" and "Wrong" as being objective. This naive conception is philosophically indefensible, but the best descriptor of what people throughout history, and even nowadays, have believed. It is hard to defend the idea that a person thinks they are referring to X and are in fact referring to Y when X and Y are drastically different things and the person is not thinking of Y on any level of their brain- the likely case for, say, a typical Stone Age man arguing a moral point.

That's fine. But in that case, all moral sentences are false (or nonsense, depending on how you feel about references to non-entities). I agree that there is a sense in which that is true which you outlined here. In this case we can start from scratch and just make the entire enterprise about figuring out what we we really truly want to do with the world-- and then do that. Personally I find that interpretation of moral language a bit uncharitable. And it turns out people are pretty stuck on the whole morality idea and don't like it when you tell them their moral beliefs are false.

Subjectivism seems both more charitable and friendlier-- but ultimately these are two different ways of saying the same thing. The debates between varieties of anti-realism seem entirely semantic to me.

I understand, this is what I'm dealing with in the second to last paragraph.

I can see how you can sort of structure the arguments and questions and get it to output "moral realism" if you really had to. You say that the word "right" designates particular facts about worlds such that worlds can be objectively evaluated according to that concept. But to me, it is weird and confusing to ignore the fact that the rule uniting those facts about the world is determined by our attitudes-- especially since we can't right now enumerate the rigid contents of our moral language and have to apply the rule in most circumstances.

There is a sense in which all concepts both exist subjectively and objectively. There is some mathematical function that describes all the things that ChrisHallquist thinks are funny just like there is a mathematical function that describes the behavior of atoms. We can get into the nitty-gritty about what makes a concept subjective and what makes a concept objective. But I don't see what the case for morality counting as "objective" is unless we're just going to count all concepts as objective.

I am a moral subjectivist and a moral realist.

Traditional usage defines those terms to exclude the possibility of being both. The standard definition of a moral realist is someone who believes that moral judgments express mind-independent facts; while the standard definition of a moral subjectivist is someone who believes moral judgments express mind-dependent facts.

So I don't know quite what you mean.

a non psycho human moral universalist.

You mean someone who doesn't believe that there are moral universals among humans? One too many adjectives for me.

And I don't think you get a compelling answer from some smuggled self recursion, but from your best estimate of what people actually are.

If I understand this right: you're contrasting trying to come up with some self-justifying method for resolving disagreement (recursively finding consensus on how to find consensus) with... descriptive moral psychology? I'm not sure I follow.

But if\ we're going to be moral subjectivists, we should realize that "we ought" too easily glosses over the fact that ought_you is not identical to ought_me.

Does the fact that people have different opinions about non moral claims, means there are no objective, scientific facts?

Multiple philosophers have suggested that this stance seems similar to "rigid designation", i.e., when I say 'fair' it intrinsically, rigidly refers to something-to-do-with-equal-division. I confess I don't see it that way myself - if somebody thinks of Euclidean geometry when you utter the sound "num-berz" they're not doing anything false, they're associating the sound to a different logical thingy. It's not about words with intrinsically rigid referential power, it's that the words are window dressing on the underlying entities. I want to talk about a particular logical entity, as it might be defined by either axioms or inchoate images, regardless of which word-sounds may be associated to it. If you want to call that "rigid designation", that seems to me like adding a level of indirection; I don't care about the word 'fair' in the first place, I care about the logical entity of fairness. (Or to put it even more sharply: since my ontology does not have room for physics, logic, plus designation, I'm not very interested in discussing this 'rigid designation' business unless it's being reduced to something else.)

He seems to have thought Rigid Designation was about a magic connection between sound wave patterns and objects, such that the sound waves would always refer to the same object, rather than that those sound waves, when spoken by such a speaker in such a context, would always refer to the same object, regardless of which possible world that object was in.

I'm sorry if that explanation was a little unclear; it was aimed at non-philosophers, but I suspect you could explain it better.

EDIT: see also prior discussion

We are better than the Pebblesorters, because we care about sentient lives, and the Pebblesorters don't.

Translated:

Humans are preferable to Pebblesorters according to human utility function, because humans care about maximizing human utility function, and the Pebblesorters don't.

But... it looks like he's implying that "h-right" is special among "right"s in that it can't be something else in another universe, but that looks wrong for simple reasons. It's also not obvious to me that h-right is a narrow cluster.

It's because you're a human. You can't divorce yourself from being human while thinking about morality.

Datapoint: I didn't find Metaethics all that confusing, although I am not sure I agree with it.

It looks like he's implying that "h-right" is special among "right"s in that it can't be something else in another universe, but that looks wrong for simple reasons. It's also not obvious to me that h-right is a narrow cluster.

I had this impression too, and have more or less the same sort-of-objection to it. I say "sort of" because I don't find "h-right as a narrow cluster" obvious, but I don't find it obviously wrong either. It feels like it should be a testable question but I'm not sure how one would go about testing it, given how crap humans are at self-reporting their values and beliefs.

On edit: Even if h-right isn't a narrow cluster, I don't think it would make the argument inconsistent; it could still work if different parts of humanity have genuinely different values modeled as, say, h1-right , h2-right, etc. At that point I'm not sure the theory would be all that useful, though.

Yes: valuing something implies that I value it, and believing something doesn't imply that it's true. Agreed.

I assume you're trying to imply that there exists some X that bears the same kind of relationship to valuing that truth has to belief, and that I'm making an analogous error by ignoring X and just talking about value as if I ignored truth and just talked about belief.

Then again, maybe not. You seem fond of making these sorts of gnomic statements and leaving it to others to unpack your meaning. I'm not really sure why.

Anyway, if that is your point and you feel like talking about what you think the X I'm illegitimately ignoring is, or if your point is something else and you feel like actually articulating it, I'm listening.

I recall not being able to identify with the premises... some of them were really quite significant.

I now recall, it was with "The Moral Void, in which apparently I had different answers than expected.

"Would you kill babies if it was inherently the right thing to do?"

The post did discuss morality on/off switches later in the context of religion, as an argument against (wishing for / wanting to find) universally compelling arguments.

The post doesn't work for me because it seems there is an argument against the value of universally compelling arguments with the implicit assumption that since universally compelling argument don't exist, any universally compelling argument would be false.

I happen to (mostly) agree that there aren't universally compelling arguments, but I still wish there were. The metaethics sequence failed to talk me out of valuing this.

Also, there were some particular examples that didn't work for me, since I didn't have a spontaneous 'ugh' field around some of the things that were supposed to be bad.

I see Jack expressed this concept here:

And it definitely is true that much of our moral language function like rigid designators, which hides the causal history of our moral beliefs. This explains why people don't feel like morality changes under counterfactuals-- i.e. if you imagine a world in which you have a preference for innocent children being murdered you don't believe that murdering children is therefore moral in that world. I outlined this in more detail here. I didn't use the term 'rigid designator' in that post, but the point is that what we think is moral is invariant in counterfacturals.

For whatever reason, I feel like my morality changes under counterfactuals.

That's an open-ended question which I don't have many existing public resources to address - but thanks for your interest. Very briefly:

• I like evolution, Yukdowsky seems to dislike it. Ethically, Yukdowsky is an intellectual descendant of Huxley, while I see myself as thinking more along the lines of Kropotkin.

• Yukdowsky seems to like evolutionary psychology. So far evolutionary psychology has only really looked at human universals. To take understanding of the mind further, it is necessary to move to a framework of gene-meme coevolution. Evolutionary psychology is politically correct - through not examining huamn differences - but is scientifically very limited in what it can say, because of the significance of cultural transmission on human behaviour.

• Yudkowsky likes utilitarianism. I view utilitarianism largely as a pretty unrealistic ethical philosophy adopted by ethical philosophers for signalling reasons.

• Yukdowsky is an ethical philosopher - and seems to be on a mission to persuade people that giving control a machine that aggregates their preferences will be OK. I don't have a similar axe to grind.

Agreed this is a good idea to prevent illusion of transparency.

I reaaally wish that were true.

Yes; if you click the "Show Help" button below the bottom right of the comment box, and then click the Polls Help link, you will find details about how to code polls.

Because a large subset of people who don't understand things are unaware of their misunderstanding?

I think that framing the issue of AI safety in terms of "morality" or "friendliness" is a form of misleading anthropomorphization. Morality and friendliness are specific traits of human psychology which won't necessarily generalize well to artificial agents (even attempts to generalize them to non-human animals are often far-fetched).
I think that AI safety would be probably best dealt with in the framework of safety engineering.

although humans are free-willed, some actions do or do not contribute to achieving a human's natural purpose in life. Said purpose is meant to be coherent, unlike evolutionary purpose- so better achievement would lead to achieving satisfaction in the long run

If I understand you correctly, your claim that if this turns out to be true, then I ought to perform those acts which contribute to achieving my natural purpose, whether I net-value satisfaction or not. Yes?

When there is a conflict between two desires both which feel like they have some claim to moral rightness, correct metaethics is essential to sort out what best to do.

Is it? It seems like object-level ethics achieves this purpose perfectly well. If it returns the result that they are equally good to do, then the correct thing to do is pick one. What do I need metaethics for, here?