Is the orthogonality thesis at odds with moral realism?

What the SEP actually says is, "Moral realists are those who think that, in these respects, things should be taken at face value—moral claims do purport to report facts and are true if they get the facts right," and that's it.

This is all a matter of misunderstanding the meaning of words, and nobody is objectively right or wrong about that, since the disagreement is widespread - I'm not the only one to complain.

To me, an unqualified "fact" is, by implication, a simple claim about the universe, not a fact about the person holding the belief in that fact. An unqualified "fact" should be true or false in itself, without requiring you to further specify you meant the instance-of-that-fact that applies to some particular person with particular moral beliefs.

If SEP's usage of "fact" is taken to mean "a fact about the person holding the moral belief", the fact being that the person does hold that belief, then I don't understand what it would mean to say that there aren't any moral facts (i.e. moral anti-realism). Would it mean to claim that people have no moral beliefs? That's obviously false.

On Eliezer's view, as I understand it, human!morality just is morality, simpliciter.

That's exactly what bothers me - that he (and other people agree with this) redefines the word "morality" to mean human!morality, and this confuses people (I'm not the only one) who expect that word to mean something else, depending on context. (For example, the meta-concept of morality, as opposed to a concrete set of moral beliefs such as Eliezer!morality or humanity!morality.)

I agree that if everyone agreed to Eliezer's usage, then discussing morality would be easier. But it's just a fact that many people use the word differently from him. And when faced with such inconsistency, I would prefer that people either always qualify their usage, or taboo the word entirely.

The SEP doesn't say this.

Yes, it does. But it says it in the article Moral Anti-Realism, not the article cited above, Moral Realism. The former article is very interested in objectivity constraints, but expresses a great deal of confusion about how to make sense of them; the latter article mentions them only to toss them out for being too confused. (It would not be too surprising if this has something to do with the latter author being more convinced of the truth of 'realism', hence wanting to make the Realism brand simple, clean, and appealing to a wider audience.)

If your encyclopedia has an 'Apples' article and a 'Non-Apples' article, and the two articles completely disagree about what it means to be an 'Apple', then you have your first clue that the word 'Apple' should always come pre-tabooed.

(ETA: More generally, be aware that 'the SEP says X' is less reliable than 'SEP article Y says X', because articles may disagree with each other. SEP is an anthology of introductory essays. We wouldn't normally say 'Very Short Introductions says X', even if we trust the VSI brand quite a bit.)

What the SEP actually says is, "Moral realists are those who think that, in these respects, things should be taken at face value—moral claims do purport to report facts and are true if they get the facts right," and that's it.

Almost. Moral realists (even on the more inclusive definitions) also demand that at least one moral claim of this sort be true. (This is asserted in the sentence right after your quotation terminates.) That's why error theory is not a form of moral realism; realism is a (perhaps improper) subset of success theory.

Oh, well, that makes some sense, actually. Since everybody knows that "cognitivism" means that moral statements have truth-values, whereas "realism" seems to be a confused notion - I actually interpreted it to mean the same thing as cognitivism because otherwise I don't know what on earth realism should even be.

I thought that when humans and Clippy speak about morality, they speak about the same thing (assuming that they are not lying and not making mistakes).

The difference is in connotations. For humans, morality has a connotation "the thing that should be done". For Clippy, morality has a connotation "this weird stuff humans care about".

So, you could explain the concept of morality to Clippy, and then also explain that X is obviously moral. And Clippy would agree with you. It just wouldn't make Clippy any more likely to do X; the "should" emotion would not get across. The only result would be Clippy remembering that humans feel a desire to do X; and that information could be later used to create more paperclips.

Clippy's equivalent of "should" is connected to maximizing the number of paperclips. The fact that X is moral is about as much important for it as an existence of a specific paperclip is for us. "Sure, X is moral. I see. I have no use of this fact. Now stop bothering me, because I want to make another paperclip."

According to his theory, when you use the word "morality", you refer to "Human!morality", and there are objective facts about that.

If this is a theory about what people mean when they say "morality", then he is wrong about a significant percentage of people, as a matter of simple fact.

Group-level moral relativism just is the belief that moral truths are indexed to groups. Since relativism is uncontroversially opposed to realism, "indexical realist" is a bit of a contradiction.

'The SEP says' is also a mistake. The article you linked to defines 'moral realism' one way; the article on moral anti-realism defines it in a completely different way.

Thank you for pointing this out.

For the rest, please see my response here.

The sentence is question is either false or true and uninteresting depending on exactly what is meant by "necessarily motivate".

For the most obvious interpretation, it is uninteresting since even modus ponens doesn't necessarily motivate.

See Motivational internalism/externalism (you might get better quality results if you asked specifically 'is motivational internalism true?' and provided that link; it's basically the same as what you asked but less open to interpretation.)

My personal understanding is that motivational internalism is true in proportion to the level of systematization-preference of the agent. That is, for agents who spend a lot of time building and refining their internal meaning structures, motivational internalism is more true (for THEM, moral judgements tend to inherently motivate); in other cases, motivational externalism is true.

I have weak anecdotal evidence of this (and also of correlation of 'moral judgements inherently compel me' with low self worth -- the 'people who think they are bad work harder at being good' dynamic.)

TL;DR: My impression is that motivational externalism is true by default (I answered 'No' to your poll); And motivational internalism is something that individual agents may acquire as a result of elaborating and maintaining their internal meaning structures.

I would argue that acquiring a degree of motivational internalism is beneficial to humans. but it's probably unjustifiable to assume either that a) motivational internalism is beneficial to AIs, or b) if it is, then an AI will necessarily acquire it (rather than developing an alternative strategy, or nothing at all of the kind).

YES! I think this is exactly right: moral realism is not at odds with the orthogonality thesis; but the conjunction of moral realism with moral internalism is.

And it is this conjunction that many people seem to believe, although I cannot see why, because I can't even imagine what it would mean for the world to be such that it is true. So I find it obvious that the conjunction isn't true. It's not quite so clear which of the conjuncts is false (if not perhaps both), though.

This is more of a question about what qualifies as a moral judgment. It's possible to make moral judgments (under one definition) from the outside about other systems of morality or other people's utility functions, e.g. "According to Christianity, masturbation is a sin" doesn't motivate you to stop masturbating unless you firmly believe in Christianity, and "According to Bob's utility function, he should donate more to charity" needn't motivate you to donate more to charity. On the other hand, it's impossible to believe "According to my moral system, I should do X" and not think X is the right thing for you to do.

Do 'moral judgments necessarily motivate'

On the one hand, nothing is necessarily true about an arbitrary mind, because nothing is true about all minds, for the same reason that there are no universally compelling arguments.

On the other hand, this is just another disagreement about what words refer to: someone who says "moral judgments necessarily motivate" is just saying "a judgement that does not motivate, does not fit my definition of moral". This is not a fact about the world or about morality, it's a fact about the way that person uses the words "moral judgment".

If there is indeed wide disagreement on the answer to this question - I write this before voting and haven't seen the results yet - then that is yet another argument for tabooing the word "morality".

Is this a poll about whether moral judgements necessarily motivate, or whether that's the key question?

Would people (especially those who haven't read philosophical background) say what they think this question means. I suspect giant misinterpretation.

You can prove everything from a contradiction, but you can't prove everything from a false premise. I take it that you mean that we can derive a contradiction from the assumption of moral realism. That may be true (although I'd hesitate to call either moral realism or free will logically impossible), but I doubt many arguments from moral realism to other claims (e.g. the denial of the orthogonality thesis) rely on the derivation of a contradiction as an intermediate step.

I wasn't making an argument against moral realism in the sentence you quoted.

What consequences? That claim is badly in need of support.

No, it isn't. It's Less Wrong/MIRI boilerplate. I'm not really interested in rehashing that stuff with someone who isn't already familiar with it.

The (possible extreme) difficulty of figuring out objective morality in a way that can be coded into an AI is not an argument against moral realism. If it were, we would have to disbelieve in language, consciousness and other difficult issues.

The question was "is the orthogonality thesis at odds with moral realism?". I answered: "maybe not, but moral anti-realism is certainly closely aligned with the orthogonality thesis-- it's actually a trivial implication of moral anti-realism."

If you are concerned that people aren't taking the orthogonality thesis seriously enough then emphasizing that there is as much evidence for moral realism as there is for God is a pretty good way to frame the issue.

The reaction to the supposed doctrine of the "Double Truth" illustrates this quite well, even if it's not quite the right timeframe. This doctrine was supposed (though we don't know if correctly) to be a doctrine that although reason dictated truths contrary to faith, people are obliged to believe on Faith anyway. It was supressed.

I'm aware of the existence of the Summa.

And yet, you claim that "philosophers could used whatever half-baked premises they wanted in constructing arguments for the existence of God, and have little fear of being contradicted" even though the Summa contains refutations of weak arguments for the existence of God. Also, The Church specifically denounced the Doctine of the Double Truth, which by all accounts is a premise that would, in practice, act to protect religious claims from falsification. "Philosophers" would have risked Inquisitional investigation had they not dropped their "half-baked premises they wanted in constructing arguments for the existence of God".

I admit I was mostly thinking of the 17th/18th centuries when I wrote the above paragraph... but it was dangerous to be a heretic in the 13th century too.

I don't think he is claiming it wasn't dangerous to be a heretic in the 13th century. I'm pretty sure he is calling into question the claim that "it was dangerous to question that the existence of God could be proven through reason", which was a very common belief throughout most of the middle ages and was held with very little danger as far as I can tell. I'm surprized that you are unaware of this given that you "have master's degree in philosophy from Notre Dame".

EDIT: Carinthium beat me to the punch.

I usually try to avoid the term "moral realism" - due to associated ambiguities - and abuse of the term "realism".

Perhaps the word 'Error' is inappropriate and should be replaced with 'misunderstanding'. I don't mean to say that one or the other viewpoint is obviously correct, I'm just trying to point out a possible source of confusion. I did mention that it was just a guess.

That said, it's possible that the root misunderstanding here is simple and obvious. No need to assume it must be complicated.