Solid and unsurprising.

I feel like your austere meta-ethicist is mostly missing the point. It's utterly routine for different people to have conflicting beliefs about whether a given act is moral*. And often they can have a useful discussion, at the end of which one or both participants change their beliefs. These conversations can happen without the participants changing their definitions of words like 'moral', and often without them having a clear definition at all.

[This is my first LW comment -- if I do something wrong, please bear with me]

This suggests that precise definitions or agreement about definitions isn't all that critical. But it's sometimes useful to be able to reason from stipulated and mutually agreed definitions, in which case meta-ethical speculation and reasoning is doing useful work if it offers a menu of crisp, useful, definitions that can be used in discussion of specific moral claims. Relatedly, it's also doing useful work by offering a set of definitions that help people conceptualize and articulate their personal feelings about morality, even absent a concrete first-order question.

And part of what goes into picking definitions is to understand their consequences. A philosopher is doing useful work for me if he shows me that a tempting-sounding definition of 'morality' doesn't pick out the set of things I want it to pick out, or that some other definition turns out not to refer to any clear set at all.

Many mathematical entities have multiple logically equivalent definitions, that are of different utility in different contexts. (E.g., sometimes I want to think about a circle as a locus of points, and sometimes as the solution set to an equation.) In the real world, something similar happens.

When I discuss, say, abortion, with somebody, probably there are multiple working definitions of 'moral' that could be mutually agreed upon for the purpose of the conversation, and the underlying dispute would still be nontrivial and intelligible. But some definitions might be more directly applicable to the discussion -- and philosophical reasoning might be helpful in figuring out what the consequences of various definitions are. For instance, a non-cognitive strikes me intuitively as less likely to be useful -- but I'd be open to an argument showing how it could be useful in a debate.

Probably a great deal of academic writing on meta-ethics is low value. But that's true of most writing on most topics and doesn't show that the topic is pointless. (With academics being major offenders, but not the only offenders.)

*I'm thinking of the individual personal changes in belief that went along with increased opposition to official racism in America over the course of the 20th century. Or opposition to slavery in the 19th.

When I have serious conversations with thoughtful religious people who have faith but no major theological training, I find it helpful to think of their statements about "God" as being statements about "all worldly optimization processes stronger than me that I don't have time to understand in very much detail like evolution, entropy, economics, democratic politics, organizational dynamics, similar regularities in the structure of the world that science hasn't started analyzing yet, plus many small activist groups throughout history, and a huge number of specific powerful agents silently influencing my life right now like various investors and celebrities, the local chief of police, the local school principal, my employer, my ancestors, and so on".

I can imagine a relatively simple life heuristic, H, that might successfully navigate this vast and bewildering array of optimization pressures in their life and can ask "Does God want you to H". Also, this translation scheme helps me to listen to evangelical radio and learn things from it :-)

I bring this up because it feels to me like you're doing a lot of work to resuscitate ideas from moral philosophy that are significantly helped by "pluralistic reduction" to unpack the ideas into more specific and coherent claims, but you seem to be doing it in a lop-sided way by not unpacking the ideas of "the other side" in a similarly generous manner. Also, to a lesser extent, it seems to be leaving some of "our" ideas unpacked that could probably use some pluralistic reduction but might not look as shiny if unpacked this way.

I guess what I'm trying to say is that "acoustic messenger ferries" in the "ether" seem to me like perfectly adequate placeholder terms if I'm in a conversation with someone whose starting vocabulary uses them as atomic concepts, but if I'm tossing those terms out then "an ideally instrumentally rational and fully informed agent" seems roughly as questionable given how much difficulty people seem to have when using mind-shaped conceptually-atomic entities in their theories.

Do you think my impression of lopsided conceptual unpacking is accurate? If yes, I'm wondering if you could try to introspect on your writing process and try to articulate how you decided which things to unpack and which to leave fuzzy.

I don't understand the terms "world of is" and "world of is not". Does "talking about world of is not" mean "deducing from false assumptions", or is there something more to it? Anyway, "talking about world of is" sounds like the worst kind of continental philosophy babble.

Else, the article is clear, comprehensible and well readable.

If we taboo and reduce, then the question of "...but is it good?" is out of place. The reply is: "Yes it is, because I just told you that's what I mean to communicate when I use the word-tool 'good' for this discussion. I'm not here to debate definitions; I'm here to get something done."

I just wanted to flag that a non-reductionist moral realist (like myself) is also "not here to debate definitions". See my post on The Importance of Implications. This is compatible with thinking well of the Open Question Argument, if we think we have an adequate grasp of some fundamental normative concept (be it 'good', 'reason', or 'ought' -- I lean towards 'reason', myself, such that to speak of a person's welfare is just to talk about what a sympathetic party has reason to desire for the person's sake).

Note that if we're right to consider some normative concepts to be conceptually primitive (not analytically reducible to non-normative concepts) then your practice of "tabooing" all normative vocabulary actually has the effect of depriving us of the conceptual tools necessary to even talk about the normative sphere. Consequent talk of people's (incl. God's) desires or dispositions is simply changing the subject, on this way of looking at things.

Out of interest: Will you be arguing anywhere in this sequence against non-reductionist moral realism? Or are you simply assuming its falsity from the start, and exploring the implications from there? (Even the latter, more modest project is of course worth pursing, but I personally would be more interested in the former.) Either way, it'd be good to be clear about this. (You could then skip the silly rhetoric about how what is not "is", must be "is not".)

Tangentially:

facts about the well-being of conscious creatures are mind-dependent facts

How so? (Note that a proposition may be in some sense about minds without its truth value being mind-dependent. E.g. "Any experience of red is an experience of colour" is true regardless of what minds exist. I would think the same is true of, e.g., "All else equal, pain is bad for the experiencer.")

That's why we need to decode the cognitive algorithms that generate our questions about value and morality. ... So how can the Empathic Metaethicist answer Alex's question? We don't know the details yet. For example, we don't have a completed cognitive neuroscience.

Assume you have a complete knowledge of all details of the way human brain works, and a detailed trace of the sequence of neurological events that leads people to ask moral questions. Then what?

My only guess is that you look this trace over using your current moral judgment, and decide that you expect that changing certain things in the algorithm will make the judgments of this brain better. But this is not a FAI-grade tool for defining morality (unless we have to go the uploads-driven way, in which case you just gradually and manually improve humans for a very long time).

The problem is not that there is no way to identify 'good' or 'right' (as used intuitively, without tabooing) with a certain X. The problem is that X is huge and complicated and we don't (yet) have access to its structure.

Strictly speaking, we can exhibit any definition of "good", even one that doesn't make any of the errors you pointed out, and still ask "Is it good?". The criteria for exhibiting a particular definition are ultimately non-rigorous, even if the selected definition is, so we can always examine them further.

Moore's argument might fail in the unintended use case of post-FAI morality not because at some point there might be no more potential for asking the question, but because, as with "Does 2+2 equal 4?", there is a point at which we are certain enough to turn to other projects, even if in principle some uncertainty and lack of clarity in the intended meaning remains. It's not at all clear this will ever happen to morality.

In The Is_Ought Gap, Luke writes

If someone makes a claim of the 'ought' type, either they are talking about the world of is, or they are talking about the world of is not. If they are talking about the world of is not, then I quickly lose interest because the world of is not isn't my subject of interest.

Ironically, this is where I quickly lost interest in this article, because glib word-play isn't my subject of interest.

Consider this dialog:

Student: "Wise master, what ought I do?"

Wise master: "You ought to help the poor by giving 50% of your income to efficient charity and supporting the European-style welfare state."

Student: "Alright."

*student runs off and gives 50% of his or her income to efficient charity and supports the European-style welfare state

This dialog rings true as a fact about ought statements - once we become convinced of them, they do and should constrain our behavior.

But my dialogs and your dialogs contradict each other! Because if "ought" determines our behavior, and we can define what "ought" means, then we can define proper behavior into existence - a construction as absurd as Descartes defining God into existence or Plato defining man as both a hairless featherless biped and a mortal.

We must give up one, and I say give up yours. "ought" is one of those words that we are not free to define - it has a single meaning. Look to its consequences, not its causes.

I have created anchors for the sections, BTW:

Many Moral Reductionisms
Cognitivism vs. Non-Cognitivism
Objective vs. Subjective
Relative vs. Absolute
Moore's Open Question Argument
The Is-Ought Gap
Moral Realism vs. Anti-Realism
Empathic Metaethics

I think you are incorrect with regards to Hume's is-ought gap, although I find its relevance to be somewhat overstated. A hypothetical imperative such as your example relies on an equivocation between 'ought' as (1) a normative injunction and (2) conveying a possible causal pathway from here to there.

-

Here is the incorrect syllogism:

Premise 1: A desires C (is)

Premise 2: B will produce C (is)

Conclusion: A ought to do B (ought)

-

There is a hidden normative premise that is often ignored. It is

Premise 3: A should obtain its desires. (ought)

-

The correct syllogism would then be:

Premise 1 (is): A desires C

Premise 2 (is): B will produce C

Premise 3 (ought): A ought to obtain its desires.

Conclusion: A ought to do B (ought)

-

The necessity of Premise 3 is made clear by use of an admittedly extreme example:

P1: Hitler wants to kill a great number of people

P2: Zyklon B will kill a great number of people

C1: Hitler ought to use Zyklon B to kill a great number of people

While the conclusion is derived from the premises using definition (2) of the word 'ought', few would express it as a normative recommendation.

-

Hume's fact/value dichotomy remains valid. A normative conclusion can only be validly deduced from a group of premises including at least one which is itself normative.

Or, perhaps someone has a moral reductionism in mind during a particular use of 'ought' language. Perhaps by "You ought to be more forgiving" they really mean "If you are more forgiving, this is likely to increase the amount of pleasure in the world."

As you can see, it is not hard to bridge the is-ought gap.

I don't think it is impossible, but it is harder than you are making out. The examples given are not complete syllogisms, or other logical forms. It is easy to validly derive an ought form an is: you start with the factual statement and then invoke a bridging principle of the form:

if then

However, the argument is not sound unless the bridging statement is true. But the bridging statement is itself an derivation of an ought from an is, so there is a kind of circularity there. You are assuming that the ought-from-is problem has been solved in order to solve it.

As I said, I don't think the situation is hopeless. The bridging premise is not exactly the same thing as a moral argument: it is usually more of a general statement along the lines of "if X increases well being, it should be done". That provides some scope for an analytical justification of bridging principles.

[Re-post with correction]

Hi Luke,

I've questioned your metaethical views before (in your "desirist" days) and I think you're making similar mistakes now as then. But rather than rehash old criticisms I'd like to make a different point.

Since you claim to be taking a scientific or naturalized approach to philosophy I would expect you to offer evidence in support of your position. Yet I see nothing here specifically identified as evidence, and very little that could be construed as evidence. I don't see how your approach here is significantly different from the intuition-based philosophical approaches that you've criticised elsewhere.

Some people who say "Stealing is wrong" are really just trying to express emotions: "Stealing? Yuck!" Others use moral judgments like "Stealing is wrong" to express commands: "Don't steal!" Still others use moral judgments like "Stealing is wrong" to assert factual claims, such as "stealing is against the will of God" or "stealing is a practice that usually adds pain rather than pleasure to the world."

How do you know this? Where's the evidence? I don't doubt that some people say, "Stealing is wrong because it's against the will of God". But where's the evidence that they use "Stealing is wrong" to mean "Stealing is against the will of God"?

But moral terms and value terms are about what we want.

How do you know? And this seems to contradict your claim above that some people use "Stealing is wrong" to mean "stealing is against the will of God". That's not about what we want. (I say that moral terms are primarily about obligations, not wants.)

Hi Luke,

I've questioned your metaethical views before (in your "desirist" days) and I think you're making similar mistakes now as then. But rather than rehash old criticisms I'd like to make a different point.

Since you claim to be taking a scientific or naturalized approach to philosophy I would expect you to offer evidence in support of your position. Yet I see nothing here specifically identified as evidence, and very little that could be construed as evidence. I don't see how your approach here is significantly different from the intuition-based philosophical approaches that you've criticised elsewhere.

Some people who say "Stealing is wrong" are really just trying to express emotions: "Stealing? Yuck!" Others use moral judgments like "Stealing is wrong" to express commands: "Don't steal!" Still others use moral judgments like "Stealing is wrong" to assert factual claims, such as "stealing is against the will of God" or "stealing is a practice that usually adds pain rather than pleasure to the world."

How do you know this? Where's the evidence? I don't doubt that some people say, "Stealing is wrong because it's against the will of God". But where's the evidence that they use "Stealing is wrong" to mean "Stealing is against the will of God"? (Indeed, if they meant that it would be very strange to say "Stealing is wrong because it's against the will of God". That would be equivalent to saying "A is true because A is true." Yet it seems perfectly natural for someone to say this.)

But moral terms and value terms are about what we want.

How do you know? And this seems to contradict your claim above that some people use "Stealing is wrong" to mean "stealing is against the will of God". That's not about what we want. (I say that moral terms are primarily about obligations, not wants.)

the Austere Metaethicist replies:

"Tell me what you mean by 'right', and I will tell you what is the right thing to do."

That is of course, not what is right, but what she thinks is right. So far, so subjective.

You may not know what you mean by 'right.' But let's not stop there. Here, let me come alongside you and help decode the cognitive algorithms that generated your question in the first place, and then we'll be able to answer your question. Then we can tell you what the right thing to do is.

Again, that is not the right thing, that is just what she thinks. An Objective metaethicist could answer the question what is right.

But moral terms and value terms are about what we want.

No: they are value terms about what we should want and be and do.

And the "we" is important here. Your metaethicists are like therapsists or life coaches or personal shoppers who advise people how to make their individual lives spiffier. But moral action is not solipsistic: moral choices affect other people. That's why we can't stop at "whatever you think is right is right". I don't want one of your metaethicists telling my neighbour how to be a better serial killer.

Although I think this series of posts is interesting and mostly very well reasoned, I find the discussion about objectivity to be strangely crafted. At the risk of arguing about definitions: the hierarchy you lay out about objectivity is only remotely related to what I mean by objective, and my sense is that it doesn't cohere very well with common usage.

First, there seems no better reason to split off objective1 than objectiveA which is "software-independent facts". Okay, so I can't say anything objective about my web browser, just because we've said I can't. Why is this helpful? The only reason to split this out is if you are some sort of dualist; otherwise the mind is a computational phenomenon just like DNA replication or whatnot.

Second, as Emile already pointed out, nowhere in the hierarchy is uniqueness addressed, yet this is the clearest conventional distinction between subjectivity and objectivity. 5+7 = 12 for everyone. Mint chocolate chip ice cream is better than rocky road ice cream is not the case for everyone (in the conventional sense, anyway). So these things are all colloquially objective:

• Rocky road has more chocolate than mint chocolate chip
• The author of this post enjoys mint chocolate chip more than rocky road
• My IPv4 address has a higher value than does lesswrong.org
• The Bible describes God endorsing the consumption of only certain animals

Referring to God doesn't make things non-objective in the standard sense presuming God exists. Of course, without a way to measure God's preferences, you may lose your theoretical objectivity, but any other single source or self-consistent group can fill in (e.g. the Pope) as an source for objective answers to what would otherwise be subjective questions.

The issue isn't whether that is subjective or objective; it's whether that method of gaining objectivity is practical and useful.

And since humans are the only sentient beings, I really fail to see what the distinction is between 2 and 3 is in a practical way, once you split off God (or any other singularly identifiable entity).

So I strongly suggest that this section ought to be rethought. Objectivity seems central to this sort of moral reductionism, and so it is worth using definitions that are not too misleading. Either the definitions should change, or there should be much more motivation about why we care about the distinctions between any of the definitions you've offered.

In jest, I'm going to accuse you of plagiarizing my work, then tell you two problems that I have with the approach that you've outlined, and then wax e-peen and say that mine is similar, but more instructive on moral discourse among all users of it.

My problem here is that we already have a common language (in terms of wants) which reduces "should" and provides the kind of plurality that you're seeking out of this approach, so there's no need to claim, "I'm using 'is good' to mean P," and then eek out a true statement whose truth is a matter of lexical elaboration, when instead people use moral language to alter others' of their own behavior, perspective, etc. without all of that theorizing on top of it. Most people don't make moral arguments on the basis of grand (meta)ethical stances. But it seems like everyone would have to be a deep ethicist to get any traction out of this theory, and that would mean that it really only explains how experts use moral language, but not how everyday people do. But why would one have to be deep about ethics to prescribe that someone do something? Can't people prescribe an action and justify it without appealing to a definition of 'goodness' at all?

My last issue here is a potential paradox that I spotted when I made two pluralistic moral reductionists confront each other:

PMR1: "Is Harris's defined 'being good' better than Craig's defined 'being good'?"

PMR2: "What do you mean by 'better'?"

PMR1: "I mean whatever you mean when you say 'better' in questions like this."

PMR2: "But by 'better', I mean whatever you mean when you say 'better' in questions like this."

Moral facts are objective3 if they are made true or false by facts independent of the opinions of humans, otherwise they are subjective3.

Are the words of this comment subjective3 (if we drop the specifier "moral" for a moment, given that this idea is not defined in this context)? They are determined by my reasoning, but they also obey the laws of physics. The notion of "independent" is not easy to make precise.

"He's an unmarried man, but is he a bachelor?" This is a 'closed' question. The answer is obviously "Yes."

This is a false claim, unfortunately. Bachelor is not merely an "unmarried man", it is an "unmarried man who could've been married in his society" (as all the long-discussed things like "#My 5-year-old son is a bachelor" and "#The Pope is a bachelor" show). ETA: the part beginning with "who" is probably a presupposition rather than assertion ("The Pope is not a bachelor" is only felicitous if used as metalinguistic "The Pope cannot be described by the word 'bachelor'", not if used in the literal sense "The Pope is married although it is not allowed").

Austere Metaethicist: Your definition doesn't connect to reality. It's like [? · GW] talking about atom-for-atom 'indexical identity' even though the world is [? · GW] made of configurations and amplitudes instead of Newtonian billiard balls. Gods don't exist. [? · GW]

This one is also not obviously true. We can ask what Sherlock Holmes would approve of despite the fact that he never existed (and I can imagine a morality that says "good is what Sherlock approves of" - a strange morality though it would be). Why can't we take "an essentially just and loving God" as a similar literature character?

Did the next few posts Luke mentions would be about empathic metaethics ever get written? I don't see them anywhere.

I'm commenting on the post-change "Is-Ought" section. It seems to me that most of the examples given of "ought" reductions do not support the conclusion that "the is-ought gap can be bridged", because the reductions are wrong. Anyone can propose a naturalistic definition of "ought", but at a minimum, to be right a translation of an "ought" statement into an "is" statement has to preserve the truth value of the "ought" statement, and most of the reductions listed fail to do so.

Take the first example:

"X is obligatory (by deontic logic) if you assume axiomatic imperatives Y and Z."

If you give me any specific proposal for Y and Z, I'm pretty sure I can find an X such that "you ought to X" is obviously false and "X is obligatory (by deontic logic) if you assume axiomatic imperatives Y and Z" is true, or vice versa.

Luke, what is the meta-purpose of this sequence? It seems that you are trying to dissolve the persisting confusion about ethics, but what is the underlying purpose? I doubt this sequence is a terminal goal of yours, am I wrong? The reason for my doubt is that, before you started writing this sequence, you decided to join the visiting fellows program at the Singularity Institute for Artificial Intelligence. Which makes me infer that the purpose of this sequence has to do with friendly AI research.

If I am right that the whole purpose of this sequence has to do with friendly AI research, then how is it useful to devote so much resources to explaining the basics, instead of trying to figure out how to design, or define mathematically, something that could extrapolate human volition?

Here are some common uses of the objective/subjective distinction in ethics:

• Moral facts are objective1 if they are made true or false by mind-independent facts, otherwise they are subjective1.
• Moral facts are objective2 if they are made true or false by facts independent of the opinions of sentient beings, otherwise they are subjective2.
• Moral facts are objective3 if they are made true or false by facts independent of the opinions of humans, otherwise they are subjective3.

Hmm, that doesn't cover the way I understand "objective" and "subjective" - I see them as referring to whether the answer to a question varies from one person to another, i.e. whether they are function of the person speaking. "Is that play interesting?" is subjective4, "Does that play follow the three unities of classical drama?" is objective4 - so to match your pattern it would be something like "Moral facts are objective4 if they are made true or false by facts independent of any single human, otherwise they are subjective4".

i.e. it seems to me that some things are considered "objective" if they are merely social customs that may not hold in another society in another age.

But my thinking around this isn't very clear, I'm mostly reacting to the fact that none of the definitions you listed seemed to fit the way I understood the words.

Sounds like a form of speaker relativism, with the 'empathetic' project being about going beyond merely saying that people are expressing different fundamental standards, values, etc. to developing ways to bring those out into the open.

In a sense, pluralistic moral reductionism can be considered a robust form of moral 'realism', in the same way that pluralistic sound reductionism is a robust form of sound realism. "Yes, there really is sound, and we can locate it in reality — either as vibrations or as mental auditory experiences

In a sense it can't be considered robust realism. The two meanings of "sound" don't lead to any confusion in practice: we know that powerful hi fis produce a lot of sound1 and we know that deaf people standing in front of the won't hear any sound2.

However, morality is tied to the actions of oneself and others. There's no coherent situation in which gay marriage can be both right and wrong, because there is no coherent way it can be both allowed and forbidden. In practice we tend to just average out preferences when it comes to defining law and allocating resources...but the preferences in question might as well be subjective. It is difficult to see what is being bought by the objectivity in pluralistic objective reductionism.

But in another sense, pluralistic moral reductionism is 'anti-realist'. It suggests that there is no One True Theory of Morality. (We use moral terms in a variety of ways, and some of those ways refer to different sets of natural facts.)

Of course normative pluralism doesn't follow from descriptive pluralism. Some uses of moral terms be wrong. You said Craigs theological morality was wrong. If reducing and naturalising morality doesn't allow you to say anybody is right or wrong, what is the point?

And as a reductionist approach to morality, it might also leave no room for moral theories which say there are universally binding moral rules for which the universe (e.g. via a God) will hold us accountable.

Or, eg,, not via God. Unless there are: if you can say 1 theory is wrong, you can say N-1 are wrong and arrive at one not-wrong theory by elimination. So I don't see how your approach supports pluralism as a fixed principle: it can only depend on how things pan out in practice.

If someone makes a claim of the 'ought' type, either they are talking about the world of is, or they are talking about the world of is not.

When people are talking about 'ought', they are frequently mean something that's different from 'is' but is like 'is' in that it's a primary concept. For them, 'ought' is not something that can be defined in terms of 'is'.

So IMO people who are talking about 'ought' often really are talking about the world of 'ought', and that's about all you can say about it.

If they are talking about the world of is not, then I quickly lose interest because the world of is not isn't my subject of interest.

You're entitled to be uninterested in the world of 'ought' as a primary concept as well. I am not interested in it either, so I can't defend the point of view of these 'ought' believers. I have repeatedly had conversations with them, so I am sure they exist.

Why stop with morality?For any debate about some X, you can shit that there is no real disagreement, and the two parties are just talking about two different things, X1 and X2, which you arrive at by treating one set of theoreticalclaims as the definition of X1 and the other as X2..

I don't think this dissolution of disputes is a good idea in general, because I think theories aren't definitions, and Ithink there are real disputes, and I'm suspicious of unuversal solvents. But I like tthe Sound example. But I dont like the morality example.. pluralism is bad for the sane reason that subhectivism is bad ... morality has a practical, aspect... laws are passed, sanctions handed out and those either happen or don't.

I'm confused. If this article promotes pluralistic moral reduction, why does Luke M. make a statement that sounds reductionaly singular rather than plural? I mean this statement here:

But moral terms and value terms are about what we want.

Isn't this a reduction of morality to a single thing, specifically, "what we want", i.e., desire?

Isn't this defining morality as the practice of acting on our desires? And does not this definition contradict other definitions/reductions?

Am I correct in (roughly) summarizing your conclusion in the following quote?

Yes, there really is morality, and we can locate it in reality — either as a set of facts about the well-being of conscious creatures, or as a set of facts about what an ideally rational and perfectly informed agent would prefer, or as some other set of natural facts.

If so, what is the logical difference between your theory and moral relativism? What if a person's set of natural facts for morality is "those acts which the culture I was born into deem to be moral"?

Update: I've had the pleasure of discussing some of the topics from this post in episode 87 of video-podcast 'Truth-Driven Thinking.'

I'm about to rewrite the section on the is-ought gap, for clarity, so here's a copy of the original text:

Many claim that you cannot infer an 'ought' statement from a series of 'is' statements. The objection comes from Hume, who wrote that he was surprised whenever an argument made of is and is not propositions suddenly shifted to an ought or ought not claim without explanation.

Many of Hume's followers concluded that the problem is not just that this shift happens without adequate explanation, but that one can never derive an 'ought' claim from a series of 'is' claims.

But why should this be? If someone makes a claim of the 'ought' type, either they are talking about the world of is, or they are talking about the world of is not. (What they're talking about is plausibly reducible to physics, or else it isn't.) If they are talking about the world of is not, then I quickly lose interest because the world of is not isn't my subject of interest. If they are making a claim about the world of is, then I ask them which part of the world of is they are discussing. I ask which ought-reductionism they have in mind.

Often, they have in mind a common ought-reductionism known as the hypothetical imperative. This is an ought of the kind: "If you desire to lose weight, then you ought to consume fewer calories than your burn." (But usually, people leave off the implied if statement, and simply say "You should eat less and exercise more.")

A hypothetical imperative (a kind of ought statement) reduces in a straightforward way to a prediction about reality (a kind of is statement). "If you desire to lose weight, then you ought to consume fewer calories than you burn" translates to the claim "If you consume fewer calories than you burn, then you will (or are, ceteris paribus, more likely to) fulfill your desire to lose weight."

Or, perhaps someone has a moral reductionism in mind during a particular use of 'ought' language. Perhaps by "You ought to be more forgiving" they really mean "If you are more forgiving, this is likely to increase the amount of pleasure in the world."

As you can see, it is not hard to bridge the is-ought gap. 'Ought' statements either collapse into the world of is (as with hypothetical imperatives or successful moral reductionisms), or else they collapse into the world of is not (as with Craig's moral theory of divine approval).

Stealing often means wrongful taking of property... but point well taken.

I miss the discussion (on LW in general) of an approach to ethics that strives to determine what actions should be unlawful for an agent, as opposed to, say, what probability distribution over actions is optimal for an agent. (And I don't mean "deontologic", as the "unlawfulness" can be predicated on the consequences.) If you criticize this comment for confusion of "descriptive ethics vs. normative ethics vs. metaethics", try to be constructive.

The Yudkowskian response is to point out that when cognitivists use the term 'good', their intuitive notion of 'good' is captured by a massive logical function that can't be expressed in simple statements

This is the weakest part of the argument. Why should anybody believe that there is a super complicated function that determines what is 'good'? What are the alternative hypotheses?

I can think of a much simpler hypothesis that explains all of the relevant facts. Our brains come equipped with a simple function that maps "is" statements to "ought" statements. Thus, we can reason about "ought" statements just like we do with "is" statements.

The special thing about this function is that there is nothing special about it at all. It is absolutely trivial. Any "ought" statement can potentially be inferred from any "is" statement. Therefore, "ought" statements can never be conditioned by evidence. This explains not only why there is lots of disagreement among people about what is "good" and that our beliefs about what constitutes "good" will be very complicated, but also that there will be no way to resolve these disagreements.

But whatever our intended meaning of 'ought' is, the same reasoning applies. Either our intended meaning of 'ought' refers (eventually) to the world of maths and physics (in which case the is-ought gap is bridged), or else it doesn't (in which case it fails to refer).12

The is-ought problem is an epistemic problem. Being informed that some A is ultimately, ontologically, the same as some B does not tell me how that A entails that B. If I cannot see how an "is" implies an "ought", being informed that the "ought" ultimately refer to states of the world -- states of the world far too complex for me to include in my epistemic calculations -- does not help. I can't cram a (representation of a) world-state into my brain. Being informed that if I could I would no longer have an is-ought problem under the unlikely circumstances that I could do so doesn't help. The ontological claim that is's and ought's ultimately have the same referents can only be justified by some epistemic procedure. That is the only way any ontological claim is justified.