A Defense of Naive Metaethics

I share your skepticism about Luke's statement (but I've been waiting to criticize until he finishes his sequence to see if he addresses the problems later).

My goal here is to get people to feel the intuition that "I ought to do X" means something, and that thing is not "I think I ought to do X" or "I would think that I ought to do X if I were smarter and some other stuff".

To help pump that intuition, consider this analogy:

"X is true" (where X is a mathematical statement) means something, and that thing is not "I think X is true" or "I would think that X is true if I were smarter and some other stuff".

On the other hand, I think it's also possible that "I ought to do X" doesn't really mean anything. See my What does a calculator mean by "2"?. (ETA: To clarify, I mean some usages of "ought" may not really mean anything. There are some usages that clearly do, for example "If you want to accomplish X, then you ought to do Y" can in principle be straightforwardly reduced to a mathematical statement about decision theory, assuming that our current strong intuition that there is such a thing as "the right decision theory" is correct.)

People who do feel that intuition run into trouble. This is because "I ought to do X' does not refer to anything that exists. How can you make a statement that doesn't refer to anything that exists?

It refers to my preferences which are physically encoded in my brain. It feels like it doesn't refer to anything that exists because I don't have complete introspective access to the mechanisms by which my brain decides that it wants something.

On top of that, ought refers to lots of different things, and as far as I can tell, most ought statements are summaries of specific preferences (and some signals) rather than the even more complicated description of what I'm actually going to choose to do.

Will and I just spoke on the phone, so here's another way to present our discussion:

Imagine a species of artificial agents. These agents have a list of belief statements that relate physical phenomena to normative properties (let's call them 'moral primitives'):

• 'Liking' reward signals in human brains are good.
• Causing physical pain in human infants is forbidden.
• etc.

These agents also have a list of belief statements about physical phenomena in general:

• Sweet tastes on the tongue produces reward signals in human brains.
• Cutting the fingers of infants produces physical pain in infants.
• Things are made of atoms.
• etc.

These agents also have an 'ought' function that includes a series of logical statements that relate normative concepts to each other, such as:

• A thing can't be both permissible and forbidden.
• A thing can't be both obligatory and non-obligatory.
• etc.

Finally, these robots have actuators that are activated by a series of rules like:

• When the agent observes an opportunity to perform an action that is 'obligatory', then it will take that action.
• An agent will avoid any action that is labeled as 'forbidden.'

Some of these rules might include utility functions that encode ordinal or cardinal value for varying combinations of normative properties.

These agents can't see their own source code. The combination of the moral primitives and the ought function and the non-ought belief statements and a set of rules about behavior produces their action and their verbal statements about what ought to be done.

From their behavior and verbal ought statements these robots can infer to some degree how their ought function works, but they can't fully describe their ought function because they haven't run enough tests or the ought function is just too complicated or the problem is made worse because they also can't see their moral primitives.

The ought function doesn't reduce to physics because it's a set of purely logical statements. The 'meaning' of ought in this sense is determined by the role that the ought function plays in producing intentional behavior by the robots.

Of course, the robots could speak in ought language in stipulated ways, such that 'ought' means 'that which produces pleasure in human brains' or something like that, and this could be a useful way to communicate efficiently, but it wouldn't capture what the ought function is doing or how it is contributing to the production of behavior by these agents.

What Will is saying is that it's convenient to use 'ought' language to refer to this ought function only, and not also to a combination of the ought function and statements about physics, as happens when we stipulatively use 'ought' to talk about 'that which produces well-being in conscious creatures' (for example).

I'm saying that's fine, but it can also be convenient (and intuitive) for people to use 'ought' language in ways that reduce to logical-physical statements, and not only in ways that express a logical function that contains only transformations between normative properties. So we don't have substantive disagreement on this point; we merely have different intuitions about the pragmatic value of particular uses for 'ought' language.

We also drew up a simplified model of the production of human action in which there is a cognitive module that processes the 'ought' function (made of purely logical statements like in the robots' ought function), a cognitive module that processes habits, a cognitive module that processes reflexes, and so on. Each of these produces an output, and another module runs arg(max) on these action options to determine which actions 'wins' and actually occurs.

Of course, the human 'ought' function is probably spread across multiple modules, as is the 'habit' function.

Will likes to think of the 'meaning' of 'ought' as being captured by the algorithm of this 'ought' function in the human brain. This ought function doesn't contain physical beliefs, but rather processes primitive normative/moral beliefs (from outside the ought function) and outputs particular normative/moral judgments, which contribute to the production of human behavior (including spoken moral judgments). In this sense, 'ought' in Will's sense of the term doesn't reduce to physical facts, but to a logical function.

I'm fine with Will using 'ought' in that sense if he wants. I'll try to be clear how I am using the term when I use it.

Will also thinks that the 'ought' function (in his sense) inside human brains is probably very similar between humans - ones that aren't brain damaged or neurologically deranged. I don't know how probable this is because cognitive neuroscience hasn't progressed that far. But if the 'ought' function is the same in all healthy humans, then there needn't be a separate 'meaning' of ought (in Will's sense) for each speaker, but instead there could be a shared 'meaning' of ought (in Will's sense) that is captured by the algorithms of the 'ought' cognitive module that is shared by healthy human brains.

Will, did I say all of that correctly?

How can you make a statement that doesn't refer to anything that exists? I've done it, and my reasoning process is still intact, and nothing has blown up. Everything seems to be fine. No one has explained to me what isn't fine about this. Since it's intuitive, why would you not want to do it that way?

Clearly, you can make statements about things that don't exist. People do it all the time, and I don't object to it. I enjoy works of fiction, too. But if the aim of our dialogue is true claims about reality, then you've got to talk about things that exist - whether the subject matter is 'oughts' or not.

What one is forced to do by this argument, if one wants to speak only in physical statements, is to say that "should" has a really, really long definition that incorporates all components of human value. When a simple word has a really, really long definition, we should worry that something is up.

I don't see why this needs to be the case. I can stipulate short meanings of 'should' as I use the term. People do this all the time (implicitly, at least) when using hypothetical imperatives.

Also, in general I find myself confused by your way of talking about these things. It's not a language I'm familiar with, so I suspect I'm still not fully understanding you. I'm not sure which of our anticipations differ because of the disagreement you're trying to express.

I made a more topical comment to Wei_Dai's reply to this thread, but I felt it was worth adding that if anyone is interested in a work of fiction that touches on this subject, the novel The Broom of the System, by David Foster Wallace, is worth a look.

Can you explain what implications (if any) this "naive" metaethics has on the problem how to build an FAI?

Let me have another go at this, since I've now rewritten the is-ought section of 'Pluralistic Moral Reductionism' (PMR).

This time around, I was more clear that of course it's true that, as you say:

we can make statements about what should be done and what should not be done that cannot be reduced, by definition, to statements about the physical world

We can make reducible 'ought' statements. We can make irreducible 'ought' statements. We can exhibit non-cognitive (non-asserting) verbal behaviors employing the sound 'ought'.

For the purposes of the PMR post, I'm interested to investigate frameworks that allow us to determine whether a certain subset of 'ought' statements are true or false.

Thus, in the context of PMR, non-asserting verbal behaviors employing 'ought' sounds are simply a different subject matter. They do not belong to the subset of 'ought' statements I am investigating for that particular post. (I discussed this in the cognitivism vs. non-cognitivism section.) At the same time, clearly such statements can be useful. For example, you can use them to affect others' behavior. You can affect others' behavior and attitudes with non-asserting verbal behaviors, such as when you say "female circumcisions" with a gasp and a frown.

Also in the context of PMR, some 'ought' statements can quickly be tossed in the 'false' bin, because the speaker uses 'ought' language assertively to refer to things that clearly don't exist, like divine commands.

In the context of PMR, other 'ought' statements can be tossed into the 'false' bin if you are a physicalist, because the speaker uses 'ought' language assertively to refer to things that don't fit within a physicalist ontology, like non-natural moral properties. If you're not a physicalist, then our debate about such 'ought' statements can shift to a debate about physicalism vs. non-physicalism.

Will, you seem to be saying that 'ought' has only one meaning, or one definition. You also seem to be saying that this one meaning of 'ought' (or 'should') is captured by what we 'actually want' in the CEV sense. Is that right so far?

If so, I'm still not clear on your arguments for this conclusion. Your writing here has a very high ratio of unstated premises to stated premises. My own writing does that all the time for communication efficiency, in the hopes that my unstated premises are shared or else successfully inferred. But many times I find that the unstated premises didn't make it into the other's mind, and thus my enthymeme is unclear to it. And when I care enough about my argument being clear to certain people, I take the time to draw my unstated premises into the light and state them explicitly.

Since I'm having trouble guessing at the unstated premises in your arguments for singularism about the meaning of 'should' or 'ought', I'll request that you state them explicitly. Would you please state your most central argument for this singularism, without unstated premises?

Suppose an AI wants to find out what Bob means when he says "water".

This thought experiment can be sharpened by asking what Bob means by "1027". By using Bob as an intermediary, we inevitably lose some precision.

People who do feel that intuition run into trouble. This is because "I ought to do X' does not refer to anything that exists. How can you make a statement that doesn't refer to anything that exists? I've done it, and my reasoning process is still intact, and nothing has blown up. Everything seems to be fine. No one has explained to me what isn't fine about this.

Ok, I'll bite. Why does "I ought to X" have to refer to any thing?

When I see atucker's comment, for instance;

It refers to my preferences which are physically encoded in my brain. It feels like it doesn't refer to anything that exists because I don't have complete introspective access to the mechanisms by which my brain decides that it wants something.

I think "this isn't quite right, but it's kind of close"

So what, then, is "ought"? In reality it seems to derive its meaning from a finite set of functional signals. We could take it to mean:

A signal demonstrating group cohesion via expressing shared cherished beliefs "What's that? You say you aren't doing X? You ought to do X!" (murmured approval; yeah, we all do X around here, buddy!)

A signal demonstrating anxiety or indignation "Someone ought to do something to fix this!"

A signal demonstrating disapproval in the present "You oughtn't do that!", or referring to future events "You ought to do that differently from now on" or referring to the past "You ought to have done something else"

etc.

In this light we see that "ought" has no cleanly discernible referent (which makes sense because it isn't a noun), but rather derives meaning via its socially accepted usage in a set of signaling mechanisms.

"A Flatland Argument" includes a multitude of problems. It contains a few strawmen, which then turn into non sequiturs when you say that they demonstrate that we often say irreducible things. At least that's what I think you mean by the trivial statement "Not every set of claims is reducible to every other set of claims."

And yet your examples are answerable or full of holes! Why not mention the holes rather than blaming the reductionism (or, in the case of some strawmen, total lack thereof), or mention the reduction and forget the whole thing?

"I'm not interested in words, I'm interested in things. Words are just sequences of sounds or images. There's no way a sequence of arbitrary symbols could imply another sequence, or inform a decision."

"I understand how logical definitions work. I can see how, from a small set of axioms, you can derive a large number of interesting facts. But I'm not interested in words without definitions. What does "That thing, over there?" mean? Taboo finger-pointing."

"You can make statements about observations, that much is obvious. You can even talk about patterns in observations, like "the sun rises in the morning". But I don't understand your claim that there's no chocolate cake at the center of the sun. Is it about something you can see? If not, I'm not interested."

"Claims about the past make perfect sense, but I don't understand what you mean when you say something is going to happen. Sure, I see that chair, and I remember seeing the chair in the past, but what do you mean that the chair will still be there tomorrow? Taboo "will"."

Some answers, in order:

• "Imply" is a fact about cognitive processes, which occupy a very general space that includes all sorts of weird algorithms, even associating sounds or symbols, therefore the blanket statement "there's no way" is false. Given that cognitive processes and words are also things, you have also claimed that you are interested in them.

• This second example is made of only vaguely related sentences. Did you mean "that thing, over there" to not have a definition? Would tabooing finger pointing really be so hard? Just refer to a cardinal or relative direction.

• Reductionists are not, I hope, automatically ignorant of induction. "No cake at the center of the sun" is about something you can see, since the evidence is visible. And though seeing the sun is good evidence that the sun exists, that can never be perfect either - it's all evidence, whether it's the outer layer of the sun or the middle of it.

• "Will" refers to the fact that you have evidence of a 3+1-dimensional universe at least in the past and have no reason to believe that you're at a special moment, and so think that the time-dimension extends ahead of you. To say that something "will" be there tomorrow refers to a prediction about the state of the world at a time-coordinate greater than the most recent one that you know of.

It seems clear to me that I can multiply about what I care about, so I don't know quite what you want to say.

Since it's intuitive, why would you not want to do it that way?

What seems wrong with the obvious answer?

Do you think a FOOM'd self-modifying AI that cares about humanity's CEV would likely do what you consider 'right'? Why or why not? (If you object to the question, please address that issue separately.)