Multiple Moralities

post by Liam Goddard · 2019-11-03T17:06:34.374Z · LW · GW · 5 comments

(If there had been a different crew aboard the ship [LW · GW], if they were greeted by different members of the aliens…)

ALICE: Um- Bob? You should probably take a look at this.

BOB: (reading) You- but- they-

ALICE: Am I reading this right? Because it seems to me as if the only possible civilization that could have generated this data would be one that…

BOB: Eats… their children?!

ALICE: I- I have to analyze this more. If I can investigate their society, I might be able to find out something… some way to persuade them… (Alice leaves the room)

BOB: There’s a message coming in… hang on, let me hook up the translator-

BABYEATER: Greetings, human. My name is Eman’tres’ni, the king of Te’nalp’nei’la. Who are you?

BOB: Um- I’m Bob. I’m a human. I’m the captain of the Impossible Possible World, which is our ship, and…

EMAN’TRES’NI: Bob the Captain of the Humans, are-

BOB: Um- I’m not the ruler of all humans. Just this one ship. We sent you some files concerning humanity, you might have seen those…

EMAN’TRES’NI: Yes. Reading over those files, I believe that you may be rather disturbed, with some parts of our society, particularly the consuming of our live children.

BOB: Yes. We… definitely are.

EMAN’TRES’NI: May I ask why? I have not had time to read your philosophy journals, since there are quite a few of them, but I am unsure why you would have an objection to eating children.

BOB: Because- they die! They have lives ahead of them that are cut short, and plunged into oblivion- you’re annihilating innocents!

EMAN’TRES’NI: Yes, we indeed are annihilating innocents. Why does this trouble you?

BOB: Because it’s not right!

EMAN’TRES’NI: That- that doesn’t make sense. What would annihilating innocents have to do with the direction of left?

BOB: The word right has two meanings, it might not be getting through your cultural translator, I meant the other meaning.

EMAN’TRES’NI: Assigning one word two meanings seems like it could cause quite a lot of difficulties in communication. Our scientist and philosopher Yks’wok’duy wrote a sequence [? · GW] on why that sort of thing causes problems… but please rephrase that sentence, with a synonym.

BOB: I mean, Because it’s not moral!

EMAN’TRES’NI: (pause) I am beginning to believe there may be more problems with translation than I feared, as this word moral is not getting through our translator either.

BOB: Maybe the word ethical?


BOB: The word good?

EMAN’TRES’NI: Yes, that is… mostly getting through, but it seems to have multiple definitions, which generally just seem to be assigning a positive feeling, and I still do not understand why you think it is negative. Perhaps you should explain the concept of moral to us, without any synonyms. Yks’wok’duy has spoken of a process called Taboo that might work to help explain a strange notion…

BOB: Okay. So- we humans have a system that we call morality. In this system, values are assigned to each action you could do, and these values are called utility. The amount of utility that we assign to any particular action depends on its consequences. There are several criteria that determine how much utility one particular consequence should have, but some of the main ones include that happiness for yourself and others is positive utility, while sadness and pain are negative utility; longer lifespan for yourself and others is positive, death is negative; knowing the truth is positive, believing lies is negative; and so on.

EMAN’TRES’NI: This human morality system sounds interesting, Bob. But you have not told us why we should care about utility. If someone dies, it may be negative utility, but why should I want positive utility in the first place?

BOB: Well-

ALICE: (returns to main hall) Eman’tres’ni, before we respond, I think it might be best for you to explain how members of your civilization makes decisions.

EMAN’TRES’NI: Certainly. The Elp’oep’nei’la maximize happiness. We-

BOB: But wouldn’t eating people alive count as unhappiness?!

EMAN’TRES’NI: For the eaten, yes, but not the eaters. Elp’oep’nei’la children taste wonderful. I maximize only my own happiness, and care about nothing else. The rest of the Elp’oep’nei’la do the same. We are not sitting in chairs, hooked up to pleasure stimuli, but that is only because we have not yet successfully programmed nanotechnology to create anything more pleasureful than eating Elp’oep’nei’la children. We always have an abundant supply of them, and we are always happy. What else do we need?


ALICE: Bob! Screaming incoherently won’t solve anything. I am going to try to explain to Eman’tres’ni how morality works, and see if we can come to some sort of agreement.

EMAN’TRES’NI: Go ahead.

ALICE: You seem to be under the impression that our utility is akin to your happiness. That you maximize happiness, and we maximize our utility. That each of us chooses our own system, and that there’s no difference in how right something is. But morality isn’t something humans just made up. Morality is beyond any one person’s wants.

EMAN’TRES’NI: But who did create morality? I’ve read a little about humans, and I know you don’t believe in your crazy god anymore, so why this? If one of your leaders created morality, I feel fairly certain that you would have protested- your history shows quite a few revolutions. So where does it come from?

ALICE: Morality might have been discovered by humans, but that doesn’t mean we’re just blindly obeying something arbitrary.

EMAN’TRES’NI: But it is arbitrary! If whoever invented morality had been replaced by someone else, then you would be arguing against me based on an entirely different system! We have our system, and you have yours, and there is no fundamental difference between them. If you had evolved slightly differently, none of you would care about “morality.” You’re rationalizing a belief you only came up with for unimportant reasons, like god and natural selection.

BOB: But even if it was just passed down from other humans, it’s still better than your system-

EMAN’TRES’NI: What makes it better? Why should death be worse than life? What makes any one system better than another?

CAROL: Hey, due to the author of this blog post being able to manipulate our reality and create incredible coincidences to move the story along, my lunch break just ended exactly when he asked that ominous question, so that I could join this conversation.

BOB: What?

CAROL: Nothing. Anyway, the answer to your question is…

And that’s as far as I can go for now, because I don’t actually know what Carol would answer here.

I don’t exactly agree with Eman’tres’ni about everything. I’m not eating any children, even if they do taste good, and I wouldn’t even if it were legal/possible. My system of morality is very similar to the typical human morality. But I do think that this decision is pretty much personal, and there really isn’t anything more right than anything else, since every justification only requires more justifications.

If you would have something to say to Eman’tres’ni, explain your beliefs in the comments. What would you reply, if you were part of this conversation on the Impossible Possible World?


Comments sorted by top scores.

comment by FeepingCreature · 2019-11-03T18:46:33.929Z · LW(p) · GW(p)

Most morality is cashed out game theory. In humans this usually takes the form of "rule utilitarianism", because first of all humans don't have the cognitive capability to evaluate a utilitarian criterion over their entire worldstate, and second of all some if not all humans would defect if given the chance, so everyone agrees it's best we pick a system where they're not given the chance, so we can all benefit from the cooperate/cooperate outcomes. Now, when evaluating the quality of a rule, we use something called the "Rawlsian veil", which avoids temporary imbalances in power from skewing the outcomes - for instance, if in any exchange, only one player gets to decide the rules, but which player it is varies randomly, then both players will want the exchange to be judged as if both players have equal worth because otherwise you can't get the cooperate/cooperate outcomes yada yada. (Power imbalances are common in human society.) So the way our morality works, or at least this is the way we'll claim it works when asked¹, is whenever we face a moral challenge we first set the moral value of all participants to equal, and then compute the rule that maximizes outcomes across the board. Understand that this is not arbitrary - so far everything we're saying is for the purpose of maximizing payoffs and somewhat insensitive to the actual distribution of rewards, and we should expect this system - call it "rule of law" - to be somewhat convergent in alien species.

Transitioning society to this shape was not an instant process. As such, we also have a bunch of metamorality bound up; we see people who defect against weaker partners as exploiters, and people who cooperate even when stronger as heroic and good. However, because we needed to transition to this morality (it wasn't inherently inborn in our species) we also have standards for when defecting is acceptable, and it's that if defecting is ever acceptable it's against people who themselves defected, denying them the benefits of the social compact of rule of law. Again, we should expect this to be convergent.

Now, it's true that if you're really on top of things you can defect to your heart's content. However, you just met aliens. So what you're actually espousing, behaviorally, is that more powerful aliens can eat you if it makes them really really happy - you're saying that you're happy being in the defector category. If you ever meet more powerful aliens, that may be risky - in fact, you should preemptively try to guess what kinds of rules these aliens would expect arbitrary aliens to apply and live by them.

Probably "don't eat weaker participants" is not a rule you want to be exempt from...

¹ In our current society, we don't always live by these principles, but also we're in the tail end of a transition away from belief in a higher power, so we're in an unusually defecty stage. It's unclear how representative that is.

Replies from: Liam Goddard, Bunthut
comment by Liam Goddard · 2019-11-03T19:06:17.172Z · LW(p) · GW(p)

All of this makes a lot of sense when it comes to rules for society. And I understand that certain forms of government, or certain laws, would be effective for almost any utility function. What I’m questioning isn’t how you achieve your goals, it’s where goals themselves come from, your terminal values.

Replies from: gilch
comment by gilch · 2019-11-03T20:52:07.331Z · LW(p) · GW(p)

You can't persuade a rock that cheesecake tastes better than dirt, no matter how clever your arguments. Not every possible mind might agree with us, even in principle; there are No Universally Compelling Arguments [LW · GW].

We are born already in motion [LW · GW] with godshatter [LW · GW] instincts suited to our stone-age ancestral environment of evolutionary adaptation. This part is not simply arbitrary: it had the requisite survival value, or we would not be here talking about it. We are then acculturated to our surrounding society. This part is learned, but not entirely arbitrary either, because culture itself is evolving and subject to selection pressures.

To whatever extent we judge our society to be suboptimal, we must use our evolved/acculturated minds to do it. What else could we use? But there are reasons we are the way we are, principles that are not simply random.

comment by Bunthut · 2019-11-04T11:31:18.998Z · LW(p) · GW(p)
we use something called the "Rawlsian veil", which avoids temporary imbalances in power from skewing the outcomes
Power imbalances are common in human society.

But many of the power imbalances which are found in human society are not at all temporary. For instance, if the player deciding didn't vary randomly, but instead triangles always got to decide over squares, then while there might still develop a rule of law internally, its not clear what interest the triangles have in rectifying the inter-gonal situation. But we still (claim to) regard it as moral for that to happen. It seems the babyeaters are indeed in such a situation: Any adult eating the babies will never be a baby again. Further, they are almost certain to succeed in eating them, after which they will not grow big and maybe become a threat some day.

comment by Pattern · 2021-06-27T19:52:20.665Z · LW(p) · GW(p)

They should probably start with "how did you evolve to not care about anyone at all? Your kids? Like how did that happen?"