Morality is Scary

> All the rest is an act of shared imagination. It’s a dream we weave around a status game.
> They’re part of the dream of reality in which they exist, a dream that feels no less obvious and true to them than ours does to us.
> Moral ‘truths’ are acts of imagination. They’re ideas we play games with.

IDK, I feel like you could say the same sentences truthfully about math, and if you "went with the overall vibe" of them, you might be confused and mistakenly think math was "arbitrary" or "meaningless", or doesn't have a determinate tendency, etc. Like, okay, if I say "one element of moral progress is increasing universalizability", and you say "that's just the thing your status cohort assigns high status", I'm like, well, sure, but that doesn't mean it doesn't also have other interesting properties, like being a tendency across many different peoples; like being correlated with the extent to which they're reflecting, sharing information, and building understanding; like resulting in reductionist-materialist local outcomes that have more of material local things that people otherwise generally seem to like (e.g. not being punched, having food, etc.); etc. It could be that morality has tendencies, but not without hormesis and mutually assured destrubtion and similar things that might be removed by aligned AI.

You sound like you're positing the existence of two type of people: type I people who have morality based on "reason" and type II people who have morality based on the "status game". In reality, everyone's nearly everyone's morality is based on something like the status game (see also: 1 [LW · GW] 2 [LW · GW] 3 [LW · GW]). It's just that EAs and moral philosophers are playing the game in a tribe which awards status differently.

The true intrinsic values of most people do place a weight on the happiness of other people (that's roughly what we call "empathy"), but this weight is very unequally distributed [LW(p) · GW(p)].

There are definitely thorny questions regarding the best way to aggregate the values of different people in TAI. But, I think that given a reasonable solution, a lower bound on the future is imagining that the AI will build a private utopia for every person, as isolated from the other "utopias" as that person wants it to be. Probably some people's "utopias" will not be great, viewed in utilitarian terms. But, I still prefer that over paperclips (by far). And, I suspect that most people do (even if they protest it in order to play the game).

Feminist hero and birth control campaigner Marie Stopes, who was voted Woman of the Millennium by the readers of The Guardian and honoured on special Royal Mail stamps in 2008, was an anti-Semite and eugenicist

My conclusion from this is more like "successful politicians are not moral paragons". More generally, trying to find morally virtuous people by a popular vote is not going to produce great results, because the popularity plays much greater role than morality.

I googled for "woman of the year" to get more data points; found this list, containing: 2019 Greta Thunberg, 2016 Hillary Clinton, 2015 Angela Merkel, 2010 Nancy Pelosi, 2008 Michelle Obama, 1999 Madeleine Albright, 1990 Aung San Suu Kyi... clearly, being a politician dramatically increases your chances of winning. Looking at their behavior, Aung San Suu Kyi later organized a genocide.

The list also includes 2009 Malala Yousafzai, who as far as I know is an actual hero with no dark side. But that's kinda my point, that putting Malala Yousafzai on the same list as Greta Thunberg and Hillary Clinton just makes the list confusing. And if you had to choose one of them as the "woman of the millenium", I would expect most readers to vote for someone representing their political tribe. But to me that does not mean that people have no sense of morality, only that they can easily get politically mindkilled.

For the Malagasy people in Madagascar, it’s taboo [...] to sleep facing westwards, as you’ll kick the sunrise.

And this sounds silly to us, because we know that "kicking the sunrise" is impossible, because Sun is a planet, it is far away, and your kicking has no impact on it.

So, we should distinguish between people having different moral feelings, and having different models of the world. If you actually believed that kicking the Sun is possible and can have astronomical consequences, you would probably also perceive people sleeping westwards as criminally negligent, possibly psychopathic.

Kinda like being angry at people who don't wear face masks only makes sense under the assumption that the face masks prevent spreading of a potentially deadly disease. Without this context, anger towards people with no face masks would just be silly.

Even if moralities vary from culture to culture based on the local status games, I would suggest that there is still some amount of consequentialist bedrock to why certain types of norms develop. In other words, cultural relativism is not unbounded.

Generally speaking, norms evolve over time, where any given norm at one point didn't yet exist if you go back far enough. What caused these norms to develop? I would say the selective pressures for norm development come from some combination of existing culturally-specific norms and narratives (such as the sunrise being an agent that could get hurt when kicked) along with more human-universal motivations (such as empathy + {wellbeing = good, suffering = bad} -> you are bad for kicking the sunrise -> don't sleep facing west) or other instrumentally-convergent goals (such as {power = good} + "semen grants power" -> institutionalized sodomy). At every step along the evolution of a moral norm, every change needs to be justifiable (in a consequentialist sense) to the members of the community who would adopt it. Moral progress is when the norms of society come to better resonate with both the accepted narratives of society (which may come from legends or from science) and the intrinsic values of its members (which come from our biology / psychology).

In a world where alignment has been solved to most everyone's satisfaction, I think that the status-game / cultural narrative aspect of morality will necessarily have been taken into account. For example, imagine a post-Singularity world kind of like Scott Alexander's Archipelago, where the ASI cooperates with each sub-community to create a customized narrative for the members to participate in. It might then slowly adjust this narrative (over decades? centuries?) to align better with human flourishing in other dimensions. The status-game aspect could remain in play as long as status becomes sufficiently correlated with something like "uses their role in life to improve the lives of others within their sphere of control". And I think everyone would be better off if each narrative also becomes at least consistent with what we learn from science, even though the stories that define the status game will be different from one culture to another in other ways.

I am also scared of futures where "alignment is solved" under the current prevailing usage of "human values."

Humans want things that we won't end up liking, and prefer things that we will regret getting relative to other options that we previously dispreferred. We are remarkably ignorant of what we will, in retrospect, end up having liked, even over short timescales. Over longer timescales, we learn to like new things that we couldn't have predicted a priori, meaning that even our earnest and thoughtfully-considered best guess of our preferences in advance will predictably be a mismatch for what we would have preferred in retrospect. 

And this is not some kind of bug, this is centrally important to what it is to be a person; "growing up" requires a constant process of learning that you don't actually like certain things you used to like and now suddenly like new things. This truth ranges over all arenas of existence, from learning to like black coffee to realizing you want to have children.

I am personally partial to the idea of something like Coherent Extrapolated Volition. But it seems suspicious that I've never seen anybody on LW sketch out how a decision theory ought to behave in situations where the agents utility function will have predictably changed by the time the outcome arrives so the "best choice" is actually a currently dispreferred choice. (In other words, situations where the "best choice" in retrospect, and in expectation, do not match.) It seems dangerous to throw ourselves into a future where "best-in-retrospect" wins every time, because I can imagine many alterations to my utility function that I definitely wouldn't want to accept in advance, but which would make me "happier" in the end. And it also seems awful to accept a process by which "best-in-expectation" wins every time, because I think a likely result is that we are frozen into whatever our current utility function looks like forever. And I do not see any principled and philosophically obvious method by which we ought to arbitrate between in-advance and in-retrospect preferences.

Another way of saying the above is that it seems that "wanting" and "liking" ought to cohere but how they ought to cohere seems tricky to define without baking in some question-begging assumptions.

I honestly have a difficult time understanding the people (such as your "AI alignment researchers and other LWers, Moral philosophers") who actually believe in Morality with a capital M. I believe they are misguided at best, potentially dangerous at worst. 

I hadn't heard of the Status Game book you quote, but for a long time now it's seemed obvious to me that there is no objective true Morality, it's purely a cultural construct, and mostly a status game. Any deep reading of history, cultures, and religions, leads one to this conclusion.

Humans have complex values, and that is all. 

We humans cooperate and compete to optimize the universe according to those values, as we always have, as our posthuman descendants will, even without fully understanding them.

To repost my comment from a couple of weeks back [LW(p) · GW(p)], which seems to say roughly the same thing, not as well:

I don't believe alignment is possible. Humans are not aligned with other humans, and the only thing that prevents an immediate apocalypse is the lack of recursive self-improvement on short timescales. Certainly groups of humans happily destroy other groups of humans, and often destroy themselves in the process of maximizing something like the number of statues. Best we can hope for that whatever takes over the planet after meatbags are gone has some of the same goals that the more enlightened meatbags had, where "enlightened" is a very individual definition. Maybe it is a thriving and diverse Galactic civilization, maybe it is the word of God spread to the stars, maybe it is living quietly on this planet in harmony with the nature. There is no single or even shared vision of the future that can be described as "aligned" by most humans.

I'm not sure what you mean by 'astronomical waste or astronomical suffering'.  Like, you are writing that everything forever is status games, ok, sure, but then you can't turn around and appeal to a universal concept of suffering/waste, right?

Whatever you are worried about is just like Gandhi worrying about being too concerned with cattle, plus x years, yeah?  And even if you've lucked into a non status games morality such that you can perceive 'Genuine Waste' or what have you...surely by your own logic, we who are reading this are incapable of understanding, aside from in terms of status games.

So on the one hand you have values that are easily, trivially compatible, such as "I want to spend 1000 years climbing the mountains of Mars" or "I want to host blood-sports with my uncoerced friends with the holodeck safety on".

On the other hand you have insoluble, or at least apparently insoluble, conflicts: B wants to torture people, C wants there to be no torture anywhere at all. C wants to monitor everyone everywhere forever to check that they aren't torturing anyone or plotting to torture anyone, D wants privacy. E and F both want to be the best in the universe at quantum soccer, even if they have to kneecap everyone else to get that. Etc.

It's simply false that you can just put people in the throne as emperor of the universe, and they'll justly compromise about all conflicts. Or even do anything remotely like that.

How many people have conflictual values that they, effectively, value lexicographically more than their other values? Does decision theory imply that compromise will be chosen by sufficiently well-informed agents who do not have lexicographically valued conflictual values?

I'm leaning towards the more ambitious version of the project of AI alignment being about corrigible anti-goodharting, with the AI optimizing towards good trajectories within scope of relatively well-understood values, preventing overoptimized weird/controversial situations, even at the cost of astronomical waste. Absence of x-risks, including AI risks, is generally good. Within this environment, the civilization might be able to eventually work out more about values, expanding the scope of their definition and thus allowing stronger optimization. Here corrigibility is in part about continually picking up the values and their implied scope from the predictions of how they would've been worked out some time in the future.

I think this post makes an important point -- or rather, raises a very important question, with some vivid examples to get you started. On the other hand, I feel like it doesn't go further, and probably should have -- I wish it e.g. sketched a concrete scenario in which the future is dystopian not because we failed to make our AGIs "moral" but because we succeeded, or e.g. got a bit more formal and complemented the quotes with a toy model (inspired by the quotes) of how moral deliberation in a society might work, under post-AGI-alignment conditions, and how that could systematically lead to dystopia unless we manage to be foresightful and set up the social conditions just right.

I recommend not including this post, and instead including this one [LW(p) · GW(p)] and Wei Dai's exchange in the comments. [LW(p) · GW(p)]

If with "morality" you mean moral realism, then yes, I agree that it is scary.
I'm most scared by the apparent assumption that we have solved the human alignment problem.
Looking at history, I don't feel like our current situation of relative peace is very stable.
My impression is that "good" behavior is largely dependent on incentives, and so is the very definition of "good".
Perhaps markets are one of the more successful tools of creating aligned behaviour in humans, but even in that case it only seems to work if the powers of the market participants are balanced, which is not a luxury we have in alignment work.

You could read the status game argument the opposite way: Maybe status seeking causes moral beliefs without justifying them, in the same way that it can distort our factual beliefs about the world. If we can debunk moral beliefs by finding them to be only status-motivated, the status explanation can actually assist rational reflection on morality.

Also the quote from The Status Game conflates purely moral beliefs and factual beliefs in a way that IMO weakens its argument. It's not clear that many of the examples of crazy value systems would survive full logical and empirical information.

There is no unique eutopia. 

Sentient beings that collaborate outcompete ones that don't (not considering here, inner competition in a singleton). Collaboration means that interests between beings are traded/compromised. Better collaboration methods have a higher chance to win. We see this over the course of history. This is a messy evolutionary process. But I think there is a chance that this process itself can be improved e.g. with FAI. Think of an interactive "AlphaValue" that does Monte-Carlo Tree Search over collaboration opportunities. It will not converge on a unique best CEV but result in one of many possible eutopias. 

I don't follow the reasoning. How do you get from "most people's moral behaviour is explainable in terms of them 'playing' a status game" to "solving (some versions of) the alignment problem probably won't be enough to ensure a future that's free from astronomical waste or astronomical suffering"?

More details:
Regarding the quote from The Status Game: I have not read the book, so I'm not sure what the intended message is but this sounds like some sort of unwarranted pessimism about ppl's moral standing (something like a claim like "the vast majority of ppl are morally ugly in this way"). There is all the difference between being able to explain most ppl's behaviour in terms of "playing a status game" (in some sense of the word "playing"), and claiming that most ppl's conscious motivation to act morally is to win at the status game. The latter claim could plausibly warrant the pessimism; the former, not. But I don't see an argument for the latter. Why is the former claim not evidence for ugliness? For the same reason that the claim "a mother's love for their child is genetically 'hard-wired'" is not evidence that a mother's love for their child is ugly (or fake, not genuine etc). Explaining the underlying causal mechanisms of a given moral behaviour is not (in general) enough to warrant a given moral judgment. (If instead the book is arguing for some sort of meta-ethical anti-realism, well, then the discussion needs to be much longer...)

Regarding your fear about morality: is the worry that if we just aggregated everyone's values we would get a lock-in to some sort of "ugly" status game? Again, we need more details on its proposed implementation before we can judge whether its ugly (something to be scared of). 

But also, why are we assuming some sort of aggregation of first-order human value preferences (no matter the method of aggregation)? Assuming we're talking about AGI (and not CAIS), I always thought it strange to think we need to make sure it shares our own idealized preferences, as opposed to merely the preferences we would hope for in, say, a benevolent god or something. I don't see any a priori reason to believe that the preferences/goals of a benevolent shepherd are likely to be shared with/or strongly aligned with those of the shepherd's flock (no matter how you aggregate the preferences/goals of the flock). (I suppose it depends on the nature of the species we're talking about, but whether it's sheep or humans, I maintain my skepticism). In any case, I agree with you that a lot more meta-ethics needs to be discussed in the alignment research community.

Great post, thanks! Widespread value pluralism a la 'well that's just, like, your opinion man' is now a feature of modern life.  Here are a pair of responses from political philosophy which may be of some interest 

(1) Rawls/Thin Liberal Approach. Whilst we may not be able to agree on what 'the good life' is, we can at least agree on a basic system which ensures all participants can pursue their own idea of the good life.  So,(1) Protect a list of political liberties and freedoms and (2) degree of economic levelling. Beyond that, it is up to the individual what concept of the good they pursue. Scott Alexander's Archipelago is arguably a version of this theory, albeit with a plurality of communities rather than a single state. Note it is 'thin' but not nonexistent - obviously certain concepts of the good, such as 'killing/enslaving everyone for my God', are incompatible and excluded. 

(2) Nussbaum 'Capacity' Approach. Bit like Liberal+ approach. You take the liberal approach then beef it up by adding some more requirements: you need to protect the capacity of people to achieve wellbeing. Basically - protect life, environment, health/bodily integrity, education (scientific & creative), practical reason, being able to play, hold property, form emotional and social attachments. The main difference with (2) is that it is a thicker conception of the 'good life' - it will deny various traditional forms of life on the basis they do not educate their children or give them critical thinking skills. Hence, Nussbaum champions the notion of 'universal values.'

Going from (1) to (2) depends on how comfortable you are with an objective notion of flourishing. IMO it's not totally implausible given commonalities across cultures of values (which Nussbaum points out - moral relativism is often exaggerated) and various aspects of human experience. 

Don't you need AI to go through the many millions of experiences that it might take to develop a good morality strategy?

I'm entranced by Jordan Peterson's descriptions, which seem to light up the evolutionary path of morality for humans.  Shouldn't AI be set up to try to grind through the same progress?

What's truly scary is how much the beliefs and opinions of normal people make them seem like aliens to me. 

I find the paragraph beginning with these two sentences, its examples misleading and unconvincing in the point about moral disagreement across time it tries to make:

Such ‘facts’ also change across time.  We don’t have to travel back far to discover moral superstars holding moral views that would destroy them today.

I shall try to explain why, because such evidence seemed persuasive to me before I thought about it more; I made this account just for this comment after being a lurker for a while -- I have found your previous posts about moral uncertainty quite interesting. 

When comparing moral views across time to see if there is disagreement or agreement, we should see if there is an underlying principle that is stated by the person or one that can be inferred, which is different from the specific contextual judgement.

The application of the same principles across time to various concrete matters will widely vary depending on the context and information of the person applying them, that person's capabilities. Therefore when seeking to determine moral agreement or disagreement with a specific judgement, one should not give examples of specific practical moral judgements without also giving information about the underlying motivation and factual view of the world associated with it. 

One can see how much utilitarians with the same theory can diverge on what to do. But they disagree sometimes not in moral theory, but on things that are un-ambiguously empirical matters. So people can disagree on a specific moral judgment without that disagreement being of a moral nature. Because people across time differ quite a lot in their non-moral factual understanding of world, one would need to do research and interpretation from case to case to uncover the actual principles at play in various moral judgements which seem strange or disagreeable to us.  Without trying to consider the moral rules or principles being applied, and the understanding of the world they are being applied to, how can we determine if we disagree with someone about a moral fact, or another fact about the world?

To give a simple example about how moral judgements and factual judgements are related: it is morally easy for us to see that killing witches was wrong, because there were no witches. The communities that burned witches may have differed morally from us, but there is enough moral agreement between us and them that if they had not believed in witches or those specific false instances of witches, they would not have burned them (even when there was lying about witches involved in historical contexts of witch burning, that lying took place in a community that believed in witches and wanted to punish them accordingly). 

It would not make sense to say something like

But there’s about as much sense in blaming Gandhi for not sharing our modern, Western views on race as there is in blaming the Vikings for not having Netflix.

about the communities that burned witches. I think we can correctly say they were objectively mistaken to burn those witches. I'm not very sure about much involving morality, but that is the sort of modest 'moral realism' I am confident having when judging the historic moral decisions of past communities and individuals (I am not sure about the criteria for 'blaming' but I am confident we can say they were objectively wrong to do that, like about any other factual matter). The loosely held shared moral principle would be something like 'do not kill people for entirely mistaken reasons'. I am not saying that will be the case, locating a shared moral principle, for every past moral decision we might disagree with. But for more complicated cases we really do have to research before determining if there is a genuinely moral disagreement IMO, rather than just stating various specific judgements we seem to disagree with. 

So in the case of Marie Stopes and Gandhi, among other examples, I am unable to tell if I morally disagree with them based on the limited information given in the excerpt. I would need to know more about their view of the facts of the matters mentioned and try to understand and infer from what they have shared elsewhere, the moral views being applied (it is especially true in the case of innovative moral activists, that they will articulate their principles, so it is odd that the author does not tell us them). 

I hope I have been convincing and reasonable about why we should be wary of telling what someone's underlying moral view is and our agreement or disagreement with it from limited examples of the sort the author gives here; this is even more true for pre-modern cultures with very different understandings of the world, like the Malagasy and their taboos. How tightly wound implicit moral views and empirical views can be is clear with the case of communism, another example given. There is enough moral feeling, belief in common that many unusually-bad-to-us cases of moral behavior are also tied to a very different empirical view of the world. 

One final note, regarding status games and morality. I think the fact that many people are urged into behavior considered moral by the expectations of others who find moral behavior praiseworthy, shows that the concept of morality can be objectively applied in scenarios well outside specific people having an autonomous intrinsically moral motivation. Not everyone wants to act morally even if they recognize an objective moral ideal, but often instrumentally is driven to engage in acts considered moral, or meeting the respectable moral minimum at least, by society for their own amoral reasons. 

We can even imagine a world where no one acts for intrinsically moral reasons, but people still sometimes shape their actions around a concept of objective morality that arises because people are able to recognize their own selfish interests and recognize, want to reward someone 'altruistically' contributing to their interests or at least respecting them enough not to hurt them. I don't think that world is ours, but I do think in our world many people do not have positive moral motivations for much of their behavior and moral reformers, saint-types are known as such for having unusually strong intrinsic moral motivations and for driving people to further live up to the moral ideals they merely superficially meet for instrumental reasons most of the time.  

One can imagine 'moral saints' negotiating among various people's selfish interests to find or push people into the positive sum outcome they themselves could not selfishly have gotten into. I do not think primarily moral agents and 'enlightened' selfish ones are necessarily opposed at all (this sort of negotiation is how one can picture a broader human cooperation / alignment in the context of powerful moral human agents, in relation to the power of an aligned artificial intelligence).

The desire to appear more moral than one really is adds a lot of treachery to moral discourse in all communities; moral theorizing can offend people through intruding on illusions about how selfish they are and this trickiness is surely one reason for less progress on clarifying moral objectivity than there would be otherwise.  It seems likely that due to human moral imperfection there is ineliminable suboptimality in our actions and actual desires relative to all of our ideal moral concepts (even if they can genuinely partially motivate us in a more moral direction through their attraction as pure ideals), and that this shall be transferred to any agents that we create to satisfy us. It will probably be like Kant said, the learning of human values by an AI:

Out of the crooked timber of humanity, no straight thing was ever made

It seems that our morality consists of two elements. First is bias, based on game theoretical environment of our ancestors. Humans developed complex feelings around activities that promoted inclusive genetic fitness and now we are intrinsically and authentically motivated to do them for their own sake. 

There is also a limited capability for moral updates. That's what we use to resolve contradictions in our moral intuitions. And that's also what allow us to persuade ourselves that doing some status promoting thing is actually moral. One the one hand, the fact that it's so easy to change our ethics due to status reasons is kind of scary. On the other, whole ability to morally update was probably developed exactly for this, so it's kind of a miracle that we can use it differently at all.

I don't think that we can update into genuinely feeling that maximizing paperclips for its own sake is the right thing to do. All possible human minds occupy only a small part of all possible minds space. We can consider alignment to be somwhat solved if TAI guarantee us optimization in the direction of some neighbourhood of our moral bias. However, I think it's possible to do better and we do not need all humans to be moral philosophers for that. It will be enough if TAI itself is a perfect moral philosopher able to deduct our coherent extrapolated volition and become an optimization process in that direction.

You may not be interested in mutually exclusive compression schemas, but mutually exclusive compression schemas are interested in you. One nice thing is that given that the schemas use an arbitrary key to handshake with there is hope that they can be convinced to all get on the same arbitrary key without loss of useful structure.