Could evolution have selected for moral realism?

I think this is more the distinction between "objectivism" and "subjectivism", rather than between "realism" and "anti-realism".

Let's suppose that different moral agents find they are using different U-functions to evaluate consequences. Each agent describes their U as just "good" (simpliciter) rather than as "good for me" or as "good from my point of view". Each agent is utterly sincere in their ascription. Neither agent has any inconsistency in their functions, or any reflective inconsistency (i.e. neither discovers that under their existing U it would be better for them to adopt some other U' as a function instead). Neither can be persuaded to change their mind, no matter how much additional information is discovered.

In that case, we have a form of moral "subjectivism" - basically each agent has a different concept of good, and their concepts are not reconcilable. Yet for each agent there are genuine facts of the matter about what would maximize their U, so we have a form of moral "realism" as well.

Agree though that the definitions aren't precise, and many people equate "objectivism" with "realism".

What some here might call The Superinteligent Will but I see as a logical outgrowth of der Wille zur Macht, is to stamp your values on a uncaring universe.

You totally stole that from me!

What some here might call The Superinteligent Will but I see as a logical outgrowth of der Wille zur Macht, is to stamp your values on a uncaring universe.

Is this why people like Nietzsche, or do most people who like Nietzsche have different reasons?

Our values are fragile, some see this as a reason to not be too concerned with them.

I think it really depends on the exact value change we're talking about. There's an analogue for death/aging--you'd probably greatly prefer aging another 10 years, then being frozen at that biological age forever, over aging and dying normally. In the same way, I might not consider a small drift in apparently unimportant values to big a deal in the grand scheme of things, and might not choose to spend resources guarding against this (slippery slope scenarios aside).

In practice, people don't seem to be that concerned with guarding against small value changes. They do things like travel to new places, make new friends, read books, change religions, etc., all of which are likely to change what they value, often in unpredictable ways.

I think most LW readers don't see much sacrosanct about evolved values

Maybe because they think about them in far mode. If you think about values as some ancient commandments written on some old parchment, it does not seem like rewriting the parchment could be a problem.

Let's try it in the near mode. Imagine that 1000 years later you are defrosted and see a society optimized for... maximum suffering and torture. You are explained that it happened as a result of an experiment to initialize the superhuman AI with random values... and this was what the random generator has generated. It will be like this till the end of the universe. Enjoy the hell.

What is your reaction on this? Some values were replaced by some other values -- thinking abstractly enough, it seems like nothing essential has changed; we are just optimizing for Y instead of X. Most of the algorithm is the same. Even many of the AI actions are the same: it tries to better understand human psychology and physiology, get more resources, protect itself against failure or sabotage, self-improve, etc.

How could you explain what is wrong with this scenario, without using some of our evolved values in your arguments? Do you think that a pebblesorter, concerned only with sorting pebbles, would see an important difference between "human hell" and "human paradise" scenarios? Do you consider this neutrality of pebblesorter with regards to human concerns (and a neutrality of humans with regards to pebblesorter concerns) to be a desirable outcome?

(No offense to pebblesorters. If we ever meet them, I hope we can cooperate to create a universe with a lot of happy humans and properly sorted heaps of pebbles.)

It's also not even clear quite what you mean by "initial" mix. My values as a 3-year-old were much different than the ones I have today. My values prior to creating this post were likely different from my values after creating it. Which set >of values is the "initial" one that is "all I have to go on"?

Sorry, I should have been more clear about that. What I mean is that at any particular moment when one reflects upon their values, one can only use one's current value system to do so. The human value system is dynamic.

Where does inculturation stop and moral reflection begin? Is there any reason to distinguish them?

Like many things in nature, there is no perfectly clear distinction. I generally consider values that I have reflected upon to any degree, especially using my "rational thought" value, to be safe and not dogma.

Do you consider God's existence to be an "is" factual question or an "ought" values question? I consider it a factual question myself.

My "rational thought" value tells me it's an "is" question, but most people seem to consider it a value question.

If it is, hopefully "I could generate values randomly" is a workable counterargument.

But why would you do that if your existing value system wouldn't find that a good idea?

I don't think it's a fact that stealing stuff is wrong, unless you're using "wrong" as a shorthand to refer to things that have strong/widespread taboos against them.

But you also said that some such widespread conventions/taboos are good conventions. From your OP:

Yes, the rule that you can't kill people is a very good convention, and it makes sense to have heavy default punishments for breaking it.

So, here's a meta-question for you. Do you think it is a fact that "the rule that you can't kill people is a very good convention". Or was that just a matter of subjective opinion, which you expressed in the form of a factual claim for rhetorical impact? Or is it itself a convention (i.e. we have conventions to call certain things "good" and "bad" in the same way we have conventions to call certain things "right" and "wrong")?

On a related point, notice that certain conventions do create facts. It is a convention that Obama is called president, but also a fact that he is president. It is a convention that dollar bills can be used as money, and a fact that they are a form of money.

Or imagine arguing the following "It is a convention that objects with flat surfaces and four solid legs supporting them are called tables, but that doesn't mean there are any real tables".

To the best of my understanding, "subjunctively objective" means the same thing that "subjective" means in ordinary speech: dependent on something external, and objective once that something is specified. So Eliezer's morality is objective once you specify that it's his morality (or human morality, etc.) and then propositions about it can be true or false. "Turning a person into paperclips is wrong" is an ethical proposition that is Eliezer-true and Human-true and Paperclipper-false, and Eliezer's "subjunctive objective" view is that we should just call that "true".

I disagree with that approach because this is exactly what is called being "subjective" by most people, and so it's misleading. As if the existing confusion over philosophical word games wasn't bad enough.

This is why I voted for moral realism. If instead Moral realism is supposed to mean something stronger, then I'm probably not a moral realist.

The entire issue is a bit of a mess.

http://plato.stanford.edu/entries/moral-anti-realism/

the possibility remains open that there can exist universal moral attractive fixed sets, deriving entirely from such "is" transformations, regardless of the opening person-specific or species-specific moral set,

So you've got these attractive sets and maybe 90% or 99% or 99.9% or 99.99% of humans or humans plus some broader category of conscious/intelligent entities agree. What to do about the exceptions? Pretend they don't exist? Kill them because they are different and then pretend they never existed or couldn't exist? In my opinion, what you have as a fact is that 99.999% of humans agree X is wrong and .001% don't. The question of moral realism is not a factual one, it is a question of choice: do you CHOOSE to declare what 99.999% have an intuition towards as binding on the .001% that don't, or do you CHOOSE to believe that the facts are that the various intuitions have prevalences, some higher than others, some very high indeed, and that's all you actually KNOW.

I effectively feel bound by a lot of my moral intuitions, that is more or less a fact. As near as I can tell, my moral intuitions evolved as part of the social development of animals, then mammals, then primates, then homo. It is rational to assume that the mix of moral intuitions is fairly fine-tuned to optimize the social contribution to our species fitness, and it is more or less a condensation of facts to say that the social contribution to our fitness is larger than the social contribution to any other species on the planet to their fitness.

So I accept that human moral intuition is an organ like the brain or the islets of langerhans. I accept that a fair amount can be said about how the islets of langerhans function, and how the brain functions, when things are going well. Also, we know a lot about how the islets of langerhans and how the brain function when things are apparently not going so well, diseases one might say. I'd even go so far as to say I would prefer to live in a society dominated by people without diabetes and who are not sociopaths (people who seem to lack many common moral intuitions). I'd go so far as to say I would support policies including killing sociopaths and their minions, and including spending only a finite amount of resources on more expensive non-killing ways of dealing with sociopaths and diabetics.

But it is hard for me to accept that it is rational to fall in to the system instead of seeing it from outside. For me to conclude that my moral intuitions are objectively real like the charge on an electron of the electronic properties of doped silicon is projection, seems to me. It is identical to my concluding that one mammal is beautiful and sexy and another is dull, when it is really the triggering of an evolved sexual mechanism in me that paints the one mammal one way and the other the more boring way. If it is more accurate to understand that the fact that I am attracted to one mammal is not because she is objectively more beautiful than another, then it is more accurate to say that the fact that I have a moral intuition is not because I am plugged in to some moral fact of the universe, and not because of an evolved reaction I have. The fact that most men or many men find woman A beautiful and woman B to be blah doesn't mean that all men ought to find A beautiful and B blah, any more than the fact that many (modern) men feel slavery is wrong means they are not projecting their social construct into a realm of fact which could fruitfully be held to a higher standard.

Indeed, the fact that believing that our social constructs, our political truths, are REAL truths is clearly adaptive in the social species. Societies that encourage strong identifications with the values of the society are robust. Societies in which it is right to kill the apostates because they are wrong, evil, have a staying power. But my life as a scientist has consisted of my understanding that my wanting something to be true is not ANY evidence for its truth. I bring that to my American humanity. So even though I will support the killing of our enemies, I don't think that it is a FACT that it is right to kill the enemies of America any more than it is a FACT that it is right to kill the enemies of Islam.

Interesting point, but where's the problem?

Reading philosophy as an aid in moral judgement.