If it were morally correct to kill everyone on earth, would you do it?

I don't think that's what I'm asking. Here's an analogy. A person X comes to the conclusion fairly late in life that the morally best thing they can think of to do is to kill themselves in a way that looks like an accident and will their sizable life insurance policy to charity. This conclusion isn't a reducto ad absurdum of X's moral philosophy, even if X doesn't like it. Regardless of this particular example, it could presumably be correct for a person to sacrifice themselves in a way that doesn't feel heroic, isn't socially accepted, and doesn't save the whole world but maybe only a few far-away people. I think most people in such a situation (who managed not to rationalize the dilemma away) would probably not do it.

So I'm trying to envision the same situation for humanity as a whole. Is there any situation that humanity could face that would make us collectively say "Yeah doing Y is right, even though it seems bad for us. But the sacrifice is too great, we aren't going to do it". That is, if there's room for space between "considered morality" and "desires" for an individual, is there room for space between them for a species?

People aren't motivated by morality alone -- people aren't required to do what they recognize to be morally correct.

e.g. a parent may choose their kid's life over the lives of a hundred other children. Because they care more about their own child -- not because they think it's the morally correct thing to do.

Our moral sense is only one of the many things that motivate us.

Congratulations, you have successfully answered the title!

Now, on to the actual post ...

it might be that I have to die to save a marginal second, which turns out to be valuable to future humanity

Not "to future humanity", but instead according to your own extrapolated values. There might be no future humanity in this hypothetical (say, FAI determines that it's better for there to be no people). The interesting form of this paradox is where you get a stark conflict between your own extrapolated values and your intuitive perception of what you value.

Why is this so heavily downvoted?

I believe the criterion is "I want to see fewer things like this, so I'll downvote."

For the lazy, the punchline:

And if an external objective morality does say that the universe should occupy some horrifying state... let's not even ask what you're going to do about that. No, instead I ask: What would you have wished for the external objective morality to be instead? What's the best news you could have gotten, reading that stone tablet?

Go ahead. Indulge your fantasy. Would you want the stone tablet to say people should die of old age, or that people should live as long as they wanted? If you could write the stone tablet yourself, what would it say?

Maybe you should just do that?

I mean... if an external objective morality tells you to kill people, why should you even listen?

No. Humans are not perfect moral reasoners; if he booted up a Friendly superintelligence and it mentioned black people, should a racist conclude he screwed up?

It's unlikely, to my mind, that the arguments presented by the OP are correct - but they are not trivially false. There are people who already espouse similar views; hell, there are people who believe we should all commit suicide now, to spare future generations the pain of living. To simply say "I don't want to do that" is a fully general counteragument, and if it was properly integrated you would be immune to any moral argument.

(You would also have to conclude that everyone but you has a wildly different utility function, which would rather defeat the purpose of CEV.)

What if some "people" can have unbounded utility functions, where other "people" cannot. Do you think it's still appropriate to apply that heuristic?

Thanks for this response. One comment about one of your main points: I agree that the tradeoff of number of humans vs. length of life is ambiguous. But to the extent our utility function favors numbers of people over total life span, that makes the second scenario more plausible, whereas if total life span is more important, the first is more plausible.

I agree with you that both the scenarios would be totally unacceptable to me personally, because of my limited altruism. I would badly want to stop it from happening, and I would oppose creating any AI that did it. But I disagree in that I can't say that any such AI is unfriendly or "evil". Maybe if I was less egoistic, and had a better capacity to understand the consequences, I really would feel the sacrifice was worth it.

If your utility function is based on total humanity lifetime value, and you're completely altruistic, sure. Most people don't seem to be all that altruistic, though. If I had to choose between saving one or two human beings, I would choose the latter option, but I'd never sacrifice myself to save a measly two humans.

That seems like a bias/heuristic; people are known to be biased in favor of themseves, and there is instrumental value in more life to help people with.

Friendly AIs would behave differently, for one thing.

I don't assign any negative utility to ending a life.

Um ... most people don't want to die. That in and of itself would seem to suggest you may possibly have gone wrong somewhere in this line of reasoning.

More generally, are you a strict hedonic utilitarian? Because I can see how focusing solely on pleasure etc. could lead to that conclusion, but I think most LWers are closer to preference utilitarians.

I don't assign any negative utility to ending a life. It's not like it's something you can experience.

What? Are you talking about decision theoretic utility or hedonic utility? You can't experience any decision-utility, and you can't make decisions based on hedonic utility (directly). What are you trying to do?