What is Eliezer Yudkowsky's meta-ethical theory?

Note that your answer to (2) also answers (1): value uncertainty makes it seem as if there is substantive, fundamental normative disagreement even if there isn't. (Or maybe there is if you don't buy that particular element of EY's theory)

The answer to those objections, by the way, is that an "adequately objective" metaethics is impossible: the minds of complex agents (such as humans) are the only place in the universe where information about morality is to be found, and there are plenty of possible minds in mind-design space (paperclippers, pebblesorters, etc.) from which it is impossible to extract the same information.

Elizer attempted to deal with that problem by defining a certain set of things as "h-right", that is, morally right from the frame of reference of the human mind. He made clear that alien entities probably would not care about what is h-right, but that humans do, and that's good enough.

The answer to those objections, by the way, is that an "adequately objective" metaethics is impossible

That's not a reason to prefer EY's theory to an error theory (according to which properly normative properties would have to be irreducibly normative, but no such properties actually exist).

invested a lot of energy into training himself to think straight about things and respond to criticism either with adequate counterargument or by dropping the criticized belief

Maybe; I can't say I've noticed that so much myself -- e.g. he just disappeared from this discussion when I refuted his assumptions about philosophy of language (that underpin his objection to zombies), but I haven't seen him retract his claim that zombies are demonstrably incoherent.

Moral Error and Moral Disagreement confronts this

Yeah, I'm the "Richard4" in the comments thread there :-)

The part about computation doesn't change the fundamental structure of the theory. It's true that it creates more room for superficial disagreement and fallibility (of similar status to disagreements and fallibility regarding the effective means to some shared terminal values), but I see this as an improvement in degree and not in kind. It still doesn't allow for fundamental disagreement and fallibility, e.g. amongst logically omniscient agents.

(I take it to be a metaethical datum that even people with different terminal values, or different Eliezerian "computations", can share the concept of a normative reason, and sincerely disagree about which (if either) of their values/computations is correctly tracking the normative reasons. Similarly, we can coherently doubt whether even our coherently-extrapolated volitions would be on the right track or not.)

An AI with a simple utility function and full ability to analyze its own source code can be quite sure that maximizing that function is the meaning of "that-AI-right" in the sense EY is talking about.

I don't think that's right, or EY's position (I'd like evidence on that). Who's to say that maximization is precisely what's right? That might be a very good heuristic, but upon reflection the AI might decide to self-improve in a way that changes this subgoal (of the overall decision problem that includes all the other decision-making parts), by finding considerations that distinguish maximizing attitude to utility and the right attitude to utility. It would of course use its current utility-maximizing algorithm to come to that decision. But the conclusion might be that too much maximization is bad for environment or something. The AI would stop maximizing for the reason it's not the most maximizing thing, the same way as a person would not kill for the reason that action leads to a death, even though avoid-causing-death is not the whole morality and doesn't apply universally.

See also this comment.

Agreed that on EY's view (and my own), human "fundamental values" (1) have not yet been fully articulated/extrapolated; that we can't say with confidence whether X is in that set.

But AFAICT, EY rejects the idea (which you seem here to claim that he endorses?) that an AI with a simple utility function can be sure that maximizing that function is the right thing to do. It might believe that maximizing that function is the right thing to do, but it would be wrong. (2)

AFAICT this is precisely what RichardChappell considers implausible: the idea that unlike the AI, humans can correctly believe that maximizing their utility function is the right thing to do.

==

(1) Supposing there exist any such things, of which I am not convinced.

(2) Necessarily wrong, in fact, since on EY's view as I understand it there's one and only one right set of values, and humans currently implement it, and the set of values humans implement is irreducably complex and therefore cannot be captured by a simple utility function. Therefore, an AI maximizing a simple utility function is necessarily not doing the right thing on EY's view.

re: infallibility -- right, the objection is not that you could infallibly know that XYZ is right. Rather, the problem is that you could infallibly know that your fundamental values are right (though you might not know what your fundamental values are).

That asserting there are moral facts is incompatible with the fact that people disagree about what they are?

No, I think there are moral facts and that people disagree about what they are. But such substantive disagreement is incompatible with Eliezer's reductive view on which the very meaning of 'morality' differs from person to person. It treats 'morality' like an indexical (e.g. "I", "here", "now"), which obviously doesn't allow for real disagreement.

Compare: "I am tall." "No, I am not tall!" Such an exchange would be absurd -- the people are clearly just talking past each other, since there is no common referent for 'I'. But moral language doesn't plausibly function like this. It's perfectly sensible for one person to say, "I ought to have an abortion", and another to disagree: "No, you ought not to have an abortion". (Even if both are logically omniscient.) They aren't talking past each other. Rather, they're disagreeing about the morality of abortion.

I thought being persuaded of a metaethical theory entails that whenever the theory tells you you should do X, you would feel compelled to do X.

what Eliezer seems to be arguing is that what we really care about is not (a) that whatever states of affairs our brains are wired to send reward signals in response to be realized, but (b) that we experience peace and love and harmony and discovery and so on.

His motivation for thinking this way is a thought experiment - which might become real in the relatively near future - about what would happen if a superintelligent machine could rewire our brains. If what we really care about is (a), then we shouldn't object if the superintelligent machine rewires our brains to send reward signals only when we are sitting in a jar.

I don't see what plausible reasoning process could lead you to infer this unlikely statement (about motivation, given how many detail would need to be just right for the statement to happen to be true).

Also, even if you forbid modifying definition of human brain, things that initiate high-reward signals in our brains (or that we actually classify as "harmony" or "love") are very far from what we care about, just as whatever a calculator actually computes is not the same kind of consideration as the logically correct answer, even if you use a good calculator and aren't allowed sabotage. There are many reasons (and contexts) for reward in human brain to not be treated as indicative of goodness of a situation.

Can anything besides Gary's preferences provide a justification for saying that "Gary should_gary X"? (My own answer would be "No.")

This strikes me as an ill-formed question for reasons I tried to get at in No License To Be Human. When Gary asks "What is right?" he is asking the question e.g. "What state of affairs will help people have more fun?" and not "What state of affairs will match up with the current preferences of Gary's brain?" and the proof of this is that if you offer Gary a pill to change his preferences, Gary won't take it because this won't change what is right. Gary's preferences are about things like fairness, not about Gary's preferences. Asking what justifies should_Gary to Gary is either answered by having should_Gary wrap around and judge itself ("Why, yes, it does seem better to care about fairness than about one's own desires") or else is a malformed question implying that there is some floating detachable ontologically basic property of rightness, apart from particular right things, which could be ripped loose of happiness and applied to pain instead and make it good to do evil.

By saying "Gary should_gary X", do you mean

Shouldness does incorporate a concept of reflective equilibrium (people recognize apparent changes in their own preferences as cases of being "mistaken"), but should_Gary makes no mention of Gary (except insofar as Gary's welfare is one of Gary's terminal values) but instead is about a large logical function which explicitly mentions things like fairness and beauty. This large function is rightness which is why Gary knows that you can't change what is right by messing with Gary's brain structures or making Gary want to do something else.

Or, perhaps you are saying that one cannot give a concise definition of "should"

You can arrive at a concise metaethical understanding of what sort of thing shouldness is. It is not possible to concisely write out the large function that any particular human refers to by "should", which is why all attempts at definition seem to fall short; and since for any particular definition it always seems like "should" is detachable from that definition, this reinforces the false impression that "should" is an undefinable extra supernatural property a la Moore's Open Question.

By far the hardest part of naturalistic metaethics is getting people to realize that it changes absolutely nothing about morals or emotions, just like the fact of a deterministic physical universe never had any implications for the freeness of our will to begin with.

I also note that although morality is certainly not written down anywhere in the universe except human brains, what is written is not about human brains, it is about things like fairness; nor is it written that "being written in a human brain" grants any sort of normative status. So the more you talk about "fulfilling preferences", the less the subject matter of what you are discussing resembles the subject matter that other people are talking about when they talk about morality, which is about how to achieve things like fairness. But if you built a Friendly AI, you'd build it to copy "morality" out of the brains where that morality is written down, not try to manually program in things like fairness (except insofar as you were offering a temporary approximation explicitly defined as temporary). It is likewise extremely hard to get people to realize that this level of indirection, what Bostrom terms "indirect normativity", is as close as you can get to getting any piece of physical matter to compute what is right.

If you want to talk about the same thing other people are talking about when they talk about what's right, I suggest consulting William Frankena's wonderful list of some components of the large function:

"Life, consciousness, and activity; health and strength; pleasures and satisfactions of all or certain kinds; happiness, beatitude, contentment, etc.; truth; knowledge and true opinions of various kinds, understanding, wisdom; beauty, harmony, proportion in objects contemplated; aesthetic experience; morally good dispositions or virtues; mutual affection, love, friendship, cooperation; just distribution of goods and evils; harmony and proportion in one's own life; power and experiences of achievement; self-expression; freedom; peace, security; adventure and novelty; and good reputation, honor, esteem, etc."

(Just wanted to quote that so that I didn't entirely fail to talk about morality in between all this stuff about preferences and metaethics.)

Gary's preference is not itself justification, rather it recognizes moral arguments, and not because it's Gary's preference, but for its own specific reasons. Saying that "Gary's preference states that X is Gary_right" is roughly the same as "Gary should_Gary X".

(This should_T terminology was discouraged by Eliezer in the sequences, perhaps since it invites incorrect moral-relativistic thinking, as if any decision problem can be assumed as own by any other, and also makes you think of ways of referring to morality, while seeing it as a black box, instead of looking inside morality. And you have to look inside even to refer to it, but won't notice that until you stop referring and try looking.)

By saying "Gary should_gary X", do you mean that "Gary would X if Gary was fully informed and had reached a state of reflective equilibrium with regard to terminal values, moral arguments, and what Gary considers to be a moral argument"?

To a first approximation, but not quite, since it might be impossible to know what is right, for any computation not to speak of a mere human, only to make right guesses.

This makes should-statements "subjectively objective"

Every well-defined question has in a sense a "subjectively objective" answer: there's "subjectivity" in the way the question has to be interpreted by an agent that takes on a task of answering it, and "objectivity" in the rules of reasoning established by such interpretation, that makes some possible answers incorrect with respect to that abstract standard.

Or, perhaps you are saying that one cannot give a concise definition of "should,"

I don't quite see how this is opposed to the other points of your comment. If you actually start unpacking the notion, you'll find that it's a very long list. Alternatively, you might try referring to that list by mentioning it, but that's a tricky task for various reasons, including the need to use morality to locate (and precisely describe the location of) the list. Perhaps we can refer to morality concisely, but it's not clear how.

Can anything besides Gary's preferences provide a justification for saying that "Gary should_gary X"? (My own answer would be "No.")

Yes, natural laws. If Gary's preferences do not align with reality then Gary's preferences are objectively wrong'.

When people talk about morality they implicitly talk about fields like decision theory, game theory or economics. The mistake is to take an objective point of view, one similar to CEV. Something like CEV will result in some sort of game theoretic equilibrium. Yet each of us is a discrete agent that does not maximally value the extrapolated volition of other agents. People usually try to objectify, find a common ground, a compromise. This leads to all sorts of confusion between agents with maximally opposing terminal goals. In other words, if you are an outlier then there does exist no common ground and therefore something like CEV will be opposed.

ETA

' I should clarify what I mean with that sentence (if I want that people understand me).

I assume that Gary has a reward function and is the result of an evolutionary process. Gary should alter its preferences as they do not suit his reward function and decrease his fitness. I realize that in a sense I just move the problem onto another level. But if Gary's preferences can not be approached then they can be no justification for any action towards an implied goal. At that point the goal-oriented agent that is Gary will be functionally defunct and other more primitive processes will take over and consequently override Gary's preferences. In this sense reality demands that Gary should change his mind.

But there does exist no terminal goal that is objectively (intersubjectively) desirable.

If you mean intersubjectively say it. Objectively has a slightly different meaning. In particular, see 'objectively subjective'.

You're right about Eliezer's semantics. Count me as one of those who thought his terminology was confusing, which is why I don't use it when I try to describe the theory to anyone else.

Are you sure? I thought "should" could mean would_want. Note I could follow this by saying "That is he is comfortable answering "What should a Paperclip Maximiser do when faced with Newcomb's problem?" with "Rewrite itself to be an FAI".", but that would be affirming the consequent ;-), i.e. I know he says such a thing, but my and your formulation both plausibly explain it, as far as I know.

I think the difficulty is that in English "You" is used for "A hypothetical person". In German they use the word "Man" which is completely distinct from "Du". It might be easier to parse as "Man should_Raemon maximize Raemon's preferences, but of course man should_Matt maximize Matt's preferences."

On the jargon itself, Should_X means "Should, as X would understand it".

Consistent in what sense? Utility function over what domain? Under what prior? In this context, some unjustified assumptions, although understandably traditional to a point where objecting is weird.

but what do you mean when you say that I should_MattSimpson maximize your preferences?

I mean that according to my preferences, you, me, and everyone else should maximize them. If you ask what should_MattSimpson be done, the short answer is maximize my preferences. Similarly, if you ask what should_lukeproq be done, the short answer is to maximize your preferences. It doesn't matter who does the asking. If you ask should_agent should be done, you should maximize agent's preferences. There is no "should" only should_agent's. (Note, Eliezer calls should_human "should." I think it's an error of terminology, personally. It obscures his position somewhat).

We already have a term that matches Eliezer's use of "ought" and "should" quite nicely: it's called the "prudential ought." The term "moral ought" is usually applied to a different location in concept space, whether or not it successfully refers.

Then Eliezer's position is that all normativity is prudential normativity. But without the pop-culture connotations that come with this position. In other words, this doesn't mean you can "do whatever you want." You probably do, in fact, value other people, you're a human after all. So murdering them is not ok, even if you know you can get away with it. (Note that this last conclusion might be salvageable even if there is no should_human.)

As for why Eliezer (and others here) think there is a should_human (or that human values are similar enough to talk about such a thing), the essence of the argument rests on ev-psych, but I don't know the details beyond "ev-psych suggests that our minds would be very similar."

And yes, if one's own preferences are the foundation of ethics, most philosophers would simply call this subject matter practical rationality rather than morality.

They would be missing some important distinctions between what we think of as our moral values and what we think of as "chocolate/vanilla" preferences. For one obvious example, consider an alien ray gun that 'switches the way I feel' about two things, X and Y, without otherwise affecting my utility function or anything else of value to me.

If X were, say, licorice jelly beans (yum) and Y were, say, buttered popcorn jelly beans (yuck), then I wouldn't be too deeply bothered by the prospect of being zapped with this gun. (Same for sexual preference, etc.) But if X were "autonomy of individuals" and Y were "uniformity of individuals", I would flee screaming from the prospect of being messed with that way, and would take some extreme actions (if I knew I'd be zapped) to prevent my new preferences from having large effects in the world.

Now we can develop whole theories about what this kind of difference consists in, but it's at least relevant to the question of metaethics. In fact, I think that calling this wider class of volitions "preferences" is sneaking in an unfortunate connotation that they "shouldn't really matter then".

I certainly balk at the suggestion that there is a should_human, but I'd need to understand Eliezer in more detail on that point.

We'd need to do something specific with the world, there's no reason any one person gets to have the privilege, and creating an agent for every human and having them fight it out is probably not the best possible solution.

No, this is called preference utilitarianism.

Usually utilitarianism means maximize the utility of all people/agents/beings of moral worth (average or sum depending on the flavor of utilitarianism). Eliezer's metaethics says only maximize your own utility. There is a clear distinction.

Edit: but you are correct about considering preferences the foundation of ethics. I should have been more clear

I don't think the coherence part is particularly relevant here.

Consider two people, you (Peter) and me (Matt). Suppose I prefer to be able to murder people and you prefer that no one ever be murdered. Suppose I have the opportunity to murder someone (call him John) without getting caught or causing any other relevant positive or negative consequences (both under your preferences and mine). What should I do? Well, I should_Matt murder John. My preferences say "yay murder" and there are no downsides, so I should_Matt go ahead with it. But I should_Peter NOT murder John. Your preferences say "boo murder" and there are no other benefits to murdering John, so I should_Peter just leave John alone. But what should I do? Tell me what you mean by should and I'll tell you. Presumably you mean should_Peter or should_(most people), in which case, then I shouldn't murder.

(EY's theory would further add that I don't, in fact, value murder as an empirical claim - and that would be correct, but it isn't particularly relevant to the hypothetical. It may, however, be relevant to this method of testing metaethical theories, depending on how you intended to use it.)

EY-metaethics, as you have stated it, doesn't wash too well, since there is an implication that those who value murder should murder, those who value paperclips should maximise paperclips, etc.

Let me fix that sentence for you:

EY-metaethics, as you have stated it, doesn't wash too well, since there is an implication that those who value murder should_(those who value murder) murder, those who value paperclips should_(those who value paperclips) maximise paperclips, etc.

In other words, there is no "should," unless you define it to be a specific should_x. EY would define it as should_(human CEV) or something similar, and that's the "should" you should be running through the test.

Some will recognise that as a form of the well known and widely rejected theory of ethical egoism.

It isn't. Egoism says be selfish. There's no reason why someone can't have altruistic preferences, and in fact people do. (Unless that's not what you mean by egoism, but sure, this is egoism, but that's a misleading definition and the connotations don't apply).

What wedrifid said. But also, what is the criterion by which you would change your (extrapolated) preferences? This criterion must contain some or all of the things that you care about. Therefore, by definition it's part of your current (extrapolated) preferences. Edit: Which tells you that under "normal" circumstances you won't prefer to change your preferences.

My current prefernces? Why shouldn't I change them?

Current extrapolated preferences. That is, maximise whatever it is that you want to change your preferences to.

I think the term nihilism is getting in the way here. Let's instead talk about "the zero axiom system". This is where you don't say that any universes are morally preferable to any others. They may be appetite-preferable, love-for-people-close-to-you preferable, etc.

If no universes are morally preferable, one strategy is to be as ruthlessly self-serving as possible. I predict this would fail to make most people happy, however, because most people have a desire to help others as well as themselves.

So a second strategy is to just "go with the flow" and let yourself give as much as your knee-jerk guilt or sympathy-driven reactions tell you to. You don't research charities and you still eat meat, but maybe you give to a disaster relief appeal when the people suffering are rich enough or similar enough to you to make you sympathetic.

All I'm really saying is that this second approach is also anti-strategic once you get to a certain level of self-consistency, and desire for further self-consistency becomes strong enough to over-rule desire for some other comforts.

I find myself in a bind where I can't care nothing, and I can't just follow my emotional moral compass. I must instead adopt making the world a better place as a top-level goal, and work strategically to make that happen. That requires me to adopt some definition of what constitutes a better universe that isn't rooted in my self-interest. In other words, my self-interest depends on having goals that don't themselves refer to my self-interest. And those goals have to do that in entirely good-faith. I can't fake this, because that contradicts my need for self-consistency.

In other words, I'm saying that someone becomes vegetarian when their need for a consistent self-image about whether they behave morally starts to over-rule the sensory, health and social benefits of eating meat. Someone starts to tithe to charity when their need for moral consistency starts to over-rule their need for an extra 10% of their income.

So you can always do the calculations about why someone did something, and take it back to their self-interest, and what strategies they're using to achieve that self-interest. Utilitarianism is just the strategy of adopting self-external goals as a way to meet your need for some self-image or guilt-reassurance. But it's powerful because it's difficult to fake: if you adopt this goal of making the world a better place, you can then start calculating.

There are some people who see the fact that this is all derivable from self-interest, and think that it means it isn't moral. They say "well okay, you just have these needs that make you do x, y or z, and those things just happen to help other people. You're still being selfish!".

This is just arguing about the meaning of "moral", and defining it in a way that I believe is actually impossible. What matters is that the people are helped. What matters is the actual outcomes of your actions. If someone doesn't care what happens to other people at all, they are amoral. If someone cares only enough to give $2 to a backpacker in a koala suit once every six months, they are a very little bit moral. Someone who cares enough to sincerely try to solve problems and gets things done is very moral. What matters is what's likely to happen.

Why distinguish between moral and non-moral preferences? Why are moral preferences more mutable than non-moral ones?

The basic drive to adopt some sort of ethical system is essentially the same as other preferences, and is non-mutable. It's a preference to believe that you are making the world a better place, rather than a worse place. This introduces a definitional question of what constitutes a good world and what constitutes a bad world, which is something I think people can change their minds about.

Having written that, one question that occurs to me now is, is the basic preference to believe that you're making the world a better place, or is it to simply believe you're a good person? I prefer people who make the world a better place, so the two produce the same outcomes for me. But other people might not. If you instead had a preference for people who followed good principles or exhibited certain virtues, you wouldn't feel it necessary to make the world a better place. I shouldn't assume that such people don't exist.

So maybe instead of talking adopting a definition of which universes are good and bad, I should talk about adopting a definition of good and bad people. If you define a good person by the consequences of their actions, then you'd go on to define which universes are good and bad. But otherwise you might instead define which principles are good, or which virtues.

I have much the same attitude toward Anglophone philosophy

Non-anglophone philosophy is worse. (Phenomenology, deconstructionism,...)

Given all the actually content-bearing comments in this discussion, the +7 points of this remark somehow saddens me.

One project is the descriptive one of moral psychology and moral anthropology. Because Coherent Extrapolated Volition begins with data from moral psychology and moral anthropology, that descriptive project is important for Eliezer's design of Friendly AI.

Moral psychology and anthropology are pretty useless, because morality is too complex for humans to manually capture with accuracy, and too fragile to allow capturing without accuracy. We need better tools.

(1) Whatever moral facts exist, they must be part of the natural world. (Moral naturalism.)

In a manner of speaking, yes. Moral facts are facts about the output of a particular computation under particular conditions, so they are "part of the natural world" essentially to whatever extent you'd say the same thing about mathematical deductions. (See Math is Subjunctively Objective, Morality as Fixed Computation, and Abstracted Idealized Dynamics.)

(2) Moral facts are not written into the "book" of the universe - values must be derived from a consideration of preferences. (In philosophical parlance, this would be something like the claim that "The only sources of normativity are relations between preferences and states of affairs.")

No. Caring about people's preferences is part of morality, and an important part, I think, but it is not the entirety of morality, or the source of morality. (I'm not sure what a "source of normativity" is; does that refer to the causal history behind someone being moved by a moral argument, or something else?)

(The "Moral facts are not written into the 'book' of the universe" bit is correct.)

(3) What I "should" do is determined by what actions would best fulfill my preferences. (This is just a shorter way of saying that I "should" do "what I would do to satisfy my terminal values if I had correct and complete knowledge of what actions would satisfy my terminal values.")

See Inseparably Right and No License To Be Human. "Should" is not defined by your terminal values or preferences; although human minds (and things causally entangled with human minds) are the only places we can expect to find information about morality, morality is not defined by being found in human minds. It's the other way around: you happen to care about(/prefer/terminally value) being moral. If we defined "should" such that an agent "should" do whatever satisfies its terminal values (such that pebblesorters should sort pebbles into prime heaps, etc.), then morality would be a Type 2 calculator; it would have no content, it could say anything and still be correct about the question it's being asked. I suppose you could define "should" that way, but it's not an adequate unpacking of what humans are actually thinking about when they talk about morality.

At least as I understood it, yes (though I'm unsure of the "my" in "my terminal values" part, as I cannot glue it together with his vision of CEV and a singleton).

He also writes about the importance of a process of reflective equilibrium, though I'm not sure to what end.

To handle value uncertainty. If you don't know your terminal values, you have to discover them somehow.

...an alien ray gun that changed your ice cream preference from chocolate to vanilla, but you wouldn't be okay being hit with an alien ray gun that changed your preferences from not-wanting-to-rape-people to wanting-to-rape-people.

I'm completely lost about that. I don't see how vanilla preferences differ from rape preferences. We just happen to weigh them differently. But that is solely a fact about our evolutionary history.

Hm. I can say truthfully that I don't care whether I like vanilla or chocolate ice cream more. I suppose that the statement of my utility with regard to eating vanilla vs. chocolate ice cream would be 'I assign higher utility to eating the flavor of ice cream which tastes better to me.' That is, I only care about a state of my mind. So, if the circumstances changed so I could procure that state of mind by other means (ex: eating vanilla instead of chocolate ice cream), I would have no problem with that. The action that I would take after being hit by the alien ray gun does not give me any less utility after being hit by the alien ray gun than the action that I take now gives me in the present. So I don't care whether I get hit by the ray gun.

But my statement of utility with regard to people being raped would be "I assign much lower utility to someone being raped them not being raped." Here, I care about a state of the world outside of my mind. The action that I would take after being hit by the alien ray gun (rape) has less utility under my current utility function than (~rape), so my current utility function would assign negative utility to being hit by the ray gun.

This much makes sense to me.

I don't know what 'reflective equilibrium' means; this may be because I didn't really make it through the metaethics sequence. After I formulated what I've said in this comment and the above one, I wasn't getting much out of it.

Edit: Inserted some italics for the main difference between the two scenarios and removed a set of italics. No content changes.

Killing: generally wrong.

Do you agree with EY on Torture vs Dust Specks? If you agree, would killing one person be justified to save 3^^^3 from being killed? If you agree, would you call killing to be right in that case?

Um, I'll suggest that. Killing: generally wrong.

I realize I might have misunderstood moral realism. I thought moral realism proposes that there do exist agent-independent moral laws. What I meant is that nobody would suggest that the propostion 'Killing: generally wrong' is a subvenient property.