A (small) critique of total utilitarianism

For instance, it seems that there is only a small difference between the happiness of richer nations and poorer nations, while the first consume many more resources than the second. Hence to increase utility we should simply kill off all the rich, and let the poor multiply to take their place (continually bumping off any of the poor that gets too rich).

This empirical claim seems ludicrously wrong, which I find distracting from the ethical claims. Most people in rich countries (except for those unable or unwilling to work or produce kids who will) are increasing the rate of technological advance by creating demand for improved versions of products, paying taxes, contributing to the above-average local political cultures, and similar. Such advance dominates resource consumption in affecting the welfare of the global poor (and long-term welfare of future people). They make charitable donations or buy products that enrich people like Bill Gates and Warren Buffett who make highly effective donations, and pay taxes for international aid.

The scientists and farmers use thousands of products and infrastructure provided by the rest of society, and this neglects industry, resource extraction, and the many supporting sectors that make productivity in primary and secondary production so far (accountants, financial markets, policing, public health, firefighting...). Even "frivolous" sectors like Hollywood generate a lot of consumer surplus around the world (they watch Hollywood movies in sub-Saharan Africa), and sometimes create net rewards for working harder to afford more luxuries (sometimes they may encourage leisure too much by a utilitarian standard).

Regarding other points:

fact that you should follow a utility function in no way compel you towards total utilitarianism

Yes, this is silly equivocation exacerbated by the use of similar-sounding words for several concepts, and one does occasionally see people making this error.

interpersonal utility comparisons (IUC)

The whole piece assumes preference utilitarianism, but much of it also applies to hedonistic utilitarianism: you need to make seemingly-arbitrary choices in interpersonal happiness/pleasure comparison as well.

When considering competing moral theories, total utilitarianism does not "win by default" thanks to its large values as the population increases.

I agree.

The most compelling argument for total utilitarianism (basically the one that establishes the repugnant conclusion), is a very long chain of imperfect reasoning, so there is no reason for the conclusion to be solid. Considering the preferences of non-existent beings does not establish total utilitarianism.

Maybe just point to the Stanford Encyclopedia of Philosophy entry and a few standard sources on this? This has been covered very heavily by philosophers, if not ad nauseam.

There is no natural scale on which to compare utility functions. [...] Unless your theory comes with a particular [interpersonal utility comparison] method, the only way of summing these utilities is to do an essentially arbitrary choice for each individual before summing. Thus standard total utilitarianism is an arbitrary sum of ill defined, non-natural objects.

This, in my opinion, is by itself a decisive argument against utilitarianism. Without these ghostly "utilities" that are supposed to be measurable and comparable interpersonally, the whole concept doesn't even being to make sense. And yet the problem is commonly ignored routinely and nonchalantly, even here, where people pride themselves on fearless and consistent reductionism.

Note that the problem is much more fundamental than just the mathematical difficulties and counter-intuitive implications of formal utilitarian theories. Even if there were no such problems, it would still be the case that the whole theory rests on an entirely imaginary foundation. Ultimately, it's a system that postulates some metaphysical entities and a categorical moral imperative stated in terms of the supposed state of these entities. Why would we privilege that over systems that postulate metaphysical entities and associated categorical imperatives of different kinds, like e.g. traditional religions?

(If someone believes that there is a way how these interpersonally comparable utilities could actually be grounded in physical reality, I'd be extremely curious to hear it.)

In total utilitarianism, it is a morally neutral act to kill someone (in a painless and unexpected manner) and creating/giving birth to another being of comparable happiness.

I stopped reading here. To me, "total utilitarianism" means maximizing the sum of the values of individual lives. There's nothing forcing a total utilitarian to value a life by adding the happiness experienced in each moment of the life, without further regard to how the moments fit together (e.g. whether they fulfill someone's age-old hopes).

In general, people seem to mean different things by "utilitarianism", so any criticism needs to spell out what version of utilitarianism it's attacking, and acknowledge that the particular version of utilitarianism may not include everyone who self-identifies as a utilitarian.

A utility function does not compel total (or average) utilitarianism

Does anyone actually think this? Thinking that utility functions are the right way to talk about rationality !=> utilitarianism. Or any moral theory, as far as I can tell. I don't think I've seen anyone on LW actually arguing that implication, although I think most would affirm the antecedent.

There is a seemingly sound argument for the repugnant conclusion, which goes some way towards making total utilitarianism plausible. It goes like this... If all these steps increase the quality of the outcome (and it seems intuitively that they do), then the end state much be better than the starting state, agreeing with total utilitarianism

This is the complete opposite of what I'd understood the point of that argument to be: as I understand it, it's claimed that the final state is clearly not of high utility, and so there is something wrong with total utilitarianism. Which is fine for what you're arguing, but you seem to have taken it a bit the wrong way around.

As for the mathematical rigour, there are some very nice impossibility theorems proved by Arrhenius (example) that make the kind of worries exemplified by the repugnant conclusion a lot more precise. They don't even require the problematic assumptions about utility functions that you point out: they're just about axiology (ranking possible outcomes). So they're actually independent problems for utilitarians.

I think a lot of the reason that utilitarians don't tend to feel terribly worried about the difficulty of interpersonal utility calculations is that we already do them. Every time you decide to let someone else have the last cookie because they'll enjoy it more, you just did a little IUC. Obviously, it's pretty unclear how to scale that up, but it gives a strong feeling that it ought to be possible, somehow.

What seems to be overlooked in most discussions about total hedonistic utiltiarianism is that the proponents often have a specific (Parfitean) view about personal identity. Which leads to either empty or open individualism. Based on that, they may hold that it is no more rational to care about one's own future self than it is to care about any other future self. "Killing" a being would then just be failing to let a new moment of consciousness come into existence. And any notions of "preferences" would not really make sense anymore, only instrumentally.

A smaller critique of total utilitarianism:

In total utilitarianism, it is a morally neutral act to kill someone (in a painless and unexpected manner) and creating/giving birth to another being of comparable happiness (or preference satisfaction or welfare).

You can just finish there.

(In case the "sufficient cause to reject total utilitarianism" isn't clear: I don't like murder. Total utilitarianism advocates it in all sorts of scenarios that I would not. Therefore, total utilitarianism is Evil.)

Well, one reason is that there are models that make use of something like total utilitarianism to great effect. Classical economic theory, for instance, models everyone as perfectly rational expected utility maximisers.

I think you actually slightly understate the case against Utilitarianism. Yes, Classical Economics uses expected utility maximisers - but it prefers to deal with Pareto Improvements (or Kaldor-Hicks improvements) than try to do inter-personal utility comparisons.

Total utilitarianism is defined as maximising the sum of everyone's individual utility function.

That seems misleading. Most of the time "total utiltiarianism" refers to what should actually be called "hedonistic total utilitarianism". And what is maximized there is the suprlus of happiness over suffering (positive hedonic states over negative ones), which isn't necessarily synonymous with individual utility functions.

There are three different parameters for the various kinds of utilitarianism: It can either be total or average or prior-existence. Then it can be negative or classical (and in theory also "positive", even though that would be insane, forcing people to accept eternal torture if there's even the slightest chance of a moment of happiness). And then utiltiarianism can also be hedonistic or preference. Most common, and subject to this article, is (classical) total hedonistic utiltiarianism. While some combinations make very little sense, a lot of them actually have advocates. (For instance, recently someone published a paper advocating "negative average preference-utilitarianism".)

Upvoted, but as someone who, without quite being a total utilitarian, at least hopes someone might be able to rescue total utilitarianism, I don't find much to disagree with here. Points 1, 4, 5, and 6 are arguments against certain claims that total utilitarianism should be obviously true, but not arguments that it doesn't happen to be true.

Point 2 states that total utilitarianism won't magically implement itself and requires "technology" rather than philosophy; that is, people have to come up with specific contingent techniques of estimating utility, rather than just reading it off via a simple method which can be proven mathematically perfect. But we have some Stone Age utility-comparing technologies like money and the popular vote, and QALYs might be metaphorically a Bronze Age technology. I suppose I take it on faith that there's a lot of room for more advanced technology before we hit mathematical limits.

That leaves the introductory paragraph and Point 3 as the only places I still disagree:

In total utilitarianism, it is a morally neutral act to kill someone (in a painless and unexpected manner) and creating/giving birth to another being of comparable happiness (or preference satisfaction or welfare).

In hedonic utilitarianism, yes. Are you making this claim for preference utilitarianism as well? If so, on what basis? If we don't give credit for creating potential people, isn't most people's preference not to be killed enough to stop preference utilitarians from killing them?

And you also have to be certain that your theory does not allow path dependency. One can take the perfectly valid position that "If there were an existing poorer population, then the right thing to do would be to redistribute wealth, and thus lose the last copy of Akira. However, currently there is no existing poor population, hence I would oppose it coming into being, precisely because it would result in the lose of Akira." You can reject this type of reasoning, and a variety of others that block the repugnant conclusion at some stage of the chain (the Stanford Encyclopaedia of Philosophy has a good entry on the Repugnant Conclusion and the arguments surrounding it). But most reasons for doing so already pre-suppose total utilitarianism. In that case, you cannot use the above as an argument for your theory.

Can you explain this further? If we don't allow potential people to carry weight, and if we are preference rather than hedonic utilitarians, then the only thing we are checking when deciding to create all these new people is whether or not existing people would prefer to do so.

The fact that the repugnant conclusion has "repugnant" right in the name suggests that most people don't want it. Therefore if total utilitarianism is about satisfying the preferences of as many people as possible much as possible, and it results in a conclusion nobody prefers, that should be a red flag.

If existing people understand the repugnant conclusion, then they will understand it is a likely consequence of creating all these people is that the world loses most of its culture and happiness, and when we aggregate their preferences they will vote against doing so.

So I don't see what you mean when you say this reasoning "pre-supposes total utiltarianism". It presupposes people's intuitive moral preferences for a happy world full of culture to a just-barely-not-unhappy-world without, and it pretends we can solve the aggregation problem, but where's the vicious self-reference?

This would deserve to be on the front page.

There's how I see this issue (from philosophical point of view):

Moral value is, in the most general form, a function of a state of a structure, for lack of better word. The structure may be just 10 neurons in isolation, for which the moral worth may well be exact zero, or it may be 7 billion blobs of about 10^11 neurons who communicate with each other, or it may be a lot of data on a hard drive, representing a stored upload.

The moral value of two interconnected structures, in general, does not equal the sum of moral value of each structure (example: whole brain vs piece of brain, a mind on redundant hardware). Moral value of whole can (in general) be greater or less than sum of moral values of the parts. Note that I have not defined anything specific at all here, I just specified very general considerations. We have developed somewhat ad hoc approximations to some sort of ideal moral worth.

edit: Note. The moral worth of an action is in general a function of state without the action and state with the action, not necessarily difference in moral worth of one state, and moral worth of other state.

The utilitarianism of the N dustspecks worse than torture variety takes as fundamental and the ideal plenty of assumptions such as that moral worth would be distributive like W(a .. b) = W(a) + W(b) , to which we have clear counter-examples when the parts are strongly interconnected (e.g. 2 hemispheres of a brain) or correlated (double redundant hardware) but which may hold approximately for people due to them not being strongly interconnected. With very large N, clearly broken premise is taken to their extreme and then proclaimed normative, while the approximations that aren't linear are proclaimed wrong.

In total utilitarianism, it is a morally neutral act to kill someone (in a painless and unexpected manner) and creating/giving birth to another being of comparable happiness (or preference satisfaction or welfare).

Just wanted to note that this is too strong a statement. There is no requirement for the 1:1 ratio in "total utilitarianism". You end up with the "repugnant conclusion" to the Parfit's "mere addition" argument as long as this ratio is finite (known as "birth-death asymmetry"). For example, one may argue that killing 1 person to save 5 equally happy people is wrong, because killing is wrong, but as long as there is a ratio they would agree with (or, more generally, an equivalent number of saved people for each number of killed people), the repugnant conclusion argument still goes through.

You know, I've felt that examining the dust speck vs torture dilemma or stuff like that, finding a way to derive an intuitively false conclusion from intuitively true premises, and thereby concluding that the conclusion must be true after all (rather than there's some kind of flaw in the proof you can't see yet) is analogous to seeing a proof that 0 equals 1 or that a hamburger is better than eternal happiness or that no feather is dark, not seeing the mistake in the proof straight away, and thereby concluding that the conclusion must be true. Does anyone else feel the same?

Another problem with the repugnant conclusion is economic: it assumes that the cost of creating and maintaining additional barely-worth-living people is negligibly small.

The problem here seems to be about the theories not taking all things we value into account. It's therefore less certain whether their functions actually match our morals. If you calculate utility using only some of your utility values, you're not going to get the correct result. If you're trying to sum the set {1,2,3,4} but you only use 1, 2 and 4 in the calculation, you're going to get the wrong answer. Outside of special cases like "multiply each item by zero" it doesn't matter whether you add, subtract or divide, the answer will still be wrong. For example the calculations given for total utilitarianism fail to include values for continuity of experience.

This isn't to say that ethics are easy, but we're going to have a devil of a time testing them with impoverished input.

It's a good and thoughtful post.

Going through the iteration, there will come a point when the human world is going to lose its last anime, its last opera, its last copy of the Lord of the Rings, its last mathematics, its last online discussion board, its last football game - anything that might cause more-than-appropriate enjoyment. At that stage, would you be entirely sure that the loss was worthwhile, in exchange of a weakly defined "more equal" society?

I wonder if it makes sense to model a separate variable in the global utility function for "culture." In other words, I think the value I place on a hypothetical society runs something like)%20+%20log(U_c))., where x is each individual person's individual utility, and c is the overall cultural level.

A society where a million people each enjoy reading the Lord of the Rings but there are no other books would have high sigma[U(x)] and low U(c); a society where a hundred people each enjoy reading a unique book would have low total U(x) but high U(c).

That would help model the intuition that culture, even in the abstract, is worth trading off against individual happiness. I think I would prefer a Universe in which the Lord of the Rings were encoded into a durable piece of stone but otherwise had nothing else in it to a Universe in which there was a thriving colony of a few hundred cells of plankton but otherwise nothing else in it, even if there were nobody around to read the stone. Many economists would call that irrational -- but like the OP, I reject the premise that my individual utility function for the state of the world has to break down into other people's individual welfare.

For instance, it seems that there is only a small difference between the happiness of richer nations and poorer nations

What is happiness? If happiness is the "utility" that people maximise (is it?), and the richer are only slightly happier than the poorer (cite?), why is it that when people have the opportunity to vote with their feet, people in poor nations flock to richer nations whenever they can, and do not want to return?

In total utilitarianism, it is a morally neutral act to kill someone (in a painless and unexpected manner) and creating/giving birth to another being of comparable happiness. In fact if one can kill a billion people to create a billion and one, one is morally compelled to do so.

I dare to say that no self-professed "total utilitarian" actually aliefs this.

A more general problem with utilitarianisms including those that evade the critique in that article:

Suppose we have a computer running a brain sim (along with VR environment). The brain sim works as following: given current state, next state is calculated (using multiple cpus in parallel); the current state is read only, the next state is write only. Think arrays of synaptic values. After all of the next state is calculated, the arrays are switched and the old state data is written over . This is a reductionist model of 'living' that is rather easy to think about. Suppose that this being is reasonably happy.

We really want to sacrifice the old state for sake of the new state. If we are to do so based on maximizing utility (rather than seeing the update as a virtue in it's own right), the utility of new state data has to be greater than utility of the current state data. The utility has to keep rising with each simulator step. That's clearly not what anyone expects the utility to do. And it clearly has a lot of problems; e.g. when you have multiple brain sims, face a risk of hardware failure, and may want to erase some sim to use freed up memory as backup for some much older sim (whose utility grew over time to a larger value).

I'm very unconvinced that there even exist any 'utilitarian' solution here. If you want to maximize some metric over experience-moments that ever happen, then you need to keep track of the experience moments that already happened, to avoid re-doing it (you don't want to be looping sims over some happy moment). And it is still entirely immoral because you are going to want to destroy everything and create utilitronium.

In total utilitarianism, it is a morally neutral act to kill someone (in a painless and unexpected manner) and creating/giving birth to another being of comparable happiness. In fact if one can kill a billion people to create a billion and one, one is morally compelled to do so. And this is true for real people, not just thought experiment people - living people with dreams, aspirations, grudges and annoying or endearing quirks.

Keep in mind that the people being brought into existence will be equally real people, with dreams, aspirations, grudges, and annoying or endearing quirks. If the people being killed had any more of what you value overall, then it wouldn't be a utility neutral act.

Imagine that a billion people are annihilated from existence, and replaced with exact copies who're indistinguishable in any way. Don't judge a person's plan to execute this, that could entail some sort of mistake, suppose that this simply happens, so we must judge it purely by its results. Do you think that this would be a bad thing?

If not, then presumably it's not the destruction and replacement you're objecting to in and of itself, you're implicitly assuming a higher utility value for the people who're destroyed than those who're created, or some chance of an outcome other than perfect replacement of all the people with equal utility people.

An argument that I have met occasionally is that while other ethical theories such as average utilitarianism, birth-death asymmetry, path dependence, preferences of non-loss of culture, etc... may have some validity, total utilitarianism wins as the population increases because the others don't scale in the same way. By the time we reach the trillion trillion trillion mark, total utilitarianism will completely dominate, even if we gave it little weight at the beginning.

I'll admit I haven't encountered this argument before, but to me it looks like a type error. As you note, average utilitarianism counts something quite different than total utilitarianism; observers might (correctly) note that the latter can spit out much larger numbers than the former under some circumstances, but those values are unrelated abstractions, not something commensurate with each other's or those of other ethical theories absent a quantifying theory of metaethics that we don't have. It's like dividing seven by cucumber. I'd argue that the normalization process you suggest doesn't make much sense either, though; many utilitarianisms don't have well-defined upper bounds (why stop at a quadrillion?), and some don't have well-defined lower (a life not worth living might be counted as a negative contribution).

Insofar as ethical theories are models of our ethical intuitions, I can see an argument for normalizing against people's subjective satisfaction with a world-state, which is almost certainly a finite range and therefore implies some kind of diminishing returns or dynamic rather than static evaluation of state changes. But I can see arguments against this, too; in particular, it doesn't make any sense if you're trying to make a universalizable theory of ethics (which has its own problems, but it has been tried). The hedonic treadmill also raises issues.

I like that article. I wrote some on other problem with utilitarianism.

Also, by the way, regarding the use of name of Bayes, you really should thoroughly understand this paper and also get some practice solving belief propagation approximately on not so small networks full of loops and cycles (or any roughly isomorphic problem), to form opinion on self described Bayesianists.

And the sum itself is a huge problem. There is no natural scale on which to compare utility functions. Divide one utility function by a billion, multiply the other by eπ, and they are still perfectly valid utility functions. In a study group at the FHI, we've been looking at various ways of combining utility functions - equivalently, of doing interpersonal utility comparisons (IUC). Turns out it's very hard, there seems no natural way of doing this, and a lot has also been written about this, concluding little. Unless your theory comes with a particular IUC method, the only way of summing these utilities is to do an essentially arbitrary choice for each individual before summing. Thus standard total utilitarianism is an arbitrary sum of ill defined, non-natural objects.

This interests me. Do you have any literature I should read on this topic?

This one left me wondering - is "population ethics" any different from "politics"?

In total utilitarianism, it is a morally neutral act to kill someone (in a painless and unexpected manner) and creating/giving birth to another being of comparable happiness (or preference satisfaction or welfare).

Other members of society typically fail to approve of murder, and would apply sanctions to the utilitarian - probably hindering them in their pursuit of total utility. So, in practice, a human being pursuing total utilitarianism would simply not act in this way.

Good article! Here are a few related questions:

1. The problem of comparing different people's utility functions applies to average utilitarianism as well, doesn't it? For instance if your utility function is U and my utility function is V, then the average could be (U + V)/2 : however utility functions can be rescaled by any linear function, so let's make mine 1000000 x V. Now the average is U/2 + 500000 x V, which seems totally fair doesn't it? Is the right solution here to assume that each person's utility has a "best possible" case, and a "worst possible" case, and to rescale, assigning 1 to each person's best case, and 0 to their worst? That works fine if people have bounded utility, which we apparently do (it's one reason we don't fall for Pascal muggings).

2. It's true that no-one optimises utility perfectly, but even animals, plants and bacteria have an identifiable utility function (inclusive fitness), which they optimise pretty well. Why shouldn't people? And, to first approximation, why wouldn't a human's utility function also be inclusive fitness? (We can add other approximations as necessary, e.g. some sort of fitness function for culture or memes.)

3. Do you think utility functions should be defined over "worlds" or "states"? Decision theory only requires worlds, but consequentialism seems to require states. For instance if each world w consists of a sequence of states indexed by time t, then a consequentialist utility function applied to a whole world would look like U(w) = Sum d(t) x u(s(t)) where d(t) is the discount factor, and u is the utility function applied to states. Deontologists would have a completely different sort of U, but they are not immediately irrational because of that. (Seems they can still be consistent with formal decision theory.)

4. Looking at your paper on Anthropic Decision Theory, what do you think will happen if we adopt a compromise utility function somewhere between average and total utility, much as you suggest? Is the result more like SIA or SSA? Does it contain some of the strengths of each while avoiding their weaknesses? (It strikes me that the result is more like SSA, since you are avoiding the "large" utilities from total utility dominating the calculation, but I haven't tried to do the math, and wondered if you already had...)

5. Do you have views on "rule" versus "act" utilitarianism? It seems to me that advanced decision theories like TDT, UDT or ADT are already invoking a form of rule utilitarianism, right? Further that rule utilitarianism is a better "model" for our moral judgements than act utilitarianism.

I think I agree with your conclusion. But this:

to increase utility we should simply kill off all the rich, and let the poor multiply to take their place (continually bumping off any of the poor that gets too rich).

requires you to assume that the US or "the rich" have no relevant chance of producing vastly happier people in the future. This seems stronger than denying the singularity as such. And it makes targeted killing feel much more attractive to this misanthrope.

Only a slightly relevant question which nevertheless I haven't yet seen addressed: If a utilitarian desires to maximise other people's utilities and the other people are utilitarians themselves, also deriving their utility from the utilities of others (the original utilitarian included), doesn't that make utilitarianism impossible to define? The consensus seems to be that one can't take one's own mental states for argument of one's own utility function. But utilitarians rarely object to plugging others' mental states into their utility functions, so the danger of circularity isn't avoided. Is there some clever solution to this?

Why then is it so popular? Well, one reason is that there are models that make use of something like total utilitarianism to great effect. Classical economic theory, for instance, models everyone as perfectly rational expected utility maximisers.

Surely that is not the reason. Firstly, utilitarianism is not that popular. My theory about why it has any adherents at all is that it is used for signalling purposes. One use of moral systems is to broadcast what a nice person you are. Utilitarianism is a super-unselfish moral system. So, those looking for a niceness superstimulus are attracted. I think this pretty neatly explains the 'utilitarianism' demographics.

A population of TDT agents with different mostly-selfish preferences should end up with actions that closely resemble total utilitarianism for a fixed population, but oppose the adding of people at the subsistence level followed by major redistribution. (Or so it seems to me. And don't ask me what UDT would do.)

I have no interest in defending utilitarianism, but I do have an interest in a total welfare (yes I think such a concept can make sense) of sentient beings. The repugnance of the Repugnant Conclusion, I suggest, is a figment of your lack of imagination. When you imagine a universe with trillions of people whose lives are marginally worth living, you probably imagine people whose lives are a uniform grey, just barely closer to light than darkness. In other words, agonizingly boring lives. But this is unnecessary and prejudicial. Instead, imagine people with ups and downs like ours, but with a closer balance of ups and downs. Imagine rich cultures, intense personal relationships, exciting mathematical discoveries, etc., etc. - but perhaps more repression, more romantic breakups, more dead end derivations.

Perhaps there are values that are nonlocal, in the sense of not belonging to any one person and not being the sum of values belonging each to one person. And the Repugnant world you're imagining may lack those values. But that's a problem with Utilitarianism, not with Totality. In other words I suggest that insofar as moral value depends on how things go for individuals (considering individuals other than those to whom you have special obligations), it depends on the total rather than the average, or the pre-existing persons' welfare.

Why think so? Because having children normally isn't wrong, but having children when you know the child will only suffer horribly for a year and then die, is. Normal parents know, however, that there is a very slight chance of that type of horrible result. What justifies childbearing in the normal case? The obvious answer is the high probability that the child will lead a good life. Therefore, adding more good lives is a good-making feature. This doesn't show that adding more good lives is comparable to making the same number of pre-existing equally good lives twice as good - but I think that is the answer most coherent with the meager truth about personal identity.

Other comments: The "real world analogues" of the killing-and-replacing-people thought experiments turn out to be just more thought experiments. Not that there need be anything wrong with that, but the weirdness of the thought-experimental situation should be considered. If the intent is simply to show that utilitarianism faces a burden of argument in the face of counterintuitive results, it succeeds in that easy goal.

Interpersonal comparisons of utility suffer from the same difficulties, in principle, as intrapersonal comparisons. They're just a lot more intense in the former case. This applies to both preference and hedonics, and also to many more sophisticated evaluative schemes which may include both, plus more.

In Austrian economics using the framework of Praxiology the claim is made that preferences (the rough equivalent of utilities) cannot be mapped to cardinal values but different states of the world are still well ordered by an individual's preferences such that one world state can be said to be more or less desirable than another world state. This makes it impossible to numerically compare the preferences of two individuals except through the pricing/exchange mechanism of economics. E.g. would 1 billion happy people exchange their own death for the existence of 1 billion and 1 new happy people? To answer the question simply ask them and what they would do or observe what they do in that situation and that will reveal their preferences.

Taking a preference-based approach, consider the set of all individuals and the set of all world-states. Each individual has a well-ordered list of preferences of possible world states, with the only restriction that it be bounded from above by a maximal preference. In every world state the next world state is chosen by all individuals voting for the most preferable next reachable world state. In a majority voting system each individual votes for its maximally-preferred world state. In runoff and approval voting the first, second, third, etc. choices are the highest, next-highest, etc. ranked preferences for world states, respectively. Ethics thus reduces to the problem of fair voting.

An obvious criticism of Austrian economics is that it simply describes the economy as the result of all individual actions (with which individuals reveal their true preferences, by definition) with no additional predictive power. I think that by contrasting the theoretical results of perfect ethical preference voting with the theoretical results of perfectly calculating a utilitarian theory, there may be some insight. The basic difference is that economics relies on pricing to build an economy but in ethics we can cheat and ask theoretical questions about all possible world states.

Potential or hypothetical individuals would have their own preferences for world-states but, as the article mentions, their preferences may not be compatible with the set of possible next world states that we are voting on If those hypothetical individuals never have enough votes for possible next world states then they will never have any influence. Individuals currently sleeping, anesthetized, or frozen in liquid nitrogen have hypothetical preferences for future world states that may very well coincide with our preferences for future world states, and therefore they have a greater chance of existing as acting individuals in our future world. Ultimately in any ethical theory it is only our estimation of a hypothetical being's preferences that we can consider, so their preferences are subsumed into our own preferences.

3^^^3 people will probably rank their preferences for world states with and without a single dust speck in their eye as nearly indistinguishable, but world states with torture are hopefully quite lower in rank than equivalent world states without torture. The one person whose torture depends on the vote may prefer world states with 3^^^3 dust specks far more than world states with 50 years of torture, but their vote clearly doesn't matter in any conceivable voting system. Nevertheless, so long as the existence of torture is more repugnant than a single dust speck, 3^^^3 people will vote to receive the dust speck instead of allow that individual to be tortured.

Populations of nearly any size will probably not vote to replace themselves with a different population (whether of humans, paperclips, or smily-faces).

There are still problems: Bacteria and parasites may deserve a vote. Weighting may fix that problem. Hated minorities are still at a disadvantage even in the fairest voting systems. On one hand if people are not personally inconvenienced by the actions of a hated minority they will probably prefer worlds where that minority is not tortured over worlds where they are tortured, simply because of their general aversion to torture. On the other hand a large number of voters in democratic countries have not kicked torturers out of political office. This is distressing because far fewer than 3^^^3 people have been affected by, say, Maher Arar. There is apparently a tendency in humans to have a preference for the brutal punishment of an assumed criminal even if there is only a tiny marginal chance of value to themselves. I think this is a failure of rationality and probably not a failure of any particular ethical system.

The primary difference between additive functions of individual utility and preference voting is that the most important effects on individuals have the largest influence on their preferences. There is no correspondence to having a maximal utility for not having a speck in one's eye and also a maximal utility for torturing another individual. One or the other will have strictly greater preference. In approval or runoff voting the preference for torturing another individual will fall behind a series of other more pleasant preferences unless that individual actually had a major effect on the voter. In effect everyone is forced to vote for what really matters to them instead of arbitrarily ruining another individual's life for no appreciable benefit. Utilitarianism could conceivably have exactly the same ranking of values as preference voting (just enumerate all the N preferences and assign them values of i/N from least to most preferred) but there is no guarantee that an individual faced with choosing utilities would assign the same relative value to world states as an individual choosing preferences.

It appears that Eliezer went down this road a ways in http://lesswrong.com/lw/rx/is_morality_preference/ and then went off in another direction before enumerating the idea of what would happen if everyone voted based on their preferences and then acted to achieve the winning world state instead of acting to achieve only their own maximally preferred world state.