Another question about utilitarianism and selfishness

I think the standard patch is just to say that you should only take into account people's "selfish" utilities when you aggregate.

For the following analysis, I am assuming bounded utilities. I will normalize all utilities to between 0 and 1.

What you are observing is not a bug. If your preferences are 100% preference utilitarianism, then there is no reason to think that it would pull in any direction other than what maximizes preferences of everyone else. If you have any selfish goals, that is not purely utilitarianism, but that is okay!

If we are not 100% utilitarian, then there is no problem. Let's say that my preferences are 90% utilitarian, and 10% maximizing my own happiness. This fixes the problem, because there is 10% of my utility function that is unaffected by my utilitarianism. In fact, my utilitarian side includes a term for my own happiness, so my happiness actually counts for something like 10.00000001%, depending on the population. This all works fine, as long as everyone has at least a little bit of selfish preferences.

Imagine if everyone had utility functions that were at least 1% terminal goals that do not reference other people. Then in calculating my utility in a given world state, I will have my utility function pointing to someone else's, which might point back at mine. However, with each level of recursion, 1% of the remaining undefined part of the function will become actually defined.

The only time we run into a problem is in situations like where my utility function is defined to equal yours and yours is defined to equal mine. As long as we avoid this 100% recursion, we are fine.

There is not even a problem if we have negative utility coming form other peoples utility. For example, if my utility was 50% my happiness and 50% your utility, and yours was 50% your happiness and 50% one minus my utility, we are still fine. If my utility is X and my happiness is x, your utility is Y and your happyness is y, then we get X=(x+Y)/2=(2x+y-X)/4, which simplifies to X=(2x+y)/3.

Another problem I like: in our world people have very real preferences over how one should go about analyzing moral problems (including preferences over how to engage in meta-ethics). You'll find that most people are very anti-utilitarian. A true preference utilitarian will self-modify into an agent that thinks about morality in the way most preferred by the population, i.e., vaguely moral realist virtue ethics. This is less of a problem with extrapolated-preference utilitarianism (but then, how do you utilitarian-justifiably determine how to extrapolate preferences, except by looking at existing preferences about extrapolation-like processes?), and barely a problem at all for non-preference utilitarianism, as far as I can see.

Also note that the epistemic problem of preference elicitation is very real here, and in fact I do not see why a preference utilitarian wouldn't be obliged to engage in preference elicitation in an itself-utilitarianly-justified manner, which won't seem as epistemically sound as what a utilitarian would naively reckon to be a good faith preference elicitation algorithm. In general I think preference utilitarianism runs into many of the same problems as epistemic majoritarianism. A term that covers both cases might be "decision-policy majoritarianism".

... my moral theory tells me to construct a new utility function using some sort of aggregating procedure that takes as input the current utility functions of all moral agents (including my own).

Right.

But once I've done this, my own utility function has changed

Not necessarily right. And fortunately not: "change your utility function" is typically contra-utility for your existing utility function, and it would be hard to convince others to behave morally if your thesis always entailed "you should do things that will make the world worse by your own current preferences".

Utilitarian preferences with aggregated utility functions can result from negotiation, not just from remodeling your brain. In situations working with this model, your utility function doesn't change, your partners' utility functions don't change, but you all find that each of those utility functions will end up better-satisfied if you all try to optimize some weighted combination of them, because the cost of identifying and punishing defectors is still less than the cost of allowing defectors. Presumably you and your partners already assign some terminal value to others, but the negotiation process doesn't have to increase that terminal value, just add instrumental value.

This kind of problem goes back to Bentham and the very beginning of utilitarianism. (Disclaimer: what follows is based on my recollections of an essay by A. J. Ayer on Bentham, read long ago. I cannot find it now online, nor find other discussions of Bentham making the same points, and I am not any kind of expert on the matter. So when I say below "Bentham believed…", this could be true, or could be true only in Ayer's interpretation, or even only in my own interpretation of Ayer's interpretation.)

Bentham believed both that each person pursues only their own happiness, and that the good consists of the creates happiness of the greatest number. (The first if these could correspond in your formulation of the dilemma to saying that agents have utility functions and behave rationally according to them, and the second to a statement of preference utilitarianism.) Then the problem comes up of how are we be utilitarians and try to maximize global happiness if we are psychologically necessitated to care only about our own. Bentham's solution is to postulate a "lawgiver", a person whose happiness is greatest when everyone's happiness is maximized, and say that utilitarianism as a political prescription says that laws should be made by this lawgiver. (In LW-language this could correspond to FAI!)

Translating back Bentham's solution (or my memory of Ayer's interpretation of it) back to your question, I think the answer would be that utility functions don't change; if you are truly a preference utilitarian, then your utility function is already given by the ultimate fixed point of the iteration process you have outlined, so there is no dynamical changing.

I'd notice a similarity with convergent series. In particular, geometric series. Just because you approach someone an infinite number of times doesn't mean you get all that close to them. (ETA: 1/2^n approaches -1 an infinite number of times but never gets closer than 1 away from it)

Also, even if you assign yourself equal weight as anyone else, you are usually more able to affect yourself than anyone else and a lot of the others' demands cancel out. So your tiny chunk of utility, though insignificant in respect to your overall utility function, produces an very significant derivative in respect to your possible actions.

Regarding preference utilitarianism, why can't the negative utility of not having a preference fulfilled be modelled with average or total utilitarianism? That is, aren't there some actions that create so much utility that they could overcome the negative utility of one's preference not being honored? I don't see why preference fulfillment should be first class next to pleasure and pain.

Sorry if this is off-topic, that was just my first reaction to reading this.