Harsanyi's Social Aggregation Theorem and what it means for CEV

I suggest the VNM Expected Utility Theorem and this theorem should be used as a test on potential FAI researchers. Is their reaction to these theorems "of course, the FAI has to be designed that way" or "that's a cool piece of math, now let's see if we can't break it somehow"? Maybe you don't need everyone on the research team to instinctively have the latter reaction, but I think you definitely want to make sure at least some do. (I wonder what von Neumann's reaction was to his own theorem...)

I think you're overselling your case a little here. The cool thing about theorems is that their conclusions follow from their premises. If you then try to apply the theorem to the real world and someone dislikes the conclusion, the appropriate response isn't "well it's math, so you can't do that," it's "tell me which of my premises you dislike."

That's a good point. I agree, and I've edited my post to reflect that.

An additional issue here is premises which are not explicitly stated. For example, there's an implicit premise in your post of there being some fixed collection of agents with some fixed collection of preferences that you want to aggregate. Not pointing out this premise explicitly leaves your implied social policy potentially vulnerable to various attacks involving creating agents, destroying agents, or modifying agents, as I've pointed out in other comments.

I thought I was being explicit about that when I was writing it, but looking at my post again, I now see that I was not. I've edited it to try to clarify that.

Thanks for pointing those out.

Only a VNM-rational agent can have preferences in a coherent way, so if we're talking about aggregating people's preferences, I don't see any way to do it other than modeling people as having underlying VNM-rational preferences that fail to perfectly determine their decisions.

Presumably there would be first be an extrapolation phase resulting in rational preferences.

The situation analogous to Simpson's paradox can only occur if for some reason we care about some people's opinion more than others in some situations (this is analogous to the situation in Simpson's paradox where we have more data points in some parts of the table than others. It is a necessary condition for the paradox to occur.)

For example: Suppose Alice (female) values a cure for prostate cancer at 10 utils, and a cure for breast cancer at 15 utils. Bob (male) values a cure for prostate cancer at 100 utils, and a cure for breast cancer at 150 utils. Suppose that because prostate cancer largely affects men and breast cancer largely affects women we value Alice's opinion twice as much about breast cancer and Bob's opinion twice as much about prostate cancer. Then in the aggregate curing prostate cancer is 210 utils and curing breast cancer 180 utils, a preference reversal compared to either of Alice or Bob.

I don't see how it's like Simpson's paradox, actually. You want to go to Good Hospital instead of Bad Hospital even if more patients who go to Good Hospital die because they get almost the hard cases. Aggregating only hides the information needed to make a properly informed choice. Here, aggregating doesn't hide any information.

But there are a bunch of other ways things like that can happen.

This very morning I did a nonlinear curvefit on a bunch of repeats of an experiment. One of the parameters that came out had values in the range -1 to +1. I combined the data sets directly and that parameter for the combined set came out around 5.

In a way, this analogy may be even more directly applicable than Simpson's paradox. Even if A and B are complete specifications (unlike that parameter, which was one of several), the interpersonal reactions to other people can do some very nonlinear things to interpretations of A and B.

That a bad result can happen in a given strategy is not a conclusive argument against preferring that strategy.

It's possible that the AI would just happen never to confront a situation where it would choose differently than everyone else would, but not reliably. If you had an AI that violated axiom 2, it would be tempting to modify it to include the special case "If X is the best option in expectation for every morally relevant agent, then do X." It seems hard to argue that such a modification would not be an improvement. And yet only throwing in that special case would make it no longer VNM-rational. Worse than a VNM-irrational agent is pretty bad.

Why is this new failure mode supposed to be decisive in the choice between the two alternatives?

Because maximizing a weighted sum of utility functions does not have any comparably convincing failure modes. None that I've heard of anyway, and I'd be pretty shocked if you came up with a failure mode that did compete.

Subjective uncertainty doesn't seem particularly relevant to Friendly AI, since the FAI could come up with a more accurate probability estimate than everyone else, and axiom 2 could refer to what everyone would want if they knew the probabilities as well as the FAI did. Do you have any examples of undesirable effects of the Pareto property that do not involve subjective uncertainty, or do you think subjective uncertainty is more important than I think it is?

What if we also add a requirement that the FAI doesn't make anyone worse off in expected utility compared to no FAI?

Sounds obviously unreasonable to me. E.g. a situation where a person derives a large part of their utility from having kidnapped and enslaved somebody else: the kidnapper would be made worse off if their slave was freed, but the slave wouldn't become worse off if their slavery merely continued, so...

What if we also add a requirement that the FAI doesn't make anyone worse off in expected utility compared to no FAI?

I don't think that seems reasonable at all, especially when some agents want to engage in massively negative-sum games with others (like those you describe), or have massively discrete utility functions that prevent them from compromising with others (like those you describe). I'm okay with some agents being worse off with the FAI, if that's the kind of agents they are.

Luckily, I think people, given time to reflect and grown and learn, are not like that, which is probably what made the idea seem reasonable to you.

I expect that with actual people, in practice, the FAI would leave no one worse off. But I wouldn't want to hardwire that into the FAI because then its behavior would be too status quo-dependent.

What do you think about Eliezer's proposed solution of making the FAI's utility function depend on a coinflip outcome?

Can you be more specific about what you mean by a parliamentary model? (If I had to guess, though, axiom 1.)

An advantage of parliamentary models is that you don't have to know the utility functions of the individual agents, but can just use them as black boxes that output decisions. This is useful for handling moral uncertainty when you don't know how to encode all the ethical theories you're uncertain about as utility functions over the same ontology.

Do the choices of a parliament reduce to maximizing a weighted sum of utilities?

Let's say the parliament makes a Pareto optimal choice, in which case that choice is also made by maximizing some weighted sum of utilities (putting aside the coin flip issue). But the parliament doesn't reduce to maximizing that weighted sum of utilities, because the computation being done is likely very different. Saying that every method of making Pareto optimal choices reduces to maximizing a weighted sum of utilities would be like saying that every computation that outputs an integer greater than 1 reduces to multiplying a set of prime numbers.

The Social Aggregation Theorem doesn't just show that some particular way of aggregating utility functions other than by linear combination is bad; it shows that every way of aggregating utility functions other than by linear combination is bad.

Thanks! I wish the math hadn't broken down, it makes the post harder to read...

See my reply to 615C68A6.

There is no relation to Simpson's paradox. In Simpson's paradox, each of the data points comes from the same one-dimensional x-axis, so as you keep increasing x, you can run through all the data points in one group, go out the other side, and then get to another group of data points. In preference aggregation, there is no analogous meaningful way to run through one agent considering each possible state of the universe, keep going, and get to another agent considering each possible state of the universe.

Can you clarify which people you want to apply this theorem to?

I'm not entirely sure. My default answer to that is "all people alive at the time that the singularity occurs", although you pointed out a possible drawback to that (it incentivizes people to create more people with values similar to their own) in our previous discussion. This is really an instrumental question: What set of people should I suggest get to have their utility functions aggregated into the CEV so as to best maximize my utility? One possible answer is to aggregate the utilities of everyone who worked on or supported the FAI project, but I suspect that due to the influence of far thinking, that would actually be a terrible way to motivate people to work on FAI, and it should actually be much broader than that.

So it seems the FAI should be aggregating the preferences of a fixed set of people for all time. But this also seems problematic.

I don't think it would be terribly problematic. "People in the future should get exactly what we currently would want them to get if we were perfectly wise and knew their values and circumstances" seems like a pretty good rule. It is, after all, what we want.

They could just insist on a normalization scheme that is blatantly biased in favor of their utility function. In a theoretical sense, this doesn't cause a problem, since there is no objective way to define an unbiased normalization anyway. (of course, if everyone insisted on biasing the normalization in their favor, there would be a problem)

Linearly combining utility functions does not force you to reward utility monsters. It just forces you to either be willing to sacrifice large amounts of others' utility for extremely large amounts of utility monster utility, or be unwilling to sacrifice small amounts of others' utility for somewhat large amounts of utility monster utility in the same ratio. The normalization scheme could require the range of all normalized utility functions to fit within certain bounds.

Under Harsanyi's original axioms, you cannot say anything about the signs of the coefficients. My axioms are slightly stronger, but I think still not quite enough. However, if you make the even stronger (but still reasonable, I think) assumption that the agents' utility functions are linearly independent, then you can prove that all of the coefficients are non-negative. This is because the linear independence allows you create situations where each agent prefers A to B by arbitrarily specifiable relative amounts. As in, for all agents k, we can create choices A and B such that every agent prefers A to B, but the margin by which every agent other than k prefers A to B is arbitrarily small compared to the margin by which k prefers A to B, so since FAI prefers A to B, c_k must be nonnegative.

I'm pretty sure it's possible to reach the same conclusion by removing the requirement that the aggregation be VNM-rational and strengthening axiom 2 to say that the aggregation must be Pareto-optimal with respect to all prior probability distributions over choices the aggregation might face. That is, "given any prior probability distribution over pairs of gambles the aggregation might have to choose between, there is no other possible aggregation that would be better for every agent in the population in expectation." It's possible we could even reach the same conclusion just by using some such prior distribution with certain properties, instead of all such distributions.

Harsanyi's axioms seem self-evidently desirable on their own. I didn't claim that they were a consequence of the Veil of Ignorance.

You were looking at Harsanyi's explanation of a previous, similar theorem by Fleming, in section II of his paper. He proves the theorem I explained in the post in section III.

My axiom 2 was meant to include decisions involving uncertainty, like Harsanyi's postulates but unlike Fleming's postulates. Sorry if I did not make that clear.