Polymath-style attack on the Parliamentary Model for moral uncertainty

Great example. As an alternative to your three options (or maybe this falls under your first bullet), maybe negotiation should happen behind a veil of ignorance about what decisions will actually need to be made; the delegates would arrive at a decision function for all possible decisions.

Your example does make me nervous, though, on the behalf of delegates who don't have much to negotiate with. Maybe (as badger says) cardinal information does need to come into it.

This seems really similar to the problem Knightian uncertainty attempts to fix.

I think So8res's solution is essentially your option 3, with the strength of the disagreements being taken into account in the utility function, and then once you really have everything you care about accounted for, then the best choice is the standard one.

I agree that some cardinal information needs to enter in the model to generate compromise. The question is whether we can map all theories onto the same utility scale or whether each agent gets their own scale. If we put everything on the same scale, it looks like we're doing meta-utilitarianism. If each agent gets their own scale, compromise still makes sense without meta-value judgments.

Two outcomes is too degenerate if agents get their own scales, so suppose A, B, and C were options, theory 1 has ordinal preferences B > C > A, and theory 2 has preferences A > C > B. Depending on how much of a compromise C is for each agent, the outcome could vary between

• choosing C (say if C is 99% as good as the ideal for each agent),
• a 50/50 lottery over A and B (if C is only 1% better than the worst for each), or
• some other lottery (for instance, 1 thinks C achieves 90% of B and 2 thinks C achieves 40% of A. Then, a lottery with weight 2/3rds on C and 1/3rd on A gives them each 60% of the gain between their best and worst)

A possible (but I admit, quite ugly) workaround: whenever there are very few decisions to be made introduce dummy bills that would not be actually carried out. MPs wouldn't know about their existence. In this case Theory 1 might be able to negotiate their way into getting B.

I think there's a fairly natural disagreement point here: the outcome with no trade, which is just a randomisation of the top options of the different theories, with probability according to the credence in that theory.

One possibility to progress is to analyse what happens here in the two-theory case, perhaps starting with some worked examples.

I think the the Nash bargaining solution should be pretty good if there are only two members of the parliament, but it's not clear how to scale up to a larger parliament.

This looks reasonable to analyse (although I'd be interested in analysing other forms too).

I'd be tempted to start with a simpler example to get complete analysis. Perhaps 2 bills and 2 MPs. If that's easy, move to 3 MPs.

It seems like votes should be considered simultaneously to avoid complex alliances of the form: I will vote on B4 in the direction you like if you vote on B3 in the direction I like, but this is only possible in one direction WRT time. Having such an ordering and resulting negotiations means that some agents have an incentive to bargain for moving the location of a bill. It seems better to be able to make all such Bx vote for By vote trades. I'm not familiar enough with voting models to know the tradeoffs for a simultaneous system though.

A very quick thought about one type of possible negotiating strategies. A delegate might choose a subset of bills, choose another delegate to approach and offer a usual cake cutting game for two players, when the first delegate divides that subset into two "piles" and allows the second delegate to choose one of them. Then they each would decide how to vote on the bills from their respective "piles" and promise to vote in accordance to each other's decisions.

However, it is not clear to me how these two choices (marked by asterisks) should work. Also, whether the second candidate should be allowed to reject the offer to play a cake cutting game.

edit: A potential flaw. Suppose we have a bill with two possible voting options A_1 and A_2 (e.g. "yes" and "no") with no possibility to introduce a new intermediate option. If a option A is supported by a small enough minority (0.75), this minority would never be able to achieve A (even though they wouldn't know that), and utility difference U_m (A_1) - U_m (A_2) for each m would not matter, only the sign of difference would.

Remember there's no such thing as zero utility. You can assign an arbitrarily bad value to failing to resolve, but it seems a bit arbitrary.

I think LW has the right kind of community for Polymath, and I think it's a good idea to give it a try.

It looks like this problem is assuming that Parliament uses plurality voting with more than 2 options. It seems like it shouldn't be a problem if all votes involve only 2 options (an up-or-down vote on a single bill). If we want the rules to allow votes between more than 2 options, it seems fixable by using a different voting system such as approval voting.

Given delegates with a certain type of amnesia (i.e. they should not remember to have voted on an issue before, although they might have to remember some binding agreements (I am not sure about that)), we could replace a plurality vote with an elimination runoff, where at each step of elimination delegates think that this is the only vote on that issue (which is thought to be affected by randomization) and they are not allowed to introduce new options.

Well, this system might have its own disadvantages, possibly similar to these (however, at each step negotiations are allowed), although delegates wouldn't know how to actually game it.

This seems to be almost equivalent to irreversibly forming a majority voting bloc. The only difference is how they interact with the (fake) randomization: by creating a subagent, it effectively (perfectly) correlates all the future random outputs. (In general, I think this will change the outcomes unless agents' (cardinal) preferences about different decisions are independent).

The randomization trick still potentially helps here: it would be in each representative's interest to agree not to vote for such proposals, prior to knowing which such proposals will come up and in which order they're voted on. However, depending on what fraction of its potential value an agent expects to be able to achieve through negotiations, I think that some agents would not sign such an agreement if they know they will have the chance to try to lock their opponents out before they might get locked out.

Actually, there seems to be a more general issue with ordering and incompatible combinations of choices - splitting that into a different comment.

Well, the entire idea of the parliamentary approach is predicated on the idea that the parliamentarians have some actions that they consider "more bad" than other actions.

How's this for a formalization: Our parliament faces a series of decisions d[i]. For any given decision the parliament faces, there are a series of choices d[i][j] that could be made regarding it. (d[i][0], d[i][1], etc.)

Over any given session of a parliament, the parliament faces every decision d[i] and for each decision it faces, makes a choice d[i][j] regarding how to address it. A structure containing all the decisions the parliament faces and a choice for each is a "decision record". A parliamentarians' preferences are specified by an ordering of decision records from most preferred to least preferred. The total number of possible decision records is equal to the product of the numbers of choices for each individual decision.

Can't we use a hierarchy of ordinal numbers and a different ordinal sum (e.g. maybe something of Conway's) in our utility calculations?

That is, lying would be infinitely bad, but lying ten times would be infinitely worse.

To avoid the timelessness issue, the parliament could be envisioned as voting on complete courses of action over the foreseeable future, rather than separate votes taken on each action. Then the deontologists' utility function could return 0 for all unacceptable courses of action and 1 for all acceptable courses of action.

Maybe deontological theory can be formalized as parliamentary fraction that have the only one right option for each decision and always vote for this option and can't be bargained to change its vote. This formalization have an unfortunate consequence: if some deontological theory have more then the 50% credence, agent will always act on it. But if no deontological theory have more then the 50% fraction, this formalization can be reasonable.

The threats problem seems like a specific case of problems that might arise by putting real intelligence in to the agents in the system. Especially if this moral theory was being run on a superintelligent AI, it seems like the agents might be able to come up with all sorts of creative unexpected stuff. And I'm doubtful that creative unexpected stuff would make the parliament's decisions more isomorphic to the "right answer".

One way to solve this problem might be to drop any notion of "intelligence" in the delegates and instead specific a deterministic algorithm that any individual delegate follows in deciding which "deals" they accept. Or take the same idea even further and specify a deterministic algorithm for resolving moral uncertainty that is merely inspired by the function of parliaments, in the same sense that the stable marriage problem and algorithms for solving it could have been inspired by the way people decide who to marry.

Eliezer's notion of a "right answer" sounds appealing, but I'm a little skeptical. In computer science, it's possible to prove that a particular algorithm, when run, will always achieve the maximal "score" on a criterion it's attempting to optimize. But in this case, if we could formalize a score we wanted to optimize for, that would be equivalent to solving the problem! That's not to say this is a bad angle of approach, however... it may be useful to take the idea of a parliament and use it to formalize a scoring system that captures our intuitions about how different moral theories trade off and then maximize this score using whatever method seems to work best. For example waves hands perhaps we could score the total regret of our parliamentarians and minimize that.

Another approach might be to formalize a set of criteria that a good solution to the problem of moral uncertainty should achieve and then set out to design an algorithm that achieves all of these criteria. In other words, making a formal problem description that's more like that of the stable marriage problem and less like that of the assignment problem.

So one plan of attack on the moral uncertainty problem might be:

• Generate a bunch of "problem descriptions" for moral uncertainty that specify a set of criteria to satisfy/optimize.

• Figure out which "problem description" best fits our intuitions about how moral uncertainty should be solved.

• Find an algorithm that provably solves the problem as specified in its description.

The idea of explicit vote-selling is probably the easiest way to have 'enforceable contracts' without things getting particularly sticky. (If you have ordered votes and no enforceable contracts, then vote order becomes super important and trading basically breaks apart. But if you have ordered votes and vote sales, then trading is still possible because the votes can't switch.)

But I don't think the prices are going to be that interesting- if the vote's on the edge, then all votes are valuable, but as soon as one vote changes hand the immediate price of all votes drops back to 0. Calculating the value of, say, amassing enough votes to deter any trading on that vote seems like it might add a lot of murkiness without much increased efficiency.

There's a family of approaches here, but it's not clear that they recreate the same behaviour as the parliament (at least without more arguments about the parliament). Whether they are more or less desirable is a separate question.

Incidentally, the version that you suggest isn't quite well-defined, since it can be changed by adding a constant to the theory of a function. But that can easily be patched over.

I've argued that normalising the variance of the functions is the most natural of these approaches (link to a paper giving the arguments in a social choice context; forthcoming paper with Ord and MacAskill in the moral uncertainty context).

I think the key benefit of the parliamentary model is that the members will vote trade in order to maximize their expectation.

So it's worthwhile to pick which bullet to bite first and design with that in mind as a limitation rather than just getting started and later on realize you're boxed into a corner on this point.

The easiest bullet to bite is the "ordinal preferences" bullet. If you allow the group to be indifferent between options, then the impossibility disappears. (You may end up with a group that uses a sensible voting rule that is indifferent between all options, but that's because the group is balanced in its opposition.)

If every voter's preference between X and Y remains unchanged, then the group's preference between X and Y will also remain unchanged (even if voters' preferences between other pairs like X and Z, Y and Z, or Z and W change).

This is the condition I want to give up on. I'm not even convinced that it's desirable.

The first point that seems relatively obvious to me is that all rational agents will intentionally mis-state their utility functions as extremes for bargaining purposes.

Because we're working in an idealised hypothetical, we could decree that they can't do this (they must all wear their true utility functions on their sleeves). I don't see a disadvantage to demanding this.

If what you say is true about all trades being 1-for-1, that seems more like a bug than a feature; if an agent doesn't have any votes valuable enough to sway others, it seems like I'd want them to be able (i.e. properly incentivized) to offer more votes, so that the system overall can reflect the aggregate's values more sensitively. I don't have a formal criterion that says why this would be better, but maybe that points towards one.