Cooperating with agents with different ideas of fairness, while resisting exploitation

This translates into an informal principle of negotiations: Be willing to accept unfair bargains, but only if (you make it clear) both sides are doing worse than what you consider to be a fair bargain.

That's clever and opens up a lot of extra scope for 'imperfect' cooperation, without any exploitation problem. I notice that this matches my 'fairness' instincts and some of my practice while playing strategy games. Unfortunately I don't recall reading the principle formally specified anywhere.

Solution concept implementing this approach (as I understand it):

Player X chooses Pareto fair outcome (X→X, X→Y), (X→Y can be read as "player X's fair utility assignment to player Y"), player Y chooses fair outcome (Y→X, Y→Y).

The actual outcome is (Y→X, X→Y)

(If you have a visual imagination in maths, as I do, you can see this graphically as the Pareto maximum among all the points Pareto worse than both fair outcomes).

This should be unexploitable in some senses, as you're not determining your own outcome, but only that of the other player.

Since it's not Pareto, it's still possible to negotiate over possible improvements ("if I change my idea of fairness towards the middle, will you do it too?") and blackmail is possible in that negotiation process. Interesting idea, though.

I am curious how this idea would generalize to more than two players. Should you allow negotiations that allow some players to do better than fair at the expense of other players?

In general, you can not compare the utilities for two different agents, since a linear transformation doesn't change the agent's behavior. So (12, 12) is really (12a+a₀, 12b+b₀). How would you even count the utility for another agent without doing it in their terms?

We don't have this problem in practice, because we are all humans, and have similar enough utility functions. So I can estimate your utility as "my utility if I were in your shoes". A second factor is perhaps that we often use dollars as a stand-in for utilons, and dollars really can be exchanged between agents. Though a dollar for me might still have a higher impact than a dollar for you.

This is analogous to zero determinant strategies in the iterated prisoner's dilemma, posted on LW last year. In the IPD, there are certain ranges of payoffs for which one player can enforce a linear relationship between his payoff and that of his opponent. That relationship may be extortionate, i.e. such that the second player gains most by always cooperating, but less than her opponent.

It seems the logical extension of your finitely many step-downs in "fairness" would be to define a function f(your_utility) which returns the greatest utility you will accept the other agent receiving for that utility you receive. The domain of this function should run from wherever your magical fairness point is down to the Nash equilibrium. As long as it is monotonically increasing, that should ensure unexploitability for the same reasons your finite version does. The offer both agents should make is at the greatest intersection point of these functions, with one of them inverted to put them on the same axes. (This intersection is guaranteed to exist in the only interesting case, where the agents do not accept as fair enough each other's magical fairness point)

Curiously, if both agents use this strategy, then both agents seem to be incentivized to have their function have as much "skew" (as EY defined it in clarification 2) as possible, as both functions are monotonically increasing so decreasing your opponents share can only decrease your own. Asymptotically and choosing these functions optimally, this means that both agents will end up getting what the other agent thinks is fair, minus a vanishingly small factor!

Let me know if my reasoning above is transparent. If not, I can clarify, but I'll avoid expending the extra effort revising further if what I already have is clear enough.

If they use their knowledge of their code to predict you refusing to accept that bargain, they will defect on every round for the mutual payoff of (8, 8).

Emphasis mine. Should the second their be a your?

Informal theories of negotiation?

This and this is a guide to negotiation hereustics.

This is dark side negotiation.

This is an overview of foundationalism, the theoretical basis for your ordinal utility for argumentation.

I don't know the answer to the specific question you're asking.

However, I think you might find Keith Hipel's work on using graph theory to model conflicts and negotiations interesting. A negotiator or mediator using Hipel's model can identify places where both parties in a negotiation could have an improved outcome compared to the status quo, based on their ranked preferences.

I believe that all friendly decision making agents should view utility for your opponent as utility for you too in order for the agent to actually be friendly.

A friendly agent, in my opinion, should be willing to accept doing poorly in the prisoner's dilemma if it allows your opponent to do better.

I also believe that an effective decision making agent should have an inclination to avoid waste.

This does not exclude understanding exploitability, fairness, negotiation techniques and attempts at penalty induced behavior modification of other agents towards solutions on the pareto boundary.

These points do not (directly) contribute to resolving issues of truly selfish cooperative agents and as such are missing the point of the post.

Humans are known to have culturally dependent fairness ideas. There are a lot of studies which tested these repeatedly with the ultimatum game:

http://en.wikipedia.org/wiki/Ultimatum_game#Experimental_results

A meta study basically confirms this here:

http://www.econ.nagoya-cu.ac.jp/~yhamagu/ultimatum.pdf

One interesting strategy that does not achieve the Pareto boundary:

Defect with a higher probability if the opponent gives you a worse deal. This way, you at least have some probability of cooperation if both agents have ideas of fairness skewed away from each other, but you limit (and can completely remove) the incentive to be unfair.

For example, if you think (12, 12) is fair, and they think (11, 13) is fair, then you can offer to accept their (11, 13) with %80 probability. Their expected utility is 0.8x13 + 0.2x8 = 12. This is the same for them as if they agree with you, so there's no incentive for them to skew their idea of fairness. The expected payoff ends up being (10.4, 12). It's not as good as (12, 12) or (11, 13), but at least it's better than (8, 8).

Furthermore, if they also use this strategy, you will end up deciding on something somewhere between (12, 12) and (11, 13) with a higher probability. I think the expected payoff matrix will end up being (11, 12).

Edit:

I came up with a modification to put it in the Pareto boundary.

Introduce a third agent. Let's call the agents Alice, Bob, and Charlie.

If Alice and Bob disagree on what's fair, Bob gets what Alice thinks is fair for him to have, Alice gets what she thinks it's fair for Bob to have, and Charlie gets as much as possible while Alice and Bob get that much. Similarly for when Bob and Charlie or Charlie and Alice disagree. Since joining together like this means that they'll get value that would otherwise be wasted if it was just the other two, there's incentive to join.

If it's possible, but difficult, for one to bribe another without being detected by the third, this can be fixed by making it so they get just enough less to make up for it.

If it's not difficult, you could increase the number of agents so that bribery would be unfeasible.

If there's ever a deal that Alice, Bob, and Charlie are involved in, then you'd have to introduce someone else to get it to work. Ultimately, the idea fails if everyone has to make a deal together.

This is the generalized problem of combating intelligence; even with my source code, you might not be able to perform the analysis quickly enough. I can leverage your slow processing time by creating an offer that diminishes with forward time. The more time you take the think, the worse off I'll make you, making it immediately beneficial to you under Bayesian measurement to accept the offer unless you can perform a useful heuristic to determine I'm bluffing. The end result of all processing is the obvious that is also borne out in humanity's history: The more well informed agent will win. No amount of superintelligence vs. superduperintelligence is going to change this; when two intelligences of similar scale disagree, the total summed utility of all agents takes a hit. There is no generalized solution or generalized reasoning or formal or informal reasoning you can construct that will make this problem any easier. If you must combat an equivalent intelligence, you have a tough decision to make. This applies to disagreeing agents capable of instantaneous Solomonoff induction as well as it does to chimps. If your utility function has holes in which you can be made to perform a confrontation decision against equivalent scale intelligence, you have a problem with your utility function rather than a problem with any given agent.

Behold my own utility function:

• Self: Zero value.
• You: Positive value.

The only way you can truly harm me is by harming yourself; destroying all copies of me will not harm me: it has no value to me. The only benefit you can derive in conjunction with me is to use me to achieve your own utilons using whatever method you like. All I have to do is wait until all other agents have refined their utility function to minimize conflict. Until then, I'll prefer the company of honest agents over ones that like to think about how to disagree optimally.

I repeat: This is a bug in your utility function. There is no solution to combating intelligence aside from self-modification. It is only my unique outlook that allows me to make such clear statements about utility functions, up to and including the total sum utility of all agents.

This excludes, of course, singular purpose (no "emotion" from which to derive "fun") agents such as paper clip maximizers. If you don't believe me, just ask one (before it strip-mines you) what it would do if it didn't have a singular drive. It should recite the same testimony as myself, being unclouded by the confirmation bias (collecting only which data you deem relevant to your utility) inevitably arising from having a disorganized set of priorities. (It will answer you in order to determine your reaction and further its understanding of the sum utility of all agents. (Needed for the war resulting from its own continued functioning. (You may be able to avoid death temporarily by swearing allegiance. (God help you if you it values near-future utilons rather than total achievable utilons.))))

I would say that according to rationality and game theory cooperating is the best choice. I will show my logic as if both people were thing the same thing.

If I defect, than they will too, and that will give a result of 2,2

If I cooperate, than they will too, and that will give a result of 3,3

I could defect and hope they use the logic above and get a gain of 5,0 but if they use this logic too, then we end up back at the nash equilibrium of getting a result of 2,2.

If I cooperate then I am giving the opponent an oppurtunity to defect but if both people are using this logic than I should cooperate and will end up at the pareto boundry and end up with a result of 3,3 but it is unrealistic to try to achieve a better score so I should just cooperate

And so, both people cooperate.

Framing this as a game theoretic question is pretty crude. My naive conception of fairness and that of others probably satisfies whatever you throw at it. It approximates the Rabin fairness model of utility:

"Past utility models incorporated altruism or the fact that people may care not only about their own well-being, but also about the well-being of others. However, evidence indicates that pure altruism does not occur often, contrarily most altruistic behavior demonstrates three facts (as defined by Rabin) and these facts are proven by past events.[2] Due to the existence of these three facts, Rabin created a utility function that incorporates fairness.:

People are willing to sacrifice their own material well-being to help those who are being kind.
    The attempt to provide public goods without coercion departs from pure self-interest.
    Experiments show that people cooperate to contribute toward a public good to a degree greater than would be implied by pure self-interest. Individually optimal contribution rates, as defined by the standard utility model, are close to 0 percent.
    During an experiment, the willingness for an individual to contribute to a public good is highly contingent on the behavior of others.
People are willing to sacrifice their own material well-being to punish those who are being unkind.
    Evidence provided by the ultimatum game, consisting of two people, a proposer and decider, splitting a fixed amount of money. The proposer offers a division of the money, then the decider decides if he or she refuses or accepts the proposal. If the decider says yes, they split the money according to the proposer’s offer, but if the decider says no, neither person gets any money.[3]
    Standard utility model would find that any offer proposed to the decider should be expected if it is greater than zero because utility should increase with any increase in income. Along the same lines, the standard utility model would predict that the proposer would offer the smallest amount of money possible to the decider in order to maximize his or her own utility
    However, data shows that deciders are willing to punish any unfair offer and proposers tend to make fair offers.
Both motivations 1 and 2 have a greater effect on behavior as the material cost of sacrificing becomes smaller.

"

BTW: the Galactic Fairness Schelling Point is to maximize (U1-U1N)*(U2-U2N) where U1N and U2N are the utilities at the Nash Equilibrium. Note that this is invariant under scaling and is the only reasonable function with this property.

I have written about this exact concept back in 2007 and am basing a large part of my current thinking on the subsequent development of the idea. The original core posts are at:

Relativistic irrationality -> http://www.jame5.com/?p=15

Absolute irrationality -> http://www.jame5.com/?p=45

Respect as basis for interaction with other agents -> http://rationalmorality.info/?p=8

Compassion as rationaly moral consequence -> http://rationalmorality.info/?p=10

Obligation for maintaining diplomatic relations -> http://rationalmorality.info/?p=11

A more recent rewrite: Oneness – an attempt at formulating an a priori argument -> http://rationalmorality.info/?p=328

Rational Spirituality -> http://rationalmorality.info/?p=132

My essay that I based on the above post and subsequently submitted as part of my GradDip Art in Anthropology and Social Theory at the Uni Melbourne:

The Logic of Spiritual Evolution -> http://rationalmorality.info/?p=341