I found these three papers highly useful, especially the first one

1. Statistical physics of human cooperation

2. Evolutionary dynamics of group interactions on structured populations: a review

3. Understanding and Addressing Cultural Variation in Costly Antisocial Punishment

What this means is that two such agents with incompatible notions of fairness can't get all the way to the Pareto frontier, but the closer their notions of fairness are to each other, the closer they can get.

this is helpful for clarifying some thoughts. Thanks.

most collective action problems are bargaining problems.

I came to this conclusion and the nice thing about it is that it collapses a bunch of problems into the signaling landscape problem. (I don't know if there's an academic term for the prevailing state of the signaling landscape at a given time). The frustrating thing about the signaling landscape is that there's a lot of security through obscurity (eg shibboleths).

A short note to start the review that the author isn’t happy [LW(p) · GW(p)] with how it is communicated. I agree it could be clearer and this is the reason I’m scoring this 4 instead of 9. The actual content seems very useful to me.

AllAmericanBreakfast has already reviewed [LW(p) · GW(p)] this from a theoretical point of view but I wanted to look at it from a practical standpoint.

***

To test whether the conclusions of this post were true in practice I decided to take 5 examples from the Wikipedia page on the Prisoner’s dilemma and see if they were better modeled by Stag Hunt or Schelling Pub:

• Climate negotiations
• Relationships
• Marketing
• Doping in sport
• Cold war nuclear arms race

Detailed analysis of each is at the bottom of the review.

Of these 5, 3 (Climate, Relationships, Arms race) seem to me to be very well modeled by Schelling Pub. 

Due to the constraints on communication allowed between rival companies it is difficult to see marketing (where more advertising = defect) as a Schelling Pub game. There probably is an underlying structure which looks a bit like Schelling Pub but it is very hard to move between Nash Equilibria. As a result I would say that Prisoner’s Dilemma is a more natural model for marketing.

The choice of whether to dope in sport is probably best modeled as a Prisoner’s dilemma with an enforcing authority which punishes defection. As a result, I don’t think any of the 3 games are a particularly good model for any individual’s choice. However, negotiations on setting up the enforcing authority and the rules under which it operates are more like Schelling Pub. Originally I thought this should maybe count as half a point for the post but thinking about it further I would say this is actually a very strong example of what the post is talking about – if your individual choice looks like a Prisoner’s Dilemma then look for ways to make it into a Schelling Pub. If this involves setting up a central enforcement agency then negotiate to make that happen.

So I’m scoring it 4 out of 5 Prisoners Dilemmas examined are better modeled as Schelling Pubs, which is in line with the “most” claims of the post title. 

The example which was least like Schelling Pub was the one where communication was difficult/impossible due to external rules. I think the value of communication is implicit in the post but it would be helpful to have it spelled out explicitly.

One other thing which might be useful from a practical point is that things which don't initially seem iterated may be able to be iterated if you split them into smaller tasks. You don't have to reduce your nuclear arsenal or decarbonise all at once, you can do a little bit and then check that the others have done the same before continuing. This seems obvious on a national level but maybe not so obvious on a personal level.

***

(Read beyond here only if you're interested in more detail of the examples - it doesn't add to the conclusions)

Below is my analysis of the 5 items chosen from the Prisoner’s dilemmas example list on Wikipedia. In discussing Stag Hunts (SH) I use the post’s list of 4 items which might make something more like a SH than a Prisoner’s Dilemma (PD).

Climate negotiations

• PD shaped
  • Each country benefits from stable climate
  • Each would prefer that they put in minimum effort to achieve it
• SH shaped?
  • Iteration: Yes - it is iterated over years and agreements
  • Reputation: It seems like yes, although I’m not sure how this works out in practice
  • Enforecable contracts: Not really
  • Superrationality: Possibly
• SP shaped?
  • It seems yes.
    • As an example, recall the coal amendment from the Glasgow talks
    • This is an amendment which favours some countries over others and means that those disfavoured will put in relatively more effort for a given amount of CO2 reduction compared to the original wording.
  • It seems obvious that the actual agreement probably isn’t on the Pareto frontier
  • I think the final figure from the post gives a good mental model of what is going on

Relationships

• PD shaped
  • In theory, for an individual action, you’re better off if you get your own way over your partner
• SH shaped?
  • Iteration: Yes (+ your partner can just leave)
  • Reputation: Very important in small communities, less so in large ones
  • Enforceable contracts: Sometimes (prenup)
  • Superrationality: Potentially yes but not required
  • It will be better to hunt stag together just from iteration
  • Abusive relationships are not SH shaped as the partner can’t or won’t leave, especially when there’s no reputational effect that the abusive partner cares about
• SP shaped?
  • Ideally [LW · GW].

Marketing

• PD shaped
  • 2 rival companies with equally effective marketing departments are in a roughly PD shaped game (assuming customer pool is fixed size)
  • If one spends money on advertising (defect) then the other is disadvantaged if they don’t
  • But both would be better off if neither advertised
  • Often the customer pool is not of fixed size which would mean that this may not really be a PD in real life
  • It is important here to note that collusion between companies is generally forbidden so communication is not allowed
• SH shaped?
  • Iteration: Yes.
  • Reputation: Not really – there aren’t negative reputation effects to advertising
  • Enforceable contracts: No
  • Modelling other players: Yes but they are (accurately) modelling each other as playing defect
  • In theory the businesses could get into C-C but advertising is so ingrained as the default choice that this would be hard
  • Possibly there’s an availability heuristic problem here – I’ll obviously remember the examples of industries which are stuck in D-D as I constantly see their adverts (Pepsi vs Coke, tech companies, supermarkets).
    • I tried to think of industries that aren’t advertising much but I’m drawing a blank.
• SP shaped?
  • In theory maybe, however the lack of communication makes it extremely hard for companies to change between solutions

Doping in sport

• PD shaped
  • Using drugs gives you an edge but has a price potential price with medical dangers
  • There is a potential difference from “standard” PD in that doping is disadvantageous to a more able athlete if the less able athlete is not doping (which would make this an alibi game).
• SH shaped?
  • Iteration: Yes (although results of previous iterations are not very legible)
  • Reputation: Yes (again, legibility problems)
  • Enforceable contract: Yes – drugs testing and bans
  • Superrationality: No
  • The enforceable contract is probably the biggest effect here – just relying on iteration and reputation would be insufficient
• SP shaped?
  • Because the solution is an enforceable contract the decision is not very SP shaped on an individual basis
  • Negotiating what the contract should be is SP shaped
    • What makes a medical exemption ok?
    • How much inhaler is an ok amount?

Cold war nuclear arms race

• PD shaped
  • Making more nukes is equivalent to defect
• SH shaped?
  • Iteration: Yes, treaty followed treaty. Inspections were allowed to verify adherence.
  • Reputation: A bit but probably not in a relevant way
  • Enforceable contract: Not really – if one country reneged then it’s not like the other country could sue
  • Superrationality: Yes. It seems both players realised D-D was terrible and wanted to play C-C. They couldn't rely on just this so iteration was very important.
• SP shaped?
  • Deciding the terms of the agreement is SP shaped
    • What size arsenal is better for which country?
    • Which particular weapons are better for which country?

So, why do we perceive so many situations to be Prisoner's Dilemma -like rather than Stag Hunt -like?

I don't think that we do, exactly. I think that most people only know the term "prisoners' dilemma" and haven't learned any more game theory than that; and then occasionally they go and actually attempt to map things onto the Prisoners' Dilemma as a result. :-/

The goal of this post is to help us understand the similarities and differences between several different games, and to improve our intuitions about which game is the right default assumption when modeling real-world outcomes.

My main objective with this review is to check the game theoretic claims, identify the points at which this post makes empirical assertions, and see if there are any worrisome oversights or gaps. Most of my fact-checking will just be resorting to Wikipedia.

Let’s start with definitions of two key concepts.

Pareto-optimal: One dimension cannot improve without a second worsening.

Nash equilibrium: No player can do better by unilaterally changing their strategy.

Here’s the payoff matrix from the one-shot Prisoner’s Dilemma and how it relates to these key concepts.

This article outlines three possible relationships between Pareto-optimality and Nash equilibrium.

1. There are no Pareto-optimal Nash equilibria.
2. There is a single Pareto-optimal Nash equilibrium, and another equilibrium that is not Pareto-optimal.
3. There are multiple Pareto-optimal Nash equilibria, which benefit different players to different extents.

The author attempts to argue which of these arrangements best describes the world we live in, and makes the best default assumption when interpreting real-world situations as games. The claim is that real-world situations most often resemble iterated PDs, which have multiple Pareto-optimal Nash equilibria benefitting different players to different extents. I will attempt to show that the author’s conclusion only applies when modeling superrational entities, or entities with an unbounded lifespan, and give some examples where this might be relevant.

Iterated Prisoner’s Dilemma is a little more complex than the author states. If the players know how many turns the game will be played for, or if the game has a known upper limit of turns, the Nash equilibrium is always to defect. However, if the players are superrational, meaning that they not only are perfectly rational but also assume all other players are too and that superrational players always converge on the same strategy, then they’ll always cooperate.

As such, the Nash equilibrium for rational, but not superrational, players for games with fixed or upper-bounded N is the same as for the single-shot game. In real life, any game played between human beings that takes a non-zero amount of time has an upper bound on the number of turns, given that we currently must expect ourselves to die. Therefore, game theory suggests that the Nash equilibrium strategy for all iterated Prisoner’s Dilemmas between rational players is defect/defect. Therefore, the claims about iterated PD in steps 2-5 in the author’s argument summary only seems to hold if we are talking about non-human entities with unbounded life expectancies, or if humans are modeled as superrational agents.

Let’s gesture at some plausible but extremely speculative real-world examples of how games with an unbounded upper limit of turns or superrationality might be reasonable models for human games.

Social entities, such as governments, corporations, and cultures, if they can be modeled as agents, could be seen as having unbounded lifespans. When we make strict game-theoretic arguments, we can be equally strict in mathematical assumptions about other facets of reality. If understanding of physics is imperfect, and if there is a non zero possibility of some continuity of agents not just into the distant future, but infinitely into time, then they could be modeled as having unbounded lifespans. If this holds, and if social entities are causally responsible for the way that most games unfold, then this may rescue the argument.

A second angle on this idea is that humans may have a psychological tendency to interpret long periods of time as equivalent to infinite in length, analogously to how people intuitively round low probabilities to 0 and high probabilities to 1. This suggests that studies investigating strategies empirically chosen by people playing iterated PDs in the lab would increasingly approach optimal strategies for unbounded iterated PDs as the turn count increases. A complicating factor here is that we must disambiguate whether the assumption is that humans “round” long lengths of time or large numbers of turns to infinity.

Outcomes for superrational agents are better than those for rational agents in the turn-bounded iterated PD. Imagine a large set of agents, each playing a randomly-chosen strategy. Some of those strategies may match the superrational strategy in iterated PD. If they face selection pressure over time, with some spontaneous generation of new agents, agents playing the superrational strategy may eventually dominate the population of agents. Conceivably, humans, and our social ancestor species, may have been genetically hardwired via group selection to adopt superrational strategies by instinct. This aligns with the psychological explanation.

Edit: Vanessa Kosoy pointed out below that iterated PDs with a finite but unknown number of iterations, or slightly noisy agents, can have Nash equilibria involving cooperation. We therefore don't need to resort to such exotic explanations as I've offered here to explain how abramdemski's arguments 2-5 hold, and we don't need to "trick ourselves into cooperation" in such scenarios.

If this argument holds water, how does it affect the original agenda of this article, which was to inform our intuitions about how to model real-world games? It suggests that these twin questions, of psychological “rounding” and a possible group-selection account of how this might have evolved, would be important to investigate to increase our confidence in this heuristic. If true, it also suggests vulnerabilities in normal human approaches to games. Our ability to cooperate, under this hypothesis, depends on our ability to trick ourselves into cooperation by conveniently ignoring the inevitable end of our games.

The local argument made by this post needn’t be true for its conclusion to be true, and if the post seems plausible because it aligns with our real-world experience, we might want to appreciate the article for the conclusion it articulates as well as the argument it makes in support of that conclusion. The conclusion is that in most situations, we have multiple Pareto-optimal Nash equilibria, favoring different agents. Colloquially, many human problems are about fairness and resource allocation, and the threats and strategies people use to steer negotiation toward the outcome that favors them the most, while still achieving a fundamentally cooperative outcome. 

This seems to me like an articulate, usefully predictive, simple, and realistic depiction of an enormous number of fundamental challenges in human organization. Although I don’t think that the original post’s game-theoretic argument is airtight, I think its psychological and sociological plausibility in conjunction with a tweaked game-theoretic argument makes it worthwhile and interesting. I also appreciate the care the author took to summarize, update, and respond to comments. Pointing out similarities and differences in the relationship between Nash equilibria and Pareto optimality in the various games also helped me understand them better, which I appreciate.

Only a third of the way through so far, but confused – you cite "iteration and enforcement mechanisms" as things that make PDs more like Stag Hunts. But, isn't iteration and enforcement a property that either PD or SH can have? My understanding of the difference between  PD and Stag Hunt was about what the actual payoffs are, and that C/C can be a nash equilibrium because you wouldn't get more payoff by defecting (although D/D is also a nash equilibrium)

Here's my summary of this post. Is this getting at the point you're trying to make?

The essential difference between a one-off Prisoner's Dilemma and a Stag Hunt is that in the one-shot PD, the prisoners cannot punish or reward each other for cooperating. In a Stag Hunt, the hunters can punish defection and reward cooperation. In both cases, the best outcome is equally good for all players.

The essential difference between a Stag Hunt and a Battle of the Sexes is that in the Stag Hunt, the best outcome is equally successful for all. In Battle of the Sexes, the best one-off outcomes are always unfair to at least one of the participants.

In most real-world situations, we can enforce cooperation. Generally, the outcomes won't be perfectly fair. They'll resemble a Battle of the Sexes more than a Stag Hunt or one-off Prisoner's Dilemma. So the problem is negotiating which unfair outcome the participants will choose. But because the Prisoner's Dilemma is so well-known, people often resort to it as their first game-theoretic analysis of any given situation.

Nitpick:

While by the end of the article I feel like I understood what you mean by battle of the sexes, I didn't at the start and there is neither an explanation of the battle of the sexes game (even at the beginning of the section titled Battle of the Sexes!) nor is there a link to a post or article about it.

In the Alice/Bob diagrams, I am confused why the strategies are parameterized by the frequency of cooperation. Don't these frequencies depend on what the other player does, so that the same strategy can have different frequencies of cooperation depending on who the other player is?

So... I think this is one of the most important posts I've read lately (it feels like it crystalizes and gives clarity to some key societal problems). 

Unfortunately... "Battle of the Sexes" is a super distracting name, and a confusing one. I think it's reasonable for LW to not go out of their way to cave to this week's internet fashions. But, when I've tried to get other people to read this post, even people who care about game theory (but aren't necessarily super well versed in it), they had a very different expectation of what Battle of the Sexes meant. (They assumed this post had something to do with gender, which I suspect contributed to them bouncing off of it)

I'm not actually sure what to do because there isn't another great name for Battle of the Sexes. Bach vs Stravinsky is another name it has used, which at least has the advantage of not obviously suggesting any particular interpretation. It also has the same abbreviation as Battle of the Sexes, so it's a bit more resilient to summarizing.

So I weakly suggest changing the title to replace Battle of the Sexes with Bach or Stravinsky. I'm not super confident because it is a more complicated sounding title that's already pushing the boundary of complexity, but I think probably better on net.

(I think it should probably be basically required reading for LW users to know PD, SH and BoS games as core game theory concepts, but apparently we're away aways from that world)

I think I will probably write a variation on this post with a title something like "which stag hunt are you coordinating on?", approaching the content from a slightly more actionable angle while also summarizing the key pieces here. But, meanwhile I like that this post focuses explicitly on the game theory connections.

not all conflicts are zero-sum.

This should be the lede. Most real-world interactions lose a lot of options, and a lot of potential value, by being simplified to a (n iterated) PD, SH, or BotS.

In reality, there's almost always un-modeled transfer and payouts - just being able to say "good job, thanks!" after a result is FREE UTILITY! Also, non-pathological humans have terms for the other player(s) in their utility function. Most importantly, there are far too many future games in the iteration set of a human lifetime for anyone to model, so reputation and self-image effects very often will dominate the modeled payouts.

The longer (i.e., more iterations) you spend in the shaded triangles of defection the more you'll be pulled to the defect-defect equilibrium as a natural reaction to what the other person is doing and the outcome you're getting. The longer you spend in the middle "wedge of cooperation", the more you'll end moving up and to the right in Pareto improvements. So we want to make that wedge bigger.

The size of that wedge is determined by the ratio of a player's outcome from C-C to their outcome in D-D. In this case the ratio is 2:1, so the wedge is between the slopes of 2 and 1/2. If C-C only guaranteed 1.1-1.1 to each player while a defection got them at least 1, the wedge would be a tiny sliver. Conversely, if the payoff for C-C was 999-999 almost the entire square would be the wedge. 

But the bigger the wedge, the more difference there is between outcomes on the pareto frontier so the outcome of 100% C-C is a lot less stable than if any deviation from it immediately led to non-equilibrium points that degenerate to D-D.

I'm OOTL, can someone send me a couple links that explain the game theory that's being referenced when talking about a "battle of the sexes"? I have a vague intuition from the name alone, but I feel this is referencing a post I haven't read.

Edit: https://en.wikipedia.org/wiki/Battle_of_the_sexes_(game_theory)

I like this, in the sense that it's provoking fascinating thoughts and makes me want to talk with the author about it further. As a communication of a particular concept? I'm kinda having a hard time following what the intent is.

Promoted to curated: This post made a point that I have been grasping at for a while, and made it quite well. For better or for worse, I use Prisoner's Dilemma analogies at least 5 times a week, and so understanding the dynamics around those dilemmas is quite important to me. This post felt like it connected a number of ideas in this space in a way that I expect to refer back to in the future at least a few times.

I would prefer headlines to spell out the terms they use to make it clearer to a reader that scans the headlines what a post is about. 

(The titular insight seems pretty deep, thanks for sharing this)