Hell is Game Theory Folk Theorems

Seriously, what? I'm missing something critical. Under the stated rules as I understand them, I don't see why anyone would punish another player for reducing their dial.

You state that 99 is a nash equilibrium, but this just makes no sense to me. Is the key that you're stipulating that everyone must play as though everyone else is out to make it as bad as possible for them? That sounds like an incredibly irrational strategy.

I think ideas like Nash equilibrium get their importance from predictive power: do they correctly predict what will happen in the real world situation which is modeled by the game. For example, the biological situations that settle on game-theoretic equilibria even though the "players" aren't thinking at all.

In your particular game, saying "Nash equilibrium" doesn't really narrow down what will happen, as there are equilibria for all temperatures from 30 to 99.3. The 99 equilibrium in particular seems pretty brittle: if Alice breaks it unilaterally on round 1, then Bob notices that and joins in on round 2, neither of them end up punished and they get 98.6 from then on.

More generally, in any game like this where everyone's interests are perfectly aligned, I'd expect cooperation to happen. The nastiness of game theory really comes from the fact that some players can benefit by screwing over others. The game in your post doesn't have that, so any nastiness in such a game is probably an analysis artifact.

Reminds me of this from Scott Alexander's Meditations on Moloch:

Imagine a country with two rules: first, every person must spend eight hours a day giving themselves strong electric shocks. Second, if anyone fails to follow a rule (including this one), or speaks out against it, or fails to enforce it, all citizens must unite to kill that person. Suppose these rules were well-enough established by tradition that everyone expected them to be enforced.

I don't understand the motivation to preserve the min-max value, and perhaps that's why it's a folk theorem rather than an actual theorem.  Each participant knows that they can't unilaterally do better than 99.3, which they get by choosing 30 while the other players all choose 100.  But a player's maxing (of utility; min temperature) doesn't oblige them to correct or reduce their utility (by raising the temperature) just because the opponents fail to minimize the player's utility (by raising the temperature).

There is no debt created anywhere in the model or description of the players.  Everyone min-maxes as a strategy, picking the value that maximizes each player's utility assuming all opponents minimize that player's utility.  But the other players aren't REQUIRED to play maximum cruelty - they're doing the same min-max strategy, but for their own utility, leading everyone to set their dial to 30.  

The Wikipedia article has an example that is easier to understand:

Anthropology: in a community where all behavior is well known, and where members of the community know that they will continue to have to deal with each other, then any pattern of behavior (traditions, taboos, etc.) may be sustained by social norms so long as the individuals of the community are better off remaining in the community than they would be leaving the community (the minimax condition).

Correct me if I'm wrong:

The equilibrium where everyone follows "set dial to equilibrium temperature" (i.e. "don't violate the taboo, and punish taboo violators") is only a weak Nash equilibrium.

If one person instead follows "set dial to 99" (i.e. "don't violate the taboo unless someone else does, but don't punish taboo violators") then they will do just as well, because the equilibrium temp will still always be 99. That's enough to show that it's only a weak Nash equilibrium.

Note that this is also true if an arbitrary number of people deviate to this strategy.

If everyone follows this second strategy, then there's no enforcement of the taboo, so there's an active incentive for individuals to set the dial lower.

So a sequence of unilateral changes of strategy can get us to a good equilibrium without anyone having to change to a worse strategy at any point. This makes the fact of it being a (weak) Nash equilibrium not that compelling to me; people don't seem trapped unless they have some extra laziness/inertia against switching strategies.

But (h/t Noa Nabeshima) you can strengthen the original, bad equilibrium to a strong Nash equilibrium by tweaking the scenario so that people occasionally accidentally set their dials to random values. Now there's an actual reason to punish taboo violators, because taboo violations can happen even if everyone is following the original strategy.

Nice provocative post :-)

It's good to note that Nash equilibrium is only one game-theoretic solution concept.  It's popular in part because under most circumstances at least one is guaranteed to exist, but folk theorems can cause there to be a lot of them.  In contexts with lots of Nash equilibria, game theorists like to study refinements of Nash equilibrium, i.e., concepts that rule out some of the Nash equilibria.  One relevant refinement for this example is that of strong Nash equilibrium, where no subset of players can beneficially deviate together.  Many games don't have any strong Nash equilibrium, but this one does have strong Nash equilibria where everyone plays 30 (but not where everyone plays 99).  So if one adopts the stronger concept, the issue goes away.

But is that reasonable reassurance or just wishful thinking (wishful concept selection)?  Really answering that requires some theory of equilibrium selection.  One could consider learning algorithms, evolutionary dynamics, some amount of equilibrium design / steering, etc., to get insight into what equilibrium will be reached.  My own intuition is still to be cautiously optimistic about folk theorems, since I generally think the better equilibria are more likely to be selected (more robust to group deviations, satisfy individual rationality by a larger margin, etc.).  But I would also like to have a better theory of this.

I have a suspicion that the viscerally unpleasant nature of this example is making it harder for readers to engage with the math.

Example origin scenario of this Nash equilibrium from GPT-4:

In this hypothetical scenario, let's imagine that the prisoners are all part of a research experiment on group dynamics and cooperation. Prisoners come from different factions that have a history of rivalry and distrust. 

Initially, each prisoner sets their dial to 30 degrees Celsius, creating a comfortable environment. However, due to the existing distrust and rivalry, some prisoners suspect that deviations from the norm—whether upward or downward—could be a secret signal from one faction to another, possibly indicating an alliance, a plan to escape, or a strategy to undermine the other factions.

To prevent any secret communication or perceived advantage, the prisoners agree that any deviation from the agreed-upon temperature of 30 degrees Celsius should be punished with a higher temperature of 100 degrees Celsius in the next round. They believe that this punishment will deter any secret signaling and maintain a sense of fairness among the factions.

Now, imagine an external party with access to the temperature control system decides to interfere with the experiment. This person disables the dials and changes the temperature to 99 degrees Celsius for a few rounds, heightening the prisoners' confusion and distrust.

When the external party re-enables the dials, the prisoners regain control of the temperature. However, their trust has been severely damaged, and they are now unsure of each other's intentions. In an attempt to maintain fairness and prevent further perceived manipulation, they decide to adopt a strategy of punishing any deviations from the new 99 degrees Celsius temperature.

As a result, the prisoners become trapped in a suboptimal Nash equilibrium where no single prisoner has an incentive to deviate from the 99 degrees Celsius strategy, fearing retaliation in the form of higher temperatures. In this scenario, a combination of technical glitches, external interference, miscommunication, and distrust leads to the transition from an agreed-upon temperature of 30 to a suboptimal Nash equilibrium of 99 degrees Celsius.

As time goes on, this strategy becomes ingrained, and the prisoners collectively settle on an equilibrium temperature of 99 degrees Celsius. They continue to punish any deviations—upward or downward—due to their entrenched suspicion and fear of secret communication or potential advantage for one faction over another. In this situation, the fear of conspiracy and the desire to maintain fairness among the factions lead to a suboptimal Nash equilibrium where no single prisoner has an incentive to deviate from the 99 degrees Celsius strategy.

Two (related) morals of the story:

1. A really, really stupid strategy can still meet the requirements for being a Nash equilibrium, because a strategy being a Nash equilibrium only requires that no one player can get a better result for themselves when only that player is allowed to change strategies.

2. A game can have more than one Nash equilibrium, and a game with more than one Nash equilibrium can have one that's arbitrarily worse than another.

A copy of my comment from the other thread [LW · GW]:

The only thing I learned from this post is that, if you use mathematically precise axioms of behavior, then you can derive weird conclusions from game theory. This part is obvious and seems rather uninteresting.

The strong claim, namely the hell scenario, comes from then back-porting the conclusions from this mathematical rigor to our intuitions about a suggested non-rigorous scenario.

But this you cannot do unless you've confirmed that there's a proper correspondence from your axioms to the scenario.

For example, the scenario suggested that humans are imprisoned; but the axioms require unwilling, unchanging, inflexible robots. It suggests that your entire utility derives from your personal suffering from the temperature, so these suffering entities cannot be empathetic humans. Each robot is supposed to assume that, if it changes its strategy, none of the others does the same.

And so on. Once we've gone through the implications of all these axioms, we're still left with a scenario that seems bad for those inside it. But it doesn't deserve to be called hell, nor to appeal to our intuitions about how unpleasant this would be to personally experience. Because if humans were in those cages, that strategy profile absolutely would not be stable. If the math says otherwise, then the unrealistic axioms are at fault.

I don't think the 'strategy' used here (set to 99 degrees unless someone defects, then set to 100) satisfies the "individual rationality condition". Sure, when everyone is setting it to 99 degrees, it beats the minmax strategy of choosing 30. But once someone chooses 30, the minmax for everyone else is now to also choose 30 - there's no further punishment that will or could be given. So the behavior described here, where everyone punishes the 30, is worse than minmaxing. At the very least, it would be an unstable equilibrium that would have broken down in the situation described - and knowing that would give everyone an incentive to 'defect' immediately.

In a realistic setting agents will be highly incentivized to seek other forms of punishment besides turning dial. But nice toy hell.

Curated. As always, I'm fond of a good dialog. Also usually (though comes out less often), I'm fond of getting taught interesting game theory results, particularly outside the range of games most discussed, in this case, more focusing on iterated stuff which can get weirder. Kudos, I really like having posts like these on LW.

by min-maxing (that is, picking their strategy assuming other players make things as bad as possible for them, even at their own expense) [bold and italics added]

Why would anyone assume this and make decisions based on it?  I have never understood this aspect of the Nash equilibrium. (Edit: never mind, I thought it was claimed that this was part of how Nash equilibria worked, and assumed this was the thing I remembered not understanding about Nash equilibria, but that seems wrong)

Any Nash Equilibrium can be a local optimum. This example merely demonstrates that not all local optima are desirable if you are able to view the game from a broader context. Incidentally, evolution has provided us with some means to try and get out of these local optima. Usually by breaking the rules of the game or leaving the game or seemingly not acting rationally from the perspective of the local optimum.

Just to clarify, the complete equilibrium strategy alluded to here is:

"Play 99 and, if anyone deviates from any part of the strategy, play 100 to punish them until they give in"

Importantly, this includes deviations from the punishment. If you don't join the punishment, you'll get punished. That makes it rational to play 99 and punish deviators.

The point of the Folk Theorems are that the Nash Equilibrium notion has limited predictive power in repeated games like this, because essentially any payoff could be implemented as a similar Nash equilibrium. That doesn't mean it has no predictive power - if everyone in that room keeps playing that strategy, you'd play it too, eventually. That's a valid prediction.

What seems wrong about the described equilibrium is that you would expect players to negotiate away from it. If they can communicate somehow (it's hard to formalize, but you could even conceive of this communication as happening through their dials), they would likely arrange a Nash equilibrium around different temperature. Without communication, it seems unlikely that a complex Nash equilibrium like above would even arise, as players would likely start out by repeating the so-called stage game, where they just set the dial to whatever temperature they want (how would they even know they will get punished?).

In fact, the notion of renegotiation actually draws certain punishments in question too (e.g. punish forever after one deviation), which means they are less credible threats, ones that rational players would arguably not be afraid of testing out. This can be formalized as a so-called equilibrium refinement, which makes stronger predictions than the classic Nash equilibrium notion, about the types of strategies rational players may employ, see e.g. the seminal Farrell and Maskin (1989).

I don’t think this works. Here is my strategy:

1. I set the dial to 30 and don’t change it no matter what.
2. In the second round, temperature lowers to 98.3.
3. In the third round all the silly people except me push the dial to 100 and thus we get 99.3.
4. I don’t deviate from 30, no matter how many rounds of 99.3 be there.
5. At some point someone figures out that I am not going to deviate and they defect too, now we have 88.6.
6. The avalanche has started. It won’t take long to get to 30.

Now, this was a perfectly rational course of action for me. I knew that I will suffer temporarily, but in exchange I got a comfortable temperature for eternity.

Prove me wrong.

I can see why feasibility + individual rationality makes a payoff profile more likely than any profile missing one of these conditions, but I can’t see why I should consider every profile satisfying these conditions as likely enough to be worth worrying about

This is about as convincing as a scarecrow. Maybe I'm commiting some kinda taboo not trying to handle a strawman like a a real person but to me the mental exercise of trying faulty thought experiments is damaging when you can just point to the fallacy and move on.

I'd be interested in someone trying to create the proposed simulation without the presumptive biases. Would AI models given only pain and relief with such a dial to set themselves come to such equilibrium? I think obviously not. I don't hear anyone arguing that's wrong just that it's a misunderstanding of the preposition and if that's the case what the hell is the point? Are we talking about AI behavior or aren't we? Are we not talking about if or not a technological singularity is going to be "good or bad" across arbitrary non naturalistic/human centric ethics?

If it's not those things the preface must be changed, if it is those things the logic is flawed.

The All-Knowing doesn't need to punish, for He can not be threatened.