The Schelling Choice is "Rabbit", not "Stag"

post by Raemon · 2019-06-08T00:24:53.568Z · score: 88 (26 votes) · LW · GW · 24 comments

Contents

  A recap of 'Rabbit' vs 'Stag'
  A lightweight, but concrete example
  Takeaways
    Sometimes stag hunts are worth it.
    The schelling option is Rabbit
  My own strategies right now
    In general, choose Rabbit. 
    Follow rabbit trails into Stag Country
    Get curious about other people's frames
    Occasionally run "Kickstarters for Stag Hunts." If people commit, hunt stag.
None
24 comments

Followup/distillation/alternate-take on Duncan Sabien's Dragon Army Retrospective and Open Problems in Group Rationality.


There's a particular failure mode I've witnessed, and fallen into myself:

I see a problem. I see, what seems to me, to be an obvious solution to the problem. If only everyone Took Action X, we could Fix Problem Z. So I start X-ing, and maybe talking about how other people should start X-ing. Action X takes some effort on my part but it's obviously worth it.

And yet... nobody does. Or not enough people do. And a few months later, here I'm still taking Action X and feeling burned and frustrated.

Or –

– the problem is that everyone is taking Action Y, which directly causes Problem Z. If only everyone would stop Y-ing, Problem Z would go away. Action Y seems obviously bad, clearly we should be on the same page about this. So I start noting to people when they're doing Action Y, and expect them to stop.

They don't stop.

So I start subtly socially punishing them for it.

They don't stop. What's more... now they seem to be punishing me.

I find myself getting frustrated, perhaps angry. What's going on? Are people wrong-and-bad? Do they have wrong-and-bad beliefs?

Alas. So far in my experience it hasn't been that simple.


A recap of 'Rabbit' vs 'Stag'

I'd been planning to write this post for years. Duncan Sabien went ahead and wrote it before I got around to it. But, Dragon Army Retrospective and Open Problems in Group Rationality are both lengthy posts with a lot of points, and it still seemed worth highlighting this particular failure mode in a single post.

I used to think a lot in terms of Prisoner's Dilemma, and "Cooperate"/"Defect." I'd see problems that could easily be solved if everyone just put a bit of effort in, which would benefit everyone. And people didn't put the effort in, and this felt like a frustrating, obvious coordination failure. Why do people defect so much?

Eventually Duncan shifted towards using Stag Hunt rather than Prisoner's Dilemma as the model here. If you haven't read it before, it's worth reading the description in full. If you're familiar you can skip to my current thoughts below.

[note: I changed the word 'utility' in the quoted section to 'resource' which I think is more technically accurate]

My new favorite tool for modeling this is stag hunts, which are similar to prisoner’s dilemmas in that they contain two or more people each independently making decisions which affect the group. In a stag hunt:
—Imagine a hunting party venturing out into the wilderness.
— Each player may choose stag or rabbit, representing the type of game they will try to bring down.
— All game will be shared within the group (usually evenly, though things get more complex when you start adding in real-world arguments over who deserves what).
— Bringing down a stag is costly and effortful, and requires coordination, but has a large payoff. Let’s say it costs each player 5 points of resource (time, energy, bullets, etc.) to participate in a stag hunt, but a stag is worth 50 resource (in the form of food, leather, etc.) if you catch one.
— Bringing down rabbits is low-cost and low-effort and can be done unilaterally. Let’s say it only costs each player 1 point of resource to hunt rabbit, and you get 3 resource as a result.
— If any player unexpectedly chooses rabbit while others choose stag, the stag escapes through the hole in the formation and is not caught. Thus, if five players all choose stag, they lose 25 resource and gain 50 resource, for a net gain of 25 (or +5 apiece). But if four players choose stag and one chooses rabbit, they lose 21 utility and gain only 3.
This creates a strong pressure toward having the Schelling choice be rabbit. It’s saner and safer (spend 5, gain 15, net gain of 10 or +2 apiece), especially if you have any doubt about the other hunters’ ability to stick to the plan, or the other hunters’ faith in the other hunters, or in the other hunters’ current resources and ability to even take a hit of 5 resource, or in whether or not the forest contains a stag at all.
Let’s work through a specific example. Imagine that the hunting party contains the following five people:
Alexis (currently has 15 resource “in the bank”)
Blake (currently has 12)
Cameron (9)
Dallas (6)
Elliott (5)
If everyone successfully coordinates to choose stag, then the end result will be positive for everyone. The stag costs everyone 5 resource to bring down, and then its 50 resource is divided evenly so that everyone gets 10, for a net gain of 5. The array [15, 12, 9, 6, 5] has bumped up to [20, 17, 14, 11, 10].
If everyone chooses rabbit, the end result is also positive, though less excitingly so. Rabbits cost 1 to hunt and provide 3 when caught, so the party will end up at [17, 14, 11, 8, 7].
But imagine the situation where a stag hunt is attempted, but unsuccessful. Let’s say that Blake quietly decides to hunt rabbit while everyone else chooses stag. What happens?
Alexis, Cameron, Dallas, and Elliott each lose 5 resource while Blake loses 1. The rabbit that Blake catches is divided five ways, for a total of 0.6 resource apiece. Now our array looks like [10.6, 11.6, 4.6, 1.6, 0.6].
(Remember, Blake only spent 1 resource in the first place.)
If you’re Elliott, this is a super scary result to imagine. You no longer have enough resources in the bank to be self-sustaining—you can’t even go out on another rabbit hunt, at this point.
And so, if you’re Elliott, it’s tempting to preemptively choose rabbit yourself. If there’s even a chance that the other players might defect on the overall stag hunt (because they’re tired, or lazy, or whatever) or worse, if there might not even be a stag out there in the woods today, then you have a strong motivation to self-protectively husband your resources. Even if it turns out that you were wrong about the others, and you end up being the only one who chose rabbit, you still end up in a much less dangerous spot: [10.6, 7.6, 4.6, 1.6, 4.6].
Now imagine that you’re Dallas, thinking through each of these scenarios. In both cases, you end up pretty screwed, with your total resource reserves at 1.6. At that point, you’ve got to drop out of any future stag hunts, and all you can do is hunt rabbit for a while until you’ve built up your resources again.
So as Dallas, you’re reluctant to listen to any enthusiastic plan to choose stag. You’ve got enough resources to absorb one failure, and so you don’t want to do a stag hunt until you’re really darn sure that there’s a stag out there, and that everybody’s really actually for real going to work together and try their hardest. You’re not opposed to hunting stag, you’re just opposed to wild optimism and wanton, frivolous burning of resources.
Meanwhile, if you’re Alexis or Blake, you’re starting to feel pretty frustrated. I mean, why bother coming out to a stag hunt if you’re not even actually willing to put in the effort to hunt stag? Can’t these people see that we’re all better off if we pitch in hard, together? Why are Dallas and Elliott preemptively talking about rabbits when we haven’t even tried catching a stag yet?
I’ve recently been using the terms White Knight and Black Knight to refer, not to specific people like Alexis and Elliott, but to the roles that those people play in situations requiring this kind of coordination. White Knight and Black Knight are hats that people put on or take off, depending on circumstances.
The White Knight is a character who has looked at what’s going on, built a model of the situation, decided that they understand the Rules, and begun to take confident action in accordance with those Rules. In particular, the White Knight has decided that the time to choose stag is obvious, and is already common knowledge/has the Schelling nature. I mean, just look at the numbers, right?
The White Knight is often wrong, because reality is more complex than the model even if the model is a good model. Furthermore, other people often don’t notice that the White Knight is assuming that everyone knows that it’s time to choose stag—communication is hard, and the double illusion of transparency is a hell of a drug, and someone can say words like “All right, let’s all get out there and do our best” and different people in the room can draw very different conclusions about what that means.
So the White Knight burns resources over and over again, and feels defected on every time someone “wrongheadedly” chooses rabbit, and meanwhile the other players feel unfairly judged and found wanting according to a standard that they never explicitly agreed to (remember, choosing rabbit should be the Schelling option, according to me), and the whole thing is very rough for everyone.
If this process goes on long enough, the White Knight may burn out and become the Black Knight. The Black Knight is a more mercenary character—it has limited resources, so it has to watch out for itself, and it’s only allied with the group to the extent that the group’s goals match up with its own. It’s capable of teamwork and coordination, but it’s not zealous. It isn’t blinded by optimism or patriotism; it’s there to engage in mutually beneficial trade, while taking into account the realities of uncertainty and unreliability and miscommunication.
The Black Knight doesn’t like this whole frame in which doing the safe and conservative thing is judged as “defection.” It wants to know who this White Knight thinks he is, that he can just declare that it’s time to choose stag, without discussion or consideration of cost. If anyone’s defecting, it’s the White Knight, by going around getting mad at people for following local incentive gradients and doing the predictable thing.
But the Black Knight is also wrong, in that sometimes you really do have to be all-in for the thing to work. You can’t always sit back and choose the safe, calculated option—there are, sometimes, gains that can only be gotten if you have no exit strategy and leave everything you’ve got on the field.
I don’t have a solution for this particular dynamic, except for a general sense that shining more light on it (dignifying both sides, improving communication, being willing to be explicit, making it safe for both sides to be explicit) will probably help. I think that a “technique” which zeroes in on ensuring shared common-knowledge understanding of “this is what’s good in our subculture, this is what’s bad, this is when we need to fully commit, this is when we can do the minimum” is a promising candidate for defusing the whole cycle of mutual accusation and defensiveness.
(Circling with a capital “C” seems to be useful for coming at this problem sideways, whereas mission statements and manifestos and company handbooks seem to be partially-successful-but-high-cost methods of solving it directly.)

The key conceptual difference that I find helpful here is acknowledging that "Rabbit" / "Stag" are both positive choices, that bring about utility. "Defect" feels like it brings in connotations that aren't always accurate.

Saying that you're going to pay rent on time, and then not, is defecting.

But if someone shows up saying "hey let's all do Big Project X" and you're not that enthusiastic about Big Project X but you sort of nod noncommittally, and then it turns out they thought you were going to put 10 hours of work into it and you thought you were going to put in 1, and then they get mad at you... I think it's more useful to think of this as "choosing rabbit" than "defecting."

Likewise, it's "rabbit" if you say "nah, I just don't think Big Project X is important". Going about your own projects and not signing up for every person's crusade is a perfectly valid action.

Likewise, it's "rabbit" if you say "look, I realize we're in a bad equilibrium right now and it'd be better if we all switched to A New Norm. But right now the Norm is X, and unless you are actually sure that we have enough buy-in for The New Norm, I'm not going to start doing a costly thing that I don't think is even going to work."


A lightweight, but concrete example

At my office, we have Philosophy Fridays, where we try to get sync about important underlying philosophical and strategic concepts. What is our organization for? How does it connect to the big picture? What individual choices about particular site-features are going to bear on that big picture?

We generally agree that Philosophy Friday is important. But often, we seem to disagree a lot about the right way to go about it.

In a recent example: it often felt to me that our conversations were sort of meandering and inefficient. Meandering conversations that don't go anywhere is a stereotypical rationalist failure mode. I do it a lot by default myself. I wish that people would punish me when I'm steering into 'meandering mode'.

So at some point I said 'hey this seems kinda meandering.'

And it kinda meandered a bit more.

And I said, in a move designed to be somewhat socially punishing: "I don't really trust the conversation to go anywhere useful." And then I took out my laptop and mostly stopped paying attention.

And someone else on the team responded, eventually, with something like "I don't know how to fix the situation because you checked out a few minutes ago and I felt punished and wanted to respond but then you didn't give me space to."

"Hmm," I said. I don't remember exactly what happened next, but eventually he explained:

Meandering conversations were important to him, because it gave him space to actually think. I pointed to examples of meetings that I thought had gone well, that ended with google docs full of what I thought had been useful ideas and developments. And he said "those all seemed like examples of mediocre meetings to me – we had a lot of ideas, sure. But I didn't feel like I actually got to come to a real decision about anything important."

"Meandering" quality allowed a conversation to explore subtle nuances of things, to fully explore how a bunch of ideas would intersect. And this was necessary to eventually reach a firm conclusion, to leave behind the niggling doubts of "is this *really* the right path for the organization?" so that he could firmly commit to a longterm strategy.

We still debate the right way to conduct Philosophy Friday at the office. But now we have a slightly better frame for that debate, and awareness of the tradeoffs involved. We discuss ways to get the good elements of the "meandering" quality while still making sure to end with clear next-actions. And we discuss alternate modes of conversation we can intelligently shift between.

There's a time when I would have pre-emptively gotten really frustrated, and started rationalizing reasons why my teammate was willfully pursuing a bad conversational norm. Fortunately I had thought enough about this sort of problem that I noticed that I was failing into a failure mode, and shifted mindsets.

Rabbit in this case was "everyone just sort of pursues whatever conversational types seem best to them in an uncoordinated fashion", and Stag is "we deliberately choose and enforce particular conversational norms."

We haven't yet coordinated enough to really have a "stag" option we can coordinate around. But I expect that the conversational norms we eventually settle into will be better than if we had naively enforced either my or my teammate's preferred norms.


Takeaways

There seem like a couple important takeaways here, to me.

One is that, yes:

Sometimes stag hunts are worth it.

I'd like people in my social network to be aware that sometimes, it's really important for everyone to adopt a new norm, or for everyone to throw themselves 100% into something, or for a whole lot of person-hours to get thrown into a project.

When discussing whether to embark on a stag hunt, it's useful to have shorthand to communicate why you might ever want to put a lot of effort into a concerted, coordinated effort. And then you can discuss the tradeoffs seriously.

I have more to say about what sort of stag hunts seem do-able. But for this post I want to focus primarily on the fact that...

The schelling option is Rabbit

Some communities have established particular norms favoring 'stag'. But in modern, atomic, Western society you should probably not assume this as a default. If you want people to choose stag, you need to spend special effort building common knowledge [LW · GW] that Big Project X matters, and is worthwhile to pursue, and get everyone on board with it.

Corollary: Creating common knowledge is hard. If you haven't put in that work, you should assume Big Project X is going to fail, and/or that it will require a few people putting in herculean effort "above their fair share", which may not be sustainable for them.

This depends on whether effort is fungible. If you need 100 units of effort, you can make do with one person putting in 100 units of effort. If you need everyone to adopt a new norm that they haven't bought into, it just won't work.

If you are proposing what seems (to you) quite sensible, but nobody seems to agree...

...well, maybe people are being biased in some way, or motivated to avoid considering your proposed stag-hunt. People sure do seem biased about things, in general, even when they know about biases. So this may well be part of the issue.

But I think it's quite likely that you're dramatically underestimating the inferential distance – both the distance between their outlook and "why your proposed action is good", as well as the distance between your outlook and "why their current frame is weighing tradeoffs very differently than your current frame."

Much of the time, I feel like getting angry and frustrated... is something like "wasted motion" or "the wrong step in the dance."

Not entirely – anger and frustration are useful motivators. They help me notice that something about the status quo is wrong and needs fixing. But I think the specific flavor of frustration that stems from "people should be cooperating but aren't" is often, in some sense, actually wrong about reality. People are actually making reasonable decisions given the current landscape.

Anger and frustration help drive me to action, but often they come with a sort of tunnel vision. They lead me to dig in my heels, and get ready to fight – at a moment when what I really need is empathy and curiosity. I either need to figure out how to communicate better, to help someone understand why my plan is good. Or, I need to learn what tradeoffs I'm missing, which they can see more clearly than I.

My own strategies right now

In general, choose Rabbit.

Follow rabbit trails into Stag Country

Given a choice, seek out "Rabbit" actions that preferentially build option value for improved coordination later on.

(I'll hopefully have more to say about this in the future.)

Get curious about other people's frames

If a person and I have argued through the same set of points multiple times, each time expecting our points to be a solid knockdown of the other's argument... and if nobody has changed their mind...

Probably we are operating in two different frames. Communicating across frames is very hard, and beyond scope of this of this post to teach. But cultivating curiosity [LW · GW] and empathy are good first steps.

Occasionally run "Kickstarters for Stag Hunts." If people commit, hunt stag.

For example, the call-to-action in my Relationship Between the Village and Mission [LW · GW] post (where I asked people to contact me if they were serious about improving the Village) was designed to give me information about whether it's possible to coordinate on a staghunt to improve the Berkeley rationality village.

24 comments

Comments sorted by top scores.

comment by abramdemski · 2019-06-09T04:32:35.830Z · score: 27 (8 votes) · LW · GW

We should really be calling it Rabbit Hunt rather than Stag Hunt.

  • The schelling choice is rabbit. Calling it stag hunt makes the stag sound schelling.
  • The problem with stag hunt is that the schelling choice is rabbit. Saying of a situation "it's a stag hunt" generally means that the situation sucks because everyone is hunting rabbit. When everyone is hunting stag, you don't really bring it up. So, it would make way more sense if the phrase was "it's a rabbit hunt"!
  • Well, maybe you'd say "it's a rabbit hunt" when referring to the bad equilibrium you're seeing in practice, and "it's a stag hunt" when saying that a better equilibrium is a utopian dream.
  • So, yeah, calling the game "rabbit hunt" is a stag hunt.
I used to think a lot in terms of Prisoner's Dilemma, and "Cooperate"/"Defect." I'd see problems that could easily be solved if everyone just put a bit of effort in, which would benefit everyone. And people didn't put the effort in, and this felt like a frustrating, obvious coordination failure. Why do people defect so much?
Eventually Duncan shifted towards using Stag Hunt rather than Prisoner's Dilemma as the model here. If you haven't read it before, it's worth reading the description in full. If you're familiar you can skip to my current thoughts below.

In the book The Stag Hunt, Skyrms similarly says that lots of people use Prisoner's Dilemma to talk about social coordination, and he thinks people should often use Stag Hunt instead.

I think this is right. Most problems which initially seem like Prisoner's Dilemma are actually Stag Hunt, because there are potential enforcement mechanisms available. The problems discussed in Meditations on Moloch are mostly Stag Hunt problems, not Prisoner's Dilemma problems -- Scott even talks about enforcement, when he describes the dystopia where everyone has to kill anyone who doesn't enforce the terrible social norms (including the norm of enforcing).

This might initially sound like good news. Defection in Prisoner's Dilemma is an inevitable conclusion under common decision-theoretic assumptions. Trying to escape multipolar traps with exotic decision theories might seem hopeless. On the other hand, rabbit in Stag Hunt is not an inevitable conclusion, by any means.

Unfortunately, in reality, hunting stag is actually quite difficult. ("The schelling choice is Rabbit, not Stag... and that really sucks!")

Rabbit in this case was "everyone just sort of pursues whatever conversational types seem best to them in an uncoordinated fashion", and Stag is "we deliberately choose and enforce particular conversational norms."

This sounds a lot like Pavlov-style coordination vs Tit for Tat style coordination [LW · GW]. Both strategies can defeat Moloch in theory, but they have different pros and cons. TfT-style requires agreement on norms, whereas Pavlov-style doesn't. Pavlov-style can waste a lot of time flailing around before eventually coordinating. Pavlov is somewhat worse at punishing exploitative behavior, but less likely to lose a lot of utility due to feuds between parties who each think they've been wronged and must distribute justice.

When discussing whether to embark on a stag hunt, it's useful to have shorthand to communicate why you might ever want to put a lot of effort into a concerted, coordinated effort. And then you can discuss the tradeoffs seriously.
[...]
Much of the time, I feel like getting angry and frustrated... is something like "wasted motion" or "the wrong step in the dance."

Not really strongly contradicting you, but I remember Critch once outlined something like the following steps for getting out of bad equilibria. (This is almost definitely not the exact list of steps he gave; I think there were 3 instead of 4 -- but step #1 was definitely in there.)

1. Be the sort of person who can get frustrated at inefficiencies.

2. Observe the world a bunch. Get really curious about the ins and outs of the frustrating inefficiencies you notice; understand how the system works, and why the inefficiencies exist.

3. Make a detailed plan for a better equilibrium. Justify why it is better, and why it is worth the effort/resources to do this. Spend time talking to the interested parties to get feedback on this plan.

4. Finally, formally propose the plan for approval. This could mean submitting a grant proposal to a relevant funding organization, or putting something up for a vote, or other things. This is the step where you are really trying to step into the better equilibrium, which means getting credible backing for taking the step (perhaps a letter signed by a bunch of people, or a formal vote), and creating common knowledge between relevant parties (making sure everyone can trust that the new equilibrium is established). It can also mean some kind of official deliberation has to happen, depending on context (such as a vote, or some kind of due-diligence investigation, or an external audit, etc).

comment by philh · 2019-06-09T12:18:55.728Z · score: 2 (1 votes) · LW · GW

Most prob­lems which ini­tially seem like Pri­soner’s Dilemma are ac­tu­ally Stag Hunt, be­cause there are po­ten­tial en­force­ment mechanisms available.

I'm not sure I follow, can you elaborate?

Is the idea that everyone can attempt to enforce norms of "cooperate in the PD" (stag), or not enforce those norms (rabbit)? And if you have enough "stag" players to successfully "hunt a stag", then defecting in the PD becomes costly and rare, so the original PD dynamics mostly drop out?

If so, I kind of feel like I'd still model the second level game as a PD rather than a stag hunt? I'm not sure though, and before I chase that thread, I'll let you clarify whether that's actually what you meant.

comment by abramdemski · 2019-06-09T22:15:22.639Z · score: 15 (7 votes) · LW · GW

By "is a PD", I mean, there is a cooperative solution which is better than any Nash equilibrium. In some sense, the self-interest of the players is what prevents them from getting to the better solution.

By "is a SH", I mean, there is at least one good cooperative solution which is an equilibrium, but there are also other equilibria which are significantly worse. Some of the worse outcomes can be forced by unilateral action, but the better outcomes require coordinated action (and attempted-but-failed coordination is even worse than the bad solutions).

In iterated PD (with the right assumptions, eg appropriately high probabilities of the game continuing after each round), tit-for-tat is an equilibrium strategy which results in a pure-cooperation outcome. The remaining difficulty of the game is the difficulty of ending up in that equilibrium. There are many other equilibria which one could equally well end up in, including total mutual defection. In that sense, iteration can turn a PD into a SH.

Other modifications, such as commitment mechanisms or access to the other player's source code, can have similar effects.

comment by philh · 2019-06-10T22:02:51.424Z · score: 2 (1 votes) · LW · GW

Thanks, that makes sense.

Rambling:

In the specific case of iteration, I'm not sure that works so well for multiplayer games? It would depend on details, but e.g. if a player's only options are "cooperate" or "defect against everyone equally", then... mm, I guess "cooperate iff everyone else cooperated last round" is still stable, just a lot more fragile than with two players.

But you did say it's difficult, so I don't think I'm disagreeing with you. The PD-ness of it still feels more salient to me than the SH-ness, but I'm not sure that particularly means anything.

I think actually, to me the intuitive core of a PD is "players can capture value by destroying value on net". And I hadn't really thought about the core of SH prior to this post, but I think I was coming around to something like threshold effects; "players can try to capture value for themselves [it's not really important whether that's net positive or net negative]; but at a certain fairly specific point, it's strongly net negative". Under these intuitions, there's nothing stopping a game from being both PD and SH.

Not sure I'm going anywhere with this, and it feels kind of close to just arguing over definitions.

comment by Thrasymachus · 2019-06-08T16:12:44.028Z · score: 13 (6 votes) · LW · GW

It's perhaps worth noting that if you add in some chance of failure (e.g. even if everyone goes stag, there's a 5% chance of ending up -5, so Elliott might be risk-averse enough to decline even if they knew everyone else was going for sure), or some unevenness in allocation (e.g. maybe you can keep rabbits to yourself, or the stag-hunt-proposer gets more of the spoils), this further strengthens the suggested takeaways. People often aren't defecting/being insufficiently public spirited/heroic/cooperative if they aren't 'going to hunt stags with you', but are sceptical of the upside and/or more sensitive to the downsides.

One option (as you say) is to try and persuade them the value prop is better than they think. Another worth highlighting is whether there are mutually beneficial deals one can offer them to join in. If we adapt Duncan's stag hunt to have a 5% chance of failure even if everyone goes, there's some efficient risk-balancing option A-E can take (e.g. A-C pool together to offer some insurance to D-E if they go on a failed hunt with them).

[Minor: one of the downsides of 'choosing rabbit/stag' talk is it implies the people not 'joining in' agree with the proposer that they are turning down a (better-EV) 'stag' option.]


comment by Ratheka · 2019-06-08T16:33:29.148Z · score: 3 (2 votes) · LW · GW

Agree - talking things out, making everything as common knowledge as possible, and people who strongly value the harder path and who have resources committing some to fence off the worst cases of failure, seem to be necessary prerequisites to staghunting.

comment by Unnamed · 2019-06-09T00:43:45.445Z · score: 8 (4 votes) · LW · GW
And I said, in a move designed to be somewhat socially punishing: "I don't really trust the conversation to go anywhere useful." And then I took out my laptop and mostly stopped paying attention.

This 'social punishment' move seems problematic, in a way that isn't highlighted in the rest of the post.

One issue: What are you punishing them for? It seems like the punishment is intended to enforce the norm that you wanted the group to have, which is a different kind of move than enforcing a norm that is already established. Enforcing existing norms is generally prosocial, but it's more problematic if each person is trying to enforce the norms that he personally wishes the group to have.

A second thing worth highlighting is that this attempt at norm enforcement looked a lot like a norm violation (of norms against disengaging from a meeting). Sometimes "punishing others for violating norms" is a special case where it's appropriate to do something which would otherwise be a norm violation, but that's often a costly/risky way of doing things (especially when the norm you're enforcing isn't clearly established and so your actions are less legible).

comment by Raemon · 2019-06-09T00:56:17.176Z · score: 5 (2 votes) · LW · GW
Enforcing existing norms is generally prosocial, but it's more problematic if each person is trying to enforce the norms that he personally wishes the group to have.

Indeed. This is a major failure mode, important enough to deserve it's own post (or perhaps woven into some other posts – there's a lot I could say on this topic and I'm not sure how to crystallize it all). I try to catch myself when I do this behavior and am annoyed when I see others doing it and it was a major motivator for this post.

One problem is that people often have very different expectations about what norms had already been established, that people had implicitly bought into (by working at a given company, but being a 'rationalist' or whatnot).

comment by Benquo · 2019-06-13T15:33:20.214Z · score: 7 (4 votes) · LW · GW

It seems to me like the underlying situation is not that all stag hunt games are played separately with the same dynamics, but that there are lots of simultaneous stag hunt games being played, such that figuring out which stags to coordinate on first is a big problem, and in some games in some cultures the coordination point really is Stag, but Rabbit in the rest of them.

If I were to opportunistically steal stuff I like a lot from my casual acquaintances, people would come down hard to defend their stag hunt in that domain. In other games I've noticed variation. The people I know outside the SF Bay Area maintain households that can sustain extended guest-host interactions in ways that can gradually create rich, embodied social bonds among people who don't have hobbies or mass media consumption preferences in common, but no one puts effort into that in the Bay as far as I can tell, which seems like the Rabbit equilibrium there.

comment by Raemon · 2019-06-13T18:51:37.717Z · score: 5 (2 votes) · LW · GW

Yeah, I agree with this. Might edit the post to acknowledge this better.

comment by Decius · 2019-06-09T00:20:22.955Z · score: 7 (3 votes) · LW · GW

One error of the stag/rabbit hunt framing is that it makes it explicit that it's a coordination problem, not a values problem. To frame it differently would require that the stag and rabbit hunts not produce different utility numbers, but yield different resources or certainties of resource. If a rabbit hunt yields 3d2 rabbits hunted per hunter, but the stag hunt yields 1d2-1 stag hunted if all hunters work together and 0 if they don't, then even with a higher expected yield of meat and of hide from the stag hunt, for some people the rabbit hunt might yield higher expected utility, since the certainty of not starving is much more utility than an increase in the amount of hides.


In order to confidently assert that a Schelling point exists, one should have viewed the situation from everyone's point of view and applying their actual goals- NOT look at everyone's point of view and apply your goal, or the average goals, or the goals they think they have.

comment by Raemon · 2019-06-09T00:38:58.562Z · score: 5 (2 votes) · LW · GW

Thanks. I think I gained useful insights from this but found the wording a bit confusing. I assume you mean something like "The stag/rabbit frame assumes that it's a coordination, when sometimes [often?] it's a values disagreement. In real life people have different goals, so don't necessarily get the same utility from a given action. And in the literal rabbit/stag/hunters example, the various options don't actually produce the same utility since people get diminishing returns from meat."

...

"The Schelling Point is Rabbit" is (hopefully obviously) a bit of a simplification, and I agree that you'll want to actually look at everyone's goals in a given situation.

comment by shminux · 2019-06-08T03:27:20.327Z · score: 7 (3 votes) · LW · GW

A few points:

  • Yes, it's super frustrating when people ignore obviously (to you) good actions and or perform obviously (to you) bad actions. They do have their good (to them) reasons to do so, conscious or subconscious. So do you. They might be wrong, or you might be wrong, or something else might be going on. Unless you are all seeing and all knowing, how can you tell? If you are so much smarter than they are, why are you then working/volunteering there, oughtn't you be looking for a place where you can deal with your intellectual peers? And if you are not head and shoulders smarter, then how do you know that you are right and the action X is the right one, despite everyone else choosing action Y?
  • Following Scott's recent and ongoing sequence, consider that there might be culture and traditions that have evolved over a long time that compel people to do Y instead of X, even if they agree that X is better than Y if the argument for X > Y that you are advocating is considered in isolation.
  • As your example of a meandering conversation shows, people are very different and we all succumb to the typical mind fallacy. Intentionally paying attention to people likely not thinking the way you do, not incentivised by the same things you are, not emotionally connecting to other people and events the same way you do may clarify in your mind why other people act the way the do. If not, asking questions with genuine interest and curiosity and without expressing opinions can get you some ways there.

I have been and am in a situation that feels similar to yours. Here is one example of many. My direct supervisor had refused to implement a data retrieval system that would (in my opinion) greatly speed up and simplify analyzing and solving customer issues. It would be at most a week or two of work. That was 4 years ago, and I've been mentioning that we ought to do it ASAP every couple of weeks since then. He never authorized it, and instead asked if I am done whining. Meanwhile we are wasting resources many times over and likely losing customers and sales because of the field issues that could have been resolved quickly. I am not 100% sure why he is doing (or not doing) it, and it is pointless to ask, because his stated reasons would not be the real ones. I have some inkling of what is going on in his mind. In part it is probably chasing rabbits instead of stags, since he has more to lose from a scheduled slippage in a new project than from a slippage due to field support here and there, ostensibly outside his control. And maybe his strategy is the rational one, given the situation.

Let me add one more strategy to your list, though: dress stags as rabbits to get the buy in. This may sound disingenuous, but the reality is that with enough concerted effort a caught stag is later perceived as an unusually fat rabbit. People generally remember the payoff, not the effort, as long as things go smoothly.

comment by Raemon · 2019-06-08T04:43:40.645Z · score: 5 (2 votes) · LW · GW
dress stags as rabbits to get the buy in. This may sound disingenuous, but the reality is that with enough concerted effort a caught stag is later perceived as an unusually fat rabbit.

Can you give an example of what this might look like?

comment by shminux · 2019-06-08T05:49:36.608Z · score: 2 (1 votes) · LW · GW

We are all easily convinced of what we want to hear. Elliott wants to get ahead. Just scared a lot to take a chance. And perception is reality, literally. So if you help people see the benefits of cooperating (hunting a stag, taking a chance), and the drawbacks of sticking with rabbits (say, still being stuck in the muck), then their risk assessment changes and you may get them on your side. Or, alternatively, if Elliott has nothing to lose, he might go for a desperate heroic effort.

comment by Vladimir_Nesov · 2019-06-11T21:33:02.314Z · score: 6 (3 votes) · LW · GW

Utility is not a resource. In the usual expected utility setting, utility functions don't care about affine transformations (all decisions remain the same), for example decreasing all utilities by 1000 doesn't change anything, and there is no significance to utility being positive vs. negative. So requirements to have at least five units of utility in order to play a round of a game shouldn't make sense.

In this post, there is a resource whose utility could maybe possibly be given by the identity function. But in that case it could also be given by the "times two minus 87" utility function and lead to the same decisions. It's not really clear that utility is identity, since a 50/50 lottery between ending up with zero or ten units of the resource seems much worse than the certainty of obtaining five units. ("If you’re Elliott, [zero] is a super scary result to imagine.")

comment by Raemon · 2019-06-11T23:00:55.941Z · score: 4 (2 votes) · LW · GW

Not 100% sure I got all the points in this comment, but my sense is that the overall meta-point was that the framing (in particular in the quoted section?) of the Stag Hunt wasn't formalized properly (i.e. conflating resource with utility).

Do you have a suggestion for an improved wording, optimized for being accessible to non-technical people? (maybe just replacing "utility" with "resources" since I think that's a more accurate read on the particular way Duncan was framing it)

There's also a question that may be useful of when it's actually accurate for "real life stag hunts" to actually "cost resources." In the example I gave, there *is* a resource that'd need to be spent to achieve good conversational norms, but admittedly the stag hunt frame doesn't really map onto it that well (since it's not like different players had different amounts of resource, it was more like they had different beliefs about how valuable it was for the team to spend a particular resource)

comment by Vladimir_Nesov · 2019-06-11T23:12:24.623Z · score: 6 (3 votes) · LW · GW

Sure, any resource would do (like copper coins needed to pay for maintenance of equipment), or just generic "units of resources". My concern is that the term "utility" is used incorrectly in an otherwise excellent post that is tangentially related to the topic where its technical meaning matters, potentially propagating this popular misreading.

comment by Raemon · 2019-06-11T23:15:37.689Z · score: 5 (2 votes) · LW · GW

I replaced the term with 'resource' for now. (I think it's sort of important for it to showcase 'genericness')

comment by Vladimir_Nesov · 2019-06-11T23:21:28.503Z · score: 2 (1 votes) · LW · GW

Thanks!

comment by countingtoten · 2019-06-08T02:07:31.885Z · score: 5 (3 votes) · LW · GW

Meandering conversations were important to him, because it gave them space to actually think. I pointed to examples of meetings that I thought had gone well, that ended will google docs full of what I thought had been useful ideas and developments. And he said "those all seemed like examples of mediocre meetings to me – we had a lot of ideas, sure. But I didn't feel like I actually got to come to a real decision about anything important."

Interesting that you choose this as an example, since my immediate reaction to your opening was, "Hold Off On Proposing Solutions." More precisely, my reaction was that I recall Eliezer saying he recommended this before any other practical rule of rationality (to a specific mostly white male audience, anyway) and yet you didn't seem to have established that people agree with you on what the problem is.

It sounds like you got there eventually, assuming "the right path for the organization" is a meaningful category.

comment by Raemon · 2019-06-08T02:23:57.017Z · score: 5 (2 votes) · LW · GW

I think the disagreement was on a slightly different axis. It's not enough to hold off on proposing solutions. It's particular flavors and approaches to doing so.

One thing (which me-at-the-beginning-of-the-example would have liked), is that we do some sort of process like:

  • Everyone thinks independently for N minutes about what considerations are relevant, without proposing solutions
  • Everyone shares those thoughts, and discusses them a while
  • Everyone thinks independently for another N minutes about potential solutions in light of those considerations.
  • We discuss those thoughts
  • We eventually try to converge on a solution.

...with each section being somewhat time-boxed (You could fine-tune the time boxing, possibly repeating some steps, depending on how important the decision was and how much time you had)

But something about the overall process of time-boxing intrinsically got in the way of the type of thinking that this person felt they needed.

comment by philh · 2019-06-09T21:15:50.132Z · score: 2 (1 votes) · LW · GW

Meta: you have a few asterisks which I guess are just typos, but which caused me to go looking for footnotes that don't exist. "Philos­o­phy Fri­days*", "Fol­low rab­bit trails into Stag* Country", "Fol­low rab­bit trails that lead into *Stag-and-Rab­bit Coun­try".

comment by Raemon · 2019-06-09T22:48:28.771Z · score: 5 (2 votes) · LW · GW

Thanks. I got rid of them.

The two "Stag Country" ones were supposed to refer to each other (i.e. by "Stag Country" I meant "a place with lots of opportunities for both rabbit and stag hunting, a generally rich environment.")

Philosophy Friday* was going to link to a thing where I mentioned that we have a ridiculous injoke of spelling it different each time, most commonly Filosophee Φriday, but this didn't seem super worth people's attention and I decided not to make it part of the official post.