post by [deleted] · · ? · GW · 0 comments

This is a link post for

0 comments

Comments sorted by top scores.

comment by Qiaochu_Yuan · 2018-03-19T05:34:10.611Z · LW(p) · GW(p)

+A whole bunch for this. In general I think we need way more posts of the form "here is an actual thing in the world I tried and here is what I learned from it" (the Arbital post is another really good example) and I want to somehow incentivize them much harder but I'm not sure how.

comment by gwillen · 2018-03-19T07:03:47.704Z · LW(p) · GW(p)

I think I materially contributed to the creation of the Arbital post, by being an Arbital investor and specifically requesting one (as well as requesting the open-sourcing of the code). This may sound weird, but I think in that case "a good postmortem" is the best outcome I would feel entitled to hope for, and I was investing more with the goal of "learning what will happen if someone attempts this project" than making a monetary return.

I think it's a real tragedy that, in probably most major project failures, the principals are too drained by the fact of the failure to pass on knowledge that would be helpful to future attempts. That feels to me like wasting the most likely potential value to have come out of a big novel project in the first place. Most truly interesting projects will fail. I think it's good to go in hoping for success, but I also think it's irresponsible not to have a plan for how something can be learned from failure.

So, more positively: 1000 thank-yous to Duncan for taking the time to write this up.

EDIT: Oh, I guess I got distracted and failed to include sort of the punchline here: One way to encourage this is maybe to donate resources to a project, conditional (explicitly or implicitly) on being entitled to information about what happened if it fails?

comment by namespace (ingres) · 2018-03-19T06:44:38.374Z · LW(p) · GW(p)

I'm actually sitting on two posts like this. (One about a survey I conducted, another about an event I conducted in response to Scott's post on rationalists and bitcoin.)

I haven't written them because I have really high standards, some amount of anxiety about peer judgment, and very little slack to write them in.

comment by Duncan_Sabien · 2018-03-19T09:07:06.981Z · LW(p) · GW(p)

(You've probably already thought of this) For me, it's pretty low-cost to just set aside a good hour and do a stream-of-consciousness, and give myself permission to just not post it if it isn't good enough.

comment by ryan_b · 2018-03-22T14:32:36.679Z · LW(p) · GW(p)

I urge you to consider relaxing any standard apart from clarity, because I support unfinished ideas being a driver of value here. I judge good-faith contributions much more favorably than polished ones!

comment by Ben Pace (Benito) · 2018-03-19T06:31:11.344Z · LW(p) · GW(p)

+1

comment by Duncan_Sabien · 2018-03-19T09:17:58.858Z · LW(p) · GW(p)

(I think you mean +6, no? =P )

comment by Vaniver · 2018-03-19T03:42:43.374Z · LW(p) · GW(p)

[Context: I live in Dragon Army.]

I think it was good as a rationalist house, and something like 10-30% better than the rationalist house you would have had if the same group of people had just formed a house together without a clear structure. I also think it was basically a failure at the stated goals / house charter.

I'm also glad that we tried it, and would not advise past-me to not try it, and mostly agree with Duncan's advice to past-Duncan. (I also expect to stick around for the next stage of the house, which will likely be a regular rationalist house with some more structure.)

Some parts I disagree with:

I am more convinced than ever that little things like hand gestures, formal call-and-response patterns, jargon, little rituals, and explicitly outlined ways-of-interacting are a huge make-or-break for something like DA, and that I hit them lightly enough that we ended up in "break" territory.

I don't think that this was a major input into the outcome of the house, and am somewhat confused that you're more convinced of this now. It feels like there were ~3 obviously important things that didn't happen, ~6 things of uncertain importance that didn't happen, and the project didn't succeed; this state of affairs is basically equally likely in the world where those little things are make-or-break and the world where those little things are irrelevant. Where's the Bayesian evidence coming from?

Shifting gears a little—I'd tell past Duncan to start some kind of permanent record-keeping. We kept a LOT of data out on the walls, in public spaces, but it was always This Experiment or That Experiment—scattershot and inconsistent.

I don't think the versions of this that we actually tried worked very well. My model here is that the core feature is having a good metric that people are bought into, and am pessimistic about versions of this without that feature. (The past versions that we tried were proposals of metrics, which people mostly didn't buy into, and so after trying it out for three weeks or so it was abandoned and later replaced with a different proposal, which also didn't have the buy-in.)

Probably the best candidate for this was hours spent on house things. I likely would have done something like tracking this in a spreadsheet and then turning the spreadsheet contents into something physical that became a banner in the common room; my inner Duncan would have started out with tracking on a whiteboard instead of a spreadsheet, or had people put pins into a piece of foamboard that was the banner.

---

Some additional thoughts:

I don't know if there's any special advice I'd give myself about the Ghost separate from what's already been said—if there are fewer and clearer expectations better followed up on by me, and if there's a pre-move-in period where people can find out if they fit with those expectations better, and if there's a gentle off-ramp, then I suspect the Ghost is less of a problem.

I'm currently writing this from a laptop in the front room (for other readers: we have three 'common' spaces, of which the front room is the most lounge/desk-type) with two other Dragons also in the room. This is fairly abnormal; typically I'm on my desktop all day, and my desktop is in my room.

My guess is that when we decided the physical layout of the house, we should have tried harder to have people with shared or small sleeping space and then shared desk/office space. (With owned desks, such that I could leave things on it / have my monitor wall-mounted / etc.) If my computer lived in a room with the computers of other Dragons, I would see much more of them (and they would see much more of me) by default.

comment by Duncan_Sabien · 2018-03-19T04:50:05.585Z · LW(p) · GW(p)

I am not surprised by your disagreement; I think you are less ... psychoactive? ... on this axis than most-if-not-all of the other Dragons, and that more of this would not have meaningfully changed your experience. The increased confidence comes from nagging suspicions and intimations-of-doom from the near-beginning that kept turning out correct in various cases; it's less that Dragon Army provided new evidence of microculture's power, and more that it provided confirming evidence of the problems I intuited in its absence.

EDIT: I also have two widely-spaced and credible data points claiming that the causal arrow I've drawn between microculture and cohesion is a major part of why the Army does a lot of the things it does with jargon and rituals and hand gestures and weird tics about physical space. Like, not just that it's a convenient projection after the fact, but that that's why it is done and that its power has been tested and confirmed.

comment by ESRogs · 2018-03-19T09:18:20.753Z · LW(p) · GW(p)
I think you are less ... psychoactive

Nitpick: this would me that he causes fewer effects in others' minds, whereas what you meant to say is that he is less acted-on (by the hand signals, etc.), right?

At least that was my understanding of how we were using that term.

comment by Duncan_Sabien · 2018-03-19T18:57:18.701Z · LW(p) · GW(p)

Right, sorry. Like the misuses of the word "aversive" when people say "I'm aversive to this."

comment by vedrfolnir · 2018-03-22T00:46:26.527Z · LW(p) · GW(p)

Microculture and cohesion? Did you go to any particularly cohesive summer camps when you were young? If not, you might want to talk to someone who did.

I went to a few different CTY sites, and found that 1) my ranking of sites by quality matched up almost perfectly with the consensus, 2) these matched up almost perfectly with the extent to which the traditions (i.e. microculture) existed.

One thing that stands out to me is that I went to one site at which the traditions had almost completely died out. (Siena, if anyone out there remembers better than I do.) The story I heard was that the Catholic college didn't take too kindly to swarms of degenerate atheists watching Rocky Horror and so on on their campus and insisted that camp management do away with a lot of the traditions, the people who were the most into the traditions left for other sites in response, and with those few people gone, the traditions atrophied, and attendance at that site fell off a cliff. It shut down a few years after I went, and it deserved to go.

On the other hand, the site management was incompetent, so there's that too.

comment by Vaniver · 2018-03-22T19:27:59.229Z · LW(p) · GW(p)
Did you go to any particularly cohesive summer camps when you were young?

You asked this of Duncan, but I went to CTY twice; I remember my experience being dominated by the class I took, and didn't really rank the two sites against each other at the time. I seem to be recall both of them seeming tradition-heavy.

comment by gwillen · 2018-03-22T01:43:18.687Z · LW(p) · GW(p)

I would be curious to hear some examples of the jargon and rituals and hand gestures and weird tics about physical space. (Mostly it's just idle curiosity. I especially am not sure and thus curious what "weird tics about physical space" means. I'm also wondering how they compare to the examples in the fictional version of Dragon Army that you wrote up way back when.)

comment by Thrasymachus · 2018-03-19T14:59:56.430Z · LW(p) · GW(p)

Although I was sufficiently sceptical of this idea to doubt it was 'worth a shot' ex ante,(1) I was looking forward to being pleasantly surprised ex post. I'm sorry to hear it didn't turn out as well as hoped. This careful and candid write-up should definitely be included on the 'plus' side of the ledger for this project.

With the twin benefits of no skin in the game and hindsight. I'd like to float another account which may synthesize a large part of 'why it didn't work'.

Although I understand DAB wasn't meant to simply emulate 'military style' living, it did borrow quite a lot of that framing (hence the 'army' and 'barracks' bit of DAB). Yet these arrangements require a considerable difference in power and authority between the commander and their subordinates. I don't think DAB had this, and I suggest this proved its downfall.

[R]ight, as the world goes, is only in question between equals in power, while the strong do what they can and the weak suffer what they must
Thucydides - The Melian Dialogue

First, power. Having a compelling or else matters for maintaining discipline - even if people come in planning to obey, they may vacillate. In military contexts, accomodation at a barracks (or a boot camp) is a feature that can be unilaterally rescinded - if I argue back against the drill instructor, even if I'm in the right, they can make the credible threat "Either do as I say, or I kick you out". This threat has great 'downside-asymmetry': it's little skin off the instructor's nose if he boots me, but it means the end of my military career if they follow through. In consequence, the instructor has a lot more de facto bargaining power to make me do things I don't want to do. In negotiation theory, the drill sergeant's BATNA is way better than mine, and so good the only 'negotiation' they use are threats rapidly escalating to expulsion for any disobedience to their dictation.

I assume the legal 'fact on the ground' is that the participants of DAB were co-signatories on a lease, making significant financial contributions, with no mechanism for the designated 'commander' to kick people out unilaterally. Given this realpolitic, whatever 'command' the commander has in this situation is essentially roleplay. If a 'Red Knight' character decides not to play ball with the features of DAB that go beyond typical legal arrangements of house-sharing (e.g. "Nah, I'm not going to do this regular communal activity, I'm not going to do press-ups because you're telling me to, and I'm going to do my own thing instead."), the commander can't 'expel' them, but at most ask them to leave.

In fact the commander can only credibly threaten to expel subordinates from the social game of Dragon Army. If I'm a participant this might be a shame, but it might not be a great cost if I'm not feeling it anymore: I can probably (modulo some pretty dodgy social ostracising which other participants may not abide by) still interact with the remaining dragons in a normal housemate-y way.

Such a situation has pretty symmetrical downsides for both parties: if I'm the Red Knight, I probably would end up leaving (it wouldn't be fun being the 'odd man out', although the difficulty is shared by the dragons who have people in the house who aren't 'buying in', and the commander dealing with the ever-present example of the pretty mild consequence of what happens if others likewise disobey), but not until I'd lined up better accomodation, and I certainly wouldn't feel any obligation - given you're kicking me out rather than I'm choosing to leave - to pay rent after I go or help find others to 'fill my spot' to defray the proportionally increased cost of the lease (itself an incentive for other subordinates to pressure the commander to relent).

Instead of the boot camp case, where the much inferior BATNA of recruits gives them no negotiation position with their drill sergeant; in the DA case, the similarly equidistant BATNAs mean the commander and a dragon are at near-party (whatever affectations both maintain to the contrary). The stage is set for negotiation over discipline (even if coded or implicit), rather than dictation from the commander to the subordinate.

Authority (in the informal or moral sense, c.f. Weberian 'charismatic authority') can partly - though imperfectly - substitute for power (e.g. doctors and clergy tend to have more 'influence' than 'ordering people around'). What is notable that authority like this tends to be linked to some particular concrete achievement or track record, and is mainly constrained to a particular domain, with only mild 'bleed through' into general social status.

Given the nature of DAB, there is no convincing concrete thing a putative-commander could point to which legitimizes broad-raging authority over others beyond assent by the putative-subordinates this is an experiment they want to try. I suspect this results in pseudo-authority and obedience highly conditional on results: "I'm going to act as-if I am much lower status than this 'commander', and so kowtow towards them. But in terms of real status they're a peer to me, and I spot them this loan of social game status because I may benefit. If this doesn't happen, I'm less inclined to play along." The limited capital this supplies doesn't give the commander much to spend on pushing things through despite dissent: as the matters are between equals, 'right' becomes the question again.

I suggest this may explain the need for a better off-ramp (I note military or religious organisations which emphasize discipline and obedience generally provide accomodation etc. gratis, perhaps in part to provide the power initiates are there by invitation - I wonder whether reciprocal arrangements with another house substitute adequately); the reluctance to 'give enough orders' (better to maintain a smaller stake of authority than gamble a wider bid that could lose lots of face if flouted); the lack of a 'big enough stick' ("I'm happy to undergo token punishment - to lose pretend status - as a ritual for the sake of the game, but if the price of you 'disciplining' me costs real status, I'm out"); the need to avoid transgressing what less bought-in subordinates wanted (perhaps the result of a coded and implicit negotiation suggested earlier); and the shortfall between the ostensible standards asserted (set by aspiration), and those observed (set by the balance of power). One could imagine recognition of these features may have been key in transforming 'White Knights' into 'Black Knights'.

I'd guess, if this is the (forgive me) crux of the problem, whether it can really be fixed. Perhaps even more assiduous selection could do the trick, yet the base rate in the Bay Area (which isn't renowned for selecting people easygoing with authority) is not propitious. An alternative - but an exorbitantly expensive one - which would supply the commander with unquestioned power/authority would be if rent is free for subordinates: "You're here for free, so I can kick you if you don't play ball". I suspect a more modest 'Phase II' (maybe just social norms to do communal activity, with no pretended 'boss') might be the best realistic iteration.

-

1: It also seemed a bit risky from an 'abuse' (very broadly construed) perspective to me, although I stress this worry arose solely from the set-up rather than any adverse judgement of Duncan personally. Although understandable to be snarky to critics (e.g. one despicable pseudonymous commenter on the original 'pitch' recommended Duncan kill himself) I think the triumphalism of, "Although it didn't work, this 'not working' was not achieving all the good targets, rather than something horrible happening to someone - guess this wasn't a hotbed of abuse after all, and I'm not some abusive monster like you said! Eat humble pie! (but you won't, because you're irrational!)" is a bit misplaced.

The steelman of this is something like, "Although I don't think it blowing up really badly is more likely than not, the disjunction of bad outcomes along these lines is substantially higher than the base rate for a typical group-house, and this isn't outweighed across the scales by the expected benefit". On this view the observation the proposed risk wasn't realised is only mild discomfirmation for the hazard, although it rules strongly against outlandish 'disaster by default' views.

[Edit: Slight rewording, concision, and I remembered the theory I was gesturing at imprecisely]

comment by Vaniver · 2018-03-19T21:54:32.927Z · LW(p) · GW(p)
I assume the legal 'fact on the ground' is that the participants of DAB were co-signatories on a lease, making significant financial contributions, with no mechanism for the designated 'commander' to kick people out unilaterally.

This is approximately correct--not all of us were on the lease, and not all of us were making significant financial contributions. But someone who was on the lease and was making a significant financial contribution could have made it highly difficult to evict them, even if everyone else in the house wanted them gone. If it came to unilaterally rejecting people, I have some evidence that everyone would have deferred to Duncan's judgment.

I suspect that if there had been some pre-existing off-ramp, it would have been used voluntarily, and its existence would have enabled a safer discussion of "hey, is you being in the house actually win-win?".

I suspect a more modest 'Phase II' (maybe just social norms to do communal activity, with no pretended 'boss') might be the best realistic iteration.

This seems likely to be what Phase II will look like.

comment by plct · 2018-03-21T00:09:39.128Z · LW(p) · GW(p)

My understanding is that the way occupancy laws tend to work (and from what I've heard of california, although I do not live there) is that even if someone is not on the lease and not paying and unwelcome, your legal options for removing them are extremely limited and involve a substantial waiting period.

comment by adifferentface · 2018-03-20T17:16:01.564Z · LW(p) · GW(p)

This was roughly my ex ante analysis.

comment by Thrasymachus · 2018-03-20T21:16:01.417Z · LW(p) · GW(p)

Bravo - I didn't look at the initial discussion, or I would have linked your pretty accurate looking analysis (on re-skimming, Deluks also had points along similar lines). My ex ante scepticism was more a general sense than a precise pre-mortem I had in mind.

comment by sarahconstantin · 2018-03-19T05:02:45.633Z · LW(p) · GW(p)

I did update away from believing Dragon Army was dangerous after visiting there.

It was all people whom I’d expect to be able to hold their own against strong personalities. And they clearly didn’t have a strongly authoritarian culture by global or historical standards.

comment by Duncan_Sabien · 2018-03-19T05:08:32.574Z · LW(p) · GW(p)

Thanks.

I'll explicitly note that I felt extremely fairly treated by your criticisms at the time of the original post, and found constructive value in them. You are not in the reference class I was snarking at above.

I do wish more of the people who had strong concerns had taken steps such as investigating who else was involved; the list was known to within a few people even back when the charter first went up, but as far as I can recall no one publicly stated that they thought that might be crucial.

comment by Ben Pace (Benito) · 2018-03-19T03:22:36.216Z · LW(p) · GW(p)

I fee like I have a much better understanding of what the house hoped to be from this list of mistakes, rather than from the original vision document. I suppose how that's categorisation of concepts works.

comment by Zvi · 2018-03-19T17:08:57.283Z · LW(p) · GW(p)

Allow me to join the chorus thanking you for writing this up and avoiding grade inflation. There's a lot of great detail here and I look forward to digesting it more over time. Also kudos for trying the thing. Many of these lessons seem clearly valuable and correct once stated.

As with many (most?) things, the failures weren't the grand disasters some foresaw, and the biggest cost was time spent without accomplishing more - but everyone's still better off for having tried versus not. Doing things is hard, new and odd things are harder, there are tons of pieces to do and some important things won't look important slash whatever the ball is that gets dropped will end up looking obvious in hindsight.

My biggest takeaway from first reading is that it's vital to have good methods of exclusion and a willingness to walk away, either from the whole thing or from a particular relationship/member, if you want to do something most people wouldn't do. If something requires going above and beyond, do that, say goodbye to whoever isn't on board slash able to handle it, and if you then have not enough people, then you fail. Which is fine! But to accommodate (e.g. the Knights and Ghost) is to give yourself no chance.

comment by lifelonglearner · 2018-03-19T16:54:06.235Z · LW(p) · GW(p)

I'm curious about the specifics! If you were to brag, what were some of the most awesome things y'all got done? What were some ambitious projects that members undertook and succeeded?

comment by Raemon · 2018-03-19T03:58:14.182Z · LW(p) · GW(p)

Something about the Stag frame does make things click. In particular, it gets away from "cooperate/defect" and forces you to think about how much things are worth, which is a cleaner look at the actual game theory problem without confusing it with moralizing, and primes me to evaluate the sort of situations you'd run into here more appropriately.

All in all it sounds like, while I confess a little disappointment that you hadn't thought through some of the things that appear most-obvious-in-retrospect, the experience sounds like a pretty reasonable first pass at the experiment.

comment by weft · 2018-03-19T13:50:08.776Z · LW(p) · GW(p)

When I tried to actually think of examples where a Stag frame might apply in a housemates sort of situation, the payoff matrix never seemed to match the hypothetical.

Specifically, in every example I could think of that is relevant to a housemates situation, one person choosing stag is a cost to them, but still a benefit to everyone else. So perhaps in a group of 10 choosing stag costs 20, but even if everyone else defects, still results in 5 (to be split among the group... And of course a much higher payout if everyone cooperates). This means everyone else wants you to choose stag, whether they cooperate or not. In fact, their highest payout situation might be to be the sole defector.

The example was given of maintaining a clean bathroom sink. One person trying to do this is still better for everyone else. Their effort does result in a slightly cleaner bathroom.

Can someone give examples relevant to this situation where the payoff of choosing stag (when others defect) is actually zero?

comment by Vaniver · 2018-03-19T15:38:21.385Z · LW(p) · GW(p)
Can someone give examples relevant to this situation where the payoff of choosing stag (when others defect) is actually zero?

A simple example: suppose there's some house event, and you think that everyone has agreed that attendance and punctuality are very important (because of common knowledge / teamliness considerations), and in order to make it easier to schedule the event you choose to make it higher priority than other important events. So you have to cancel that other important thing, you show up on time, and then some other people are late or not there, and you have this sense of "well, if I had known this is how seriously everyone else would take it, I really wish I was at X instead."

The traditional game-theoretic stag hunt is a 2-player game, which makes the "everyone shows up" constraint much simpler to see that it's meaningful. The constraint on the payoffs is that Stag Alone < Rabbit < Stag Together and Stag Alone + Stag Together < 2*Rabbit. (The dominance is non-strict if it's <=, as in wikipedia's given example for Stag Hunt payoffs.) Also, as you're likely aware, 0 isn't special for utility functions, and so we should just be looking for things where doing it alone is worse than doing a different thing is worse than doing it together, and doing a different thing is closer to doing it together than alone.

If you have an exercise buddy and the two of you agree to show up at the gym at a particular time, then "stag" is showing up at the arranged time (if both of you are there, you work out together and get supported by each other; if only one of you is there, the relationship and desire to exercise erodes) and "rabbit" is choosing to exercise at a more convenient floating time (or not at all).

In the context of a house of 11 people, typically how this manifests is something like either 1) the event is mandatory for everyone, and everyone shows up, and each time someone isn't there it adds some cognitive cost of "yeah, we'll do it just like three times ago--wait, were you there for that?" whereas each time everyone is there it adds common knowledge of whatever happened, or 2) the event is optional for everyone, and almost no one shows up, and the event is implicitly compared to mandatory events where everyone shows up. (I think one of the problems that we ran into was that events run by individual people that were optional had an average attendance of something like 2, which made it difficult for people to want to host events, which perhaps further reduced attendance?)

comment by Richard_Ngo (ricraz) · 2018-03-20T10:25:03.369Z · LW(p) · GW(p)

A realistic example which matches this payoff matrix very well is when everyone is using the same means of transport, and so has to wait for the last to arrive before they can leave. A more speculative one: trying to create an environment of group emotional openness and vulnerability, where one person being sarcastic and prickly can ruin it for everyone.

comment by Duncan_Sabien · 2018-03-19T04:48:20.081Z · LW(p) · GW(p)

"Hadn't thought through" is true for only a half to a third; thought through but missed the important insight is more frequently the case (e.g. thought about the costs of doing a thing on the fly, measured them, decided they were low, was wrong).

comment by ahartell · 2018-03-20T01:39:02.199Z · LW(p) · GW(p)

Many of the Dragons who stepped into the role of the Ghost for a time did so softly and gradually, and it never felt like this level of absence was Notably Different from the previous level, in a paradox-of-the-heap sort of way. Set a bar, and set a gradient around that bar, and stay in contact.

As the person who fell most heavily into this role, the above resonates a lot. Below are some other thoughts on my experience.


I had the sense early on that I wasn't getting very much value out of group activities, and felt not very connected to the house. In this way I think "Black Knight"-style considerations were major contributors to my Ghost behavior. Competing commitments and general depression were also relevant. I didn't really feel like there was much the house could do to help me with that, but I don't know whether that's true. If it weren't for the Black Knight dynamic, I think I would have prioritized DA over other commitments, but depression may have been sufficient for me to end up as a Ghost anyway.

Not Getting Value Out of Group Activities

The things that the whole house can do (or even a large subset) are unlikely to be the on the capability frontier of the individual in an area of serious interest for that individual. Everyone needs to be able to do the thing, and there will be more variance in skill in areas that are a major focus of some but not all of the group. Programming ability is an example.

Because of something like this, DA group activities rarely felt like they were on a growth-edge that I cared about. In particular, group exercise usually felt costly with little benefit, and I never managed to get EE to be especially valuable for me. Social things like our weekly house dinner (a substantial fraction of Dragon Army hours) felt less fun or less growthy than the likely alternatives, but I probably put unusually low value on this kind of bonding.

Now when I imagine a group that is striving for excellence, it seems like there are two ways it can work:

1) The members share a common major project and can work together towards that goal. Here it makes sense for the group to ask for a high time commitment from its members, since time put towards the group directly advances a major goal of the individual.

2) The members have different goals. In this case it seems like the group should ask for a smaller time commitment. Members can mutually draw inspiration from each other and can coordinate when there is a shared goal, but generally the group should offer affordances, not impose requirements.

Counter-evidence: I think I would have gotten a lot of value out of covering the bases on dimensions I care about. Exercise was supposed to do this, and would do it along Duncan's version of the "capable well-rounded human" dimension. We discussed doing something like this for rationality skills, but we didn't follow through.

In this case, all members share a common goal of reaching a minimum bar in some area. Still, this can be boring for those who are already above the bar, and for me this sort of "catching up"/"covering the bases" is much less exciting than pushing forward on a main area of interest. (Which means group-time still ends up as less-fun-than-the-alternative by default.)

There were experiments intended to incentivize Dragons to do solo work on things they considered high priority, but my impression was that there was little encouragement/accountability/useful structure. Things I was originally excited about turned into homework I had to do for DA.

comment by Duncan_Sabien · 2018-03-20T02:55:26.205Z · LW(p) · GW(p)

... and the hope of the high-commitment, all-in, lock-yourself-in-a-box model was that people would chafe within the frame, without making moves to destroy the frame itself. i.e., once "stuck," things like Adom's dissatisfaction or sense of wasted time would propel him to push things in more productive directions, become more present with the group, make needs known, start making trades, etc., and then we'd iterate toward something better.

But this requires something that I and the house and Adom did not manage to synch up on, whether it's a tight enough box, or a sufficiently high level of ground safety and trust, or individual self-discipline, or what.

(above I'm trying to avoid assigning specific fault because I don't actually know which billiard balls bouncing around in the system led to the result we got; it's plausible that this specific case is all my fault or all Adom's fault or part of each or split between me and Adom and the rest of the house or influenced by outside factors etc. etc. etc.)

comment by Qiaochu_Yuan · 2018-03-20T21:35:41.534Z · LW(p) · GW(p)
But this requires something that I and the house and Adom did not manage to synch up on, whether it's a tight enough box, or a sufficiently high level of ground safety and trust, or individual self-discipline, or what.

There's a feeling I've been playing with over the last year whose True Name is something like "I would follow this man into battle."

I think many people nowadays have pretty screwy relationships with authority due to having spent a lot of time being forced to submit to authority figures that they didn't choose or respect (e.g. parents, random teachers in school), but that in principle a much healthier relationship is possible. Nearly the only example I can easily find and point to of this dynamic is fictional: it's the way Rider from Fate/Zero interacts with his army. When you respect / love / whatever your commander enough, things you are doing on their behalf should not feel in any way like impositions / homework, they should roughly feel like things you do for your romantic partners but with a different emotional tone.

I would wildly guess that the basic emotional alignment with the commander necessary to get something like Dragon Army to work is that 1) every member feels this way towards the commander, and 2) this fact is common knowledge. Unfortunately if you don't have a reference experience for what "I would follow this man into battle" feels like, you have no idea how to tell whether you're feeling it or not. I'm fairly confident I never felt this feeling before the last year, and although I now think I've felt a few weak versions of it I'd guess I haven't felt it at anywhere near full strength.

comment by ryan_b · 2018-03-22T16:27:18.703Z · LW(p) · GW(p)

A long time ago these sorts of relationships were much more common. In the history literature, one such pattern is called comitatus.

I imagine the modern literature on the subject mostly falls under cults of personality.

comment by vedrfolnir · 2018-03-22T17:51:44.975Z · LW(p) · GW(p)

Weber called it "charismatic authority", so there's another search term.

comment by John_Maxwell (John_Maxwell_IV) · 2018-03-21T03:20:02.241Z · LW(p) · GW(p)

I would guess there are some commitment & consistency effects involved here--once you've followed someone in to battle, you tend to identify with that.

comment by Duncan_Sabien · 2018-03-20T22:23:22.360Z · LW(p) · GW(p)

*person

comment by ahartell · 2018-03-20T03:41:07.391Z · LW(p) · GW(p)

Endorsed.

In addition to safety and contact, another dynamic was that I was generally not S1 expecting much value to come out of Dragon Army, so chafing more within the system seemed like pain, effort, and time spent for little expected gain.

Stag hunts, anyone?

Edit: Though, I will note that it can be hard to find the space between "I'm damaging the group by excluding my optimization power from the process" and "I'm being a Red Knight here and should be game for whatever the commander decides." It may seem like the obvious split is "expressive in discussion and game in the field" but discussion time is actually really valuable. So it seems like the actual thing is "be game until the cost to you becomes great enough that something needs to change". If you reduce the minimum size of misfit enough, then it becomes intractable to deal with everyone's needs. But then you have to figure out if a recent failure was a result of things being seriously broken or just a sign that you need to Be Better in some operationalized and "doable" way. When do you bring up the problem? It's hard.

comment by habryka (habryka4) · 2018-03-19T18:32:51.446Z · LW(p) · GW(p)

Great post, really like the structure and really really appreciate the transparency.

However, the sections about critics did leave me with a bad taste in my mouth. I am pretty sure you aren't including me in the set of uncharitable/wrongheaded critics, but I am not fully confident and I think this is kind of the problem. I imagine many of the people who had valid concerns and good criticism to feel like they are being dismissed, or at least assign a significant probability to it, and that this will have a significant chilling effect on good future criticism, which seems bad for everyone.

I think you definitely faced a lot of unfair criticism and verbal abuse, in a form that we would pretty heavily punish on the new LessWrong (man, that numbers guy), but I think it's better to try to just ignore those critics, or be very careful to not accidentally use broad enough brushstrokes to also paint many of the good critics in the same light. I know you tried to avoid this by putting in explicit disclaimers, but it's very hard from the inside to assess whether you think my criticism was constructed in good faith, and even on reflection I am not actually sure whether you think it was, and I expect others feel the same.

I want to emphasize again though that I am very happy you wrote this, and that overall this post has been quite valuable for me to read, and I am interested in chatting with you and others more about future iterations of Dragon Army and how to make it work. Keep up the good work.

comment by Duncan_Sabien · 2018-03-19T19:43:36.626Z · LW(p) · GW(p)

Re: criticizing critics ...

I think it's important to have public, common-knowledge deterrence of that sort of behavior. I think that part of what allowed it to flourish on LessWrong 1.0 is the absence of comments like my parenthetical, making it clear that that sort of thing is outside the Overton window. I claim that there were not enough defenders in the garden, and I think that a lot of LW is too unwilling to outgroup and disincentivize behavior that no one wants to see, because of something like not wanting to seem illiberal or close-minded or unwilling-to-rationally-consider-the-possibility-that-they're-wrong.

I recognize that this is a place where we disagree, and where indeed you could easily turn out to be more correct than me. But that parenthetical was carefully and deliberately constructed over the course of weeks, with me spending more than two full hours on its wording once you add up all the musing and tinkering and consideration of the consequences. It was Very Much On Purpose, and very much intended to have cultural effects.

I predict that the chilling effects on good criticism will be smaller-enough than the chilling effects on bad criticism that it's net worthwhile, in the end. In particular, I think the fact that you had difficulty telling whether I was referring to you or not is a feature, not a bug—as a result, you booted up your metacognition and judging/evaluatory algorithms, and users doing that as a matter of habit is one of my hopes/cruxes for LessWrong. We don't need crippling anxiety, but if I could push a button to make LW commenters 3% more self-consicous or 3% less self-conscious (3% more reflective and deliberate and self-evaluatory and afraid-of-their-own-biases-leaking-through or 3% less so) and it had to be across-the-board as opposed to distinguishing between different subsets of people ... I know which I'd choose.

(I imagine you don't want a long back-and-forth about this here; I would be willing to contribute to a discussion on this in Meta if you want.)

comment by Scott Alexander (scott-alexander) · 2018-03-20T20:59:08.951Z · LW(p) · GW(p)

The weatherman who predicts a 20% chance of rain on a sunny day isn't necessarily wrong. Even the weatherman who predicts 80% chance of rain on a sunny day isn't *necessarily* wrong.

If there's a norm of shaming critics who predict very bad outcomes, of the sort "20% chance this leads to disaster", then after shaming them the first four times their prediction fails to come true, they're not going to mention it the fifth time, and then nobody will be ready for the disaster.

I don't know exactly how to square this with the genuine beneficial effects of making people have skin in the game for their predictions, except maybe for everyone to be more formal about it and have institutions that manage this sort of thing in an iterated way using good math. That's why I'm glad you were willing to bet me about this, though I don't know how to solve the general case.

comment by Vaniver · 2018-03-20T21:45:30.167Z · LW(p) · GW(p)
If there's a norm of shaming critics who predict very bad outcomes

I think it is hugely important to point out that this is not the norm Duncan is operating under or proposing. I understand Duncan as saying "hey, remember those people who were nasty and uncharitable and disgusted by me and my plans? Their predictions failed to come true."

Like, quoting from you during the original discussion of the charter:

I would never participate in the linked concept and I think it will probably fail, maybe disastrously.
But I also have a (only partially endorsed) squick reaction to the comments against it. I guess I take it as more axiomatic than other people that if people want to try something weird, and are only harming themselves, that if you make fun of them for it, you’re a bully.

Of course, here, "the comments against it" isn't 'anyone who speaks about against the idea.' One can take jbeshir's comment as an example of someone pointing directly at the possibility of catastrophic abuse while maintaining good discourse and epistemic norms.

---

I note that I am generally not a fan of vaguebooking / making interventions on the abstract level instead of the object level, and if I were going to write a paragraph like the one in the OP I would have named names instead of making my claims high-context.

comment by Scott Alexander (scott-alexander) · 2018-03-20T23:19:52.567Z · LW(p) · GW(p)

Agreed that some people were awful, but I still think this problem applies.

If somebody says "There's a 80% chance of rain today, you idiot, and everyone who thinks otherwise deserves to die", then it's still not clear that a sunny day has proven them wrong. Or rather, they were always wrong to be a jerk, but a single run of the experiment doesn't do much to prove they were wronger than we already believed.

comment by Vaniver · 2018-03-21T00:11:46.755Z · LW(p) · GW(p)
Or rather, they were always wrong to be a jerk, but a single run of the experiment doesn't do much to prove they were wronger than we already believed.

To be clear, I agree with this. Furthermore, while I don't remember people giving probability distributions, I think it's fair to guess that critics as a whole (and likely even the irrational critics) put higher probability on the coarse description of what actually happened than Duncan or those of us that tried the experiment, and that makes an "I told you so!" about assigning lower probability to something that didn't happen hollow.

comment by Duncan_Sabien · 2018-03-20T23:29:59.204Z · LW(p) · GW(p)

I agree with this. Perhaps a better expression of the thing (if I had felt like it was the right spot in the piece to spend this many words) would've been:

they were systematically wrong then, in loudly espousing beliefs whose truth value was genuinely in question but for which they had insufficient justification, and wrong in terms of their belongingness within the culture of a group of people who want to call themselves "rationalists" and who care about making incremental progress toward actual truth, and I believe that the sacrifice of their specific, non-zero, non-useless data and perspective is well worth making to have the correct walls around our garden and weeding heuristics within it. And I see no reason for that to have changed in the intervening six months.

I suspect that coming out of the gate with that many words would've pattern-matched to whining, though, and that my specific parenthetical was still stronger once you take into account social reality.

I'm curious if you a) agree or disagree or something-else with the quote above, and b) agree or disagree or something-else with my prediction that the above would've garnered a worse response.

comment by Qiaochu_Yuan · 2018-03-20T21:16:23.486Z · LW(p) · GW(p)

The problem is absolutely not that people were predicting very bad outcomes. People on Tumblr were doing things like (I'm working from memory here) openly speculating about how incredibly evil and sick and twisted Duncan must be to even want to do anything like this, up to something like (again, working from memory here) talking about conspiring to take Duncan down somehow to prevent him from starting Dragon Army.

comment by Wei_Dai · 2018-03-20T21:51:25.627Z · LW(p) · GW(p)

As someone who didn't follow the original discussions either on Tumblr or LW, this was totally unclear from Duncan's parenthetical remark in the OP. So I think for the purpose of "common-knowledge deterrence of that sort of behavior" that section totally failed, since lots of people must have, like Scott and I, gotten wrong ideas about what kind of behavior Duncan wanted to deter.

comment by Duncan_Sabien · 2018-03-20T22:34:44.541Z · LW(p) · GW(p)

A part of my model here is that it's impossible from a social perspective for me to point these things out explicitly.

I can't describe the dynamic directly (my thinking contains some confusion) so I'll point out an analogous thing.

Alex and Bradley have had a breakup.

Alex is more destabilized than Bradley, by the breakup—to the point that Alex finds it impossible to occupy the same space as Bradley. This is not a claim about Bradley being bad or in the wrong or responsible (nor the opposite); it's just a brute fact about Alex's emotional state.

There's an event with open borders, or with broad-spectrum invites.

If Bradley goes, Alex cannot go. The same is not true in reverse; Bradley is comfortable shrugging and just handling it.

Alex absolutely cannot be the person to raise the question "Hey, maybe we have to do something about this situation, vis-a-vis inviting Alex or Bradley."

If Alex says that, this is inevitably interpreted as Alex doing something like taking hostages, or trying to divide up the universe and force people to take sides, or being emotionally immature and unreasonable. This is especially true because Bradley's right there, providing a contrasting example of "it's totally fine for us to coexist in the same room." Alex will look like The Source Of The Problem.

However, it's completely fine if someone else (Cameron) says "Hey, look—I think Alex needs more space, and we as a social group should figure out some way to create space for the processing and healing to happen. Maybe we invite Bradley to this one, but tell both Bradley and Alex that we'll invite Alex-and-only-Alex to the next one?"

Like, the social fabric is probably intelligent enough to handle the division without assigning fault or blame. But that requires third-party action. It can't come from Alex; it can maybe barely come from Bradley, if Bradley is particularly mature and savvy (but if Bradley doesn't feel like it, Bradley can just not).

In a similar sense, I tried real hard to point out the transgressions being made in the LW thread and on Tumblr, and this ended up backfiring on me, even though I claim with high confidence that if the objections had been raised by someone else, most LWers would've agreed with them.

So in this post, I drew the strongest possible line-in-the-sand that I could, and then primarily have sat back, rather than naming names or pulling quotes or trying to get specific. People hoping that I would get specific are (I claim) naively mispredicting what the results would have been, had I done so.

In this case, I owe the largest debts of gratitude to Qiaochu and Vaniver, for being the Cameron to my Alex-Bradley situation. They are saying things that it is unpossible for me to say, because of the way humans tend to pattern-match in such situations.

comment by philh · 2018-03-21T14:50:04.266Z · LW(p) · GW(p)

This concept feels to me like it deserves a top level post.

comment by Said Achmiz (SaidAchmiz) · 2018-03-20T23:29:30.923Z · LW(p) · GW(p)

This is indeed an important dynamic to discuss, so I’m glad you brought it up, but I think your judgment of the correct way to handle it is entirely wrong, and quite detrimental to the health of social groups and communities.

You say:

Alex will look like The Source Of The Problem.

But in fact Alex not only “will look like”, but in fact is, the source of the problem. In fact, the entirety of the problem is Alex’s emotional issues (and any consequences thereof, such as social discomfort inflicted upon third parties, conflicts that are generated due to Alex’s presence or behavior, etc.). There is no problem beyond or separately from that.

However, it’s completely fine if someone else (Charlie) says “Hey, look—I think Alex needs more space, and we as a social group should figure out some way to create space for the processing and healing to happen. Maybe we invite Bradley to this one, but tell both Bradley and Alex that we’ll invite Alex-and-only-Alex to the next one?”

This is only “fine” to the extent that the social group as a whole understands, and endorses, the fact that this “solution” constitutes taking Alex’s side.

Now, it is entirely possible that the social group does indeed understand and endorse this—that they are consciously taking Alex’s side. Maybe Alex is a good friend of many others in the group; whereas Bradley, while they like him well enough, is someone they only know through Alex—and thus they owe him a substantially lesser degree of loyalty than they do Alex. Such situations are common enough, and there is nothing inherently wrong with taking one person’s side over the other in such a case.

What is wrong is taking one person’s side, while pretending that you’re being impartial.

A truly impartial solution would look entirely different. It would look like this:

“Alex, Bradley, both of you are free to come, or not come, as you like. If one or both of you have emotional issues, or conflicts, or anything like that—work them out yourselves. Our [i.e., the group’s] relationship is with both of you separately and individually; we will thus continue to treat both of you equally and fairly, just as we treat every other one of us.”

As for Charlie… were I Bradley, I would interpret his comment as covert side-taking. (Once again: it may be justified, and not dishonorable at all. But it is absolutely not neutral.)

comment by ESRogs · 2018-03-21T00:06:49.628Z · LW(p) · GW(p)

I think the view I'd take is somewhere in between this view and the view that Duncan described.

If I'm sending out invites to a small dinner party, I'd just alternate between inviting Alex and Bradley.

However, if it's an open invite thing, it seems like the official policy should be that Alex and Bradley are both invited (assuming all parties are in good standing with the group in general), but if I happen to be close to Bradley I might privately suggest that they skip out on some events so that Alex can go, because that seems like the decent thing to do. (And if Bradley does skip, I would probably consider that closer to supererogatory rather than mandatory and award them social points for doing so.)

Similarly, if I'm close to Alex, I might nudge them towards doing whatever processing is necessary to allow them to coexist with Bradley, so that Bradley doesn't have to skip.

So I'm agreeing with you that official policy for open-invite things should be that both are invited. But I think I'm disagreeing about whether it's ever reasonable to expect Bradley to skip some events for the sake of Alex.

comment by Duncan_Sabien · 2018-03-20T23:36:57.690Z · LW(p) · GW(p)

I think your data set is impoverished. I think you could, in the space of five-minutes-by-the-clock, easily come up with multiple situations in which Alex is not at all the source of the problem, but rather Bradley, and I think you can also easily come up with multiple situations in which Alex and Bradley are equally to blame. In your response, you have focused only on cases in which it's Alex's fault, as if they represent the totality of possibility, which seems sloppy or dishonest or knee-jerk or something (a little). Your "truly impartial" solution is quite appropriate in the cases where fault is roughly equally shared, but miscalibrated in the former. Indeed, it can result in tacit social endorsement of abuse in rare-but-not-extremely-rare sorts of situations.

comment by Said Achmiz (SaidAchmiz) · 2018-03-21T00:01:09.833Z · LW(p) · GW(p)

Neither ‘blame’ nor ‘fault’ are anywhere in my comment.

And that’s the point: your perspective requires the group to assign blame, to adjudicate fault, to take sides. Mine does not. In my solution, the group treats what has transpired as something that’s between Alex and Bradley. The group takes no position on it. Alex now proclaims an inability to be in the same room as Bradley? Well, that’s unfortunate for Alex, but why should that affect the group’s relationship with Bradley? Alex has this problem; Alex will have to deal with it.

To treat the matter in any other way is to take sides.

You say:

I think you could, in the space of five-minutes-by-the-clock, easily come up with multiple situations in which Alex is not at all the source of the problem, but rather Bradley,

How can this be? By (your own) construction, Bradley is fine with things proceeding just as they always have, w.r.t. the group’s activities. Bradley makes no impositions; Bradley asks for no concessions; Bradley in fact neither does nor says anything unusual or unprecedented. If Alex were to act exactly as Bradley is acting, then the group might never even know that anything untoward had happened.

Once again: it may be right and proper for a group to take one person’s side in a conflict. (Such as in your ‘abuse’ example.) But it is dishonest, dishonorable, and ultimately corrosive to the social fabric, to take sides while pretending to be impartial.

comment by Duncan_Sabien · 2018-03-21T01:15:27.919Z · LW(p) · GW(p)

I think it's intellectually dishonest to write:

But in fact Alex not only “will look like”, but in fact is, the source of the problem. In fact, the entirety of the problem is Alex’s emotional issues

... and then say "Neither 'blame' nor 'fault' are anywhere in my comment." I smell a motte-and-bailey in that. There's obviously a difference between blame games and fault analysis (in the former, one assigns moral weight and docks karma from a person's holistic score; in the latter, one simply says "X caused Y"). But even in the dispassionate fault analysis sense, it strikes me as naive to claim that Alex's reaction is—in ALL cases that don't involve overt abuse—entirely a property of Alex and is entirely Alex's responsibility.

comment by Said Achmiz (SaidAchmiz) · 2018-03-21T01:56:48.600Z · LW(p) · GW(p)

I think you’re misunderstanding what I’m saying.

You seem to think that I’m claiming something like “it’s Alex’s fault that Alex feels this way”. But I’m claiming no such thing. In fact, basically the entirety of my point is that (in the “impartiality” scenario), as far as the group is concerned, it’s simply irrelevant why Alex feels this way. We can even go further and say: it’s irrelevant what Alex does or does not feel. Alex’s feelings are Alex’s business. The group is not interested in evaluating Alex’s feelings, in judging whether they are reasonable or unreasonable, in determine whether Alex is at fault for them or someone else is, etc. etc.

What I am saying is that Alex—specifically, Alex’s behavior (regardless of what feelings are or are not the cause of that behavior)—manifestly is the source of the problem for the group; that problem being, of course, “we now have to deal with one of our members refusing to be in the same room with another one of our members”.

As soon as you start asking why Alex feels this way, and whose fault is it that Alex feels this way, and whether it is reasonable for Alex to feel this way, etc., etc., you are committing yourself to some sort of side-taking. Here is what neutrality would look like:

Alex, to Group [i.e. spokesmember(s) thereof]: I can no longer stand to be in the same room as Bradley! Any event he’s invited to, I will not attend.

Group: Sounds like a bummer, man. Bradley’s invited to all public events, as you know (same as everyone else).

Alex: I have good reasons for feeling this way!

Group: Hey, that’s your own business. It’s not our place to evaluate your feelings, or judge whether they’re reasonable or not. Whether you come to things or not is, as always, your choice. You can attend, or not attend, for whatever reasons you like, or for no particular reason at all. You’re a free-willed adult—do what you think is best; you don’t owe us any explanations.

Alex: But it’s because…

Group (interrupting): No, really. It’s none of our business.

Alex: But if I have a really good reason for feeling this way, you’ll side with me, and stop inviting Bradley to things… right??

Group: Wrong.

Alex: Oh.


Separately:

But even in the dispassionate fault analysis sense, it strikes me as naive to claim that Alex’s reaction is—in ALL cases that don’t involve overt abuse—entirely a property of Alex and is entirely Alex’s responsibility.

Responsibility is one thing, but Alex’s reaction is obviously entirely a property of Alex. I am perplexed by the suggestion that it can be otherwise.

comment by FeepingCreature · 2018-03-21T18:49:27.819Z · LW(p) · GW(p)

Yeah but you can't derive fault from property, because by your own admission your model makes no claim of fault. At most you can say that Alex is the immediate causal source of the problem.

comment by Said Achmiz (SaidAchmiz) · 2018-03-21T21:11:30.094Z · LW(p) · GW(p)

Who ever claimed otherwise?

comment by bogus · 2018-03-20T22:50:23.891Z · LW(p) · GW(p)

Ah, but who will argue for the "Alex's" who were genuinely made uncomfortable by the proposed norms of Dragon's Army - perhaps to the point of disregarding even some good arguments and/or evidence in favor of it - and who are now being conflated with horribly abusive people as a direct result of this LW2 post? Social discomfort can be a two-way street.

comment by Duncan_Sabien · 2018-03-20T23:57:42.287Z · LW(p) · GW(p)

I disagree with your summary and frame, and so cannot really respond to your question.

comment by ESRogs · 2018-03-20T23:03:13.502Z · LW(p) · GW(p)

So this parenthetical-within-the-parenthetical didn't help, huh?

(here I am specifically not referring to those who pointed at valid failure modes and criticized the idea in constructive good faith, of whom there were many)

I guess one might not have had a clear picture what Duncan was counting as constructive criticism.

comment by chicken little · 2018-03-20T22:55:33.645Z · LW(p) · GW(p)

There were people like that, but there were also people who talked about the risks without sending ally-type signals of "but this is worth trying" or "on balance this is a good idea" who Duncan would then accuse of "bad faith" and "strawmanning," and would lump in with the people you're thinking.

comment by Duncan_Sabien · 2018-03-20T23:14:40.548Z · LW(p) · GW(p)

I request specific quotations rather than your personal summary. I acknowledge that I have not been providing specific quotations myself, and have been providing my summary; I acknowledge that I'm asking you to meet a standard I have yet to meet myself, and that it's entirely fair to ask me to meet it as well.

If you would like to proceed with both of us agreeing to the standard of "provide specific quotations with all of the relevant context, and taboo floating summaries and opinions," then I'll engage. Else, I'm going to take the fact that you created a brand-new account with a deliberately contrarian title as a signal that I should not-reply and should deal with you only through the moderation team.

comment by bogus · 2018-03-20T21:59:06.309Z · LW(p) · GW(p)

Thank you Qiaochu_Yuan for this much-needed clarification! It seems kinda important to address this sort of ambiguity well before you start casually talking about how 'some views' ought to be considered unacceptable for the sake of our community. (--Thus, I think both habryka and Duncan have some good points in the debate about what sort of criticism should be allowed here, and what standards there should be for the 'meta' level of "criticizing critics" as wrongheaded, uncharitable or whatever.)

comment by Qiaochu_Yuan · 2018-03-20T22:04:46.313Z · LW(p) · GW(p)
casually talking about how 'some views' ought to be considered unacceptable for the sake of our community!

I don't understand what this is referring to. This discussion was always about epistemic norms, not object-level positions, although I agree that this could have been made clearer. From the OP:

I myself was wrong to engage with them as if their beliefs had cruxes that would respond to things like argument and evidence.

To be clear, I'm also unhappy with the way Duncan wrote the snark paragraph, and I personally would have either omitted it or been more specific about what I thought was bad.

comment by bogus · 2018-03-20T22:23:01.304Z · LW(p) · GW(p)

I myself was wrong to engage with them as if their beliefs had cruxes that would respond to things like argument and evidence.

This is a fully-general-counterargument to any sort of involvement by people with even middling real-world concerns in LW2 - so if you mean to cite this remark approvingly as an example of how we should enforce our own standard of "perfectly rational" epistemic norms, I really have to oppose this. It is simply a fact about human psychology that "things like argument and evidence" are perhaps necessary but not sufficient to change people's minds about issues of morality or politics that they actually care about, in a deep sense! This is the whole reason why Bernard Crick developed his own list of political virtues which I cited earlier in this very comment section. We should be very careful about this, and not let non-central examples on the object level skew our thinking about these matters.

comment by habryka (habryka4) · 2018-03-19T20:03:18.060Z · LW(p) · GW(p)

Hmm, this seems like a pretty important topic to discuss in more depth, but this week is also a uniquely bad period for me to spend time on this because I need to get everything ready for the move to LessWrong.com and also have some other urgent and time-consuming commitments. This doesn't strike me as super urgent to resolve immediately, so I would leave this open for now and come back to it in a week when the big commitments I have are resolved, if that seems reasonable to you. I apologize for not being available as much as I would like for this.

comment by Duncan_Sabien · 2018-03-19T20:25:18.252Z · LW(p) · GW(p)

No worries!

comment by ozymandias · 2018-03-21T15:05:06.613Z · LW(p) · GW(p)

I think a problem with this strategy is that the Chicken Littles don't particularly like you or care about your opinion, and so the fact that you disapprove of their behavior has little to no deterrent effect.

comment by Thrasymachus · 2018-03-21T21:55:31.062Z · LW(p) · GW(p)

+1

It also risks a backfire effect. If one is in essence a troll happy to sneer at what rationalists do regardless of merit (e.g. "LOL, look at those losers trying to LARP enders game!"), seeing things like Duncan's snarky parenthetical remarks would just spur me on, as it implies I'm successfully 'getting a rise' out of the target of my abuse.

It seems responses to criticism that is unpleasant or uncharitable are best addressed specifically to the offending remarks (if they're on LW2, this seems like pointing out the fallacies/downvoting as appropriate), or just ignored. More broadcasted admonishment ("I know this doesn't apply to everyone, but there's this minority who said stupid things about this") seems unlikely to marshall a corps of people who will act together to defend conversational norms, but bickering and uncertainty about whether or not one is included in this 'bad fraction'.

(For similar reasons, I think amplifying rebuttals along the lines of, "You're misinterpreting me, and that people who don't interpret others correctly is one of the key problems with the LW community" seems apt to go poorly - few want to be painted as barbarians at the gates, and prompts those otherwise inclined to admit their mistake to instead double down or argue the case further.)

comment by gwillen · 2018-03-21T17:59:40.403Z · LW(p) · GW(p)

The "common knowledge" aspect implies e.g. other people not engaging with them, though. (And other people not looking down on Duncan for not engaging with them, although this is hard to measure, but still makes sense as a goal.)

comment by ozymandias · 2018-03-21T20:10:02.443Z · LW(p) · GW(p)

I mean, I suspect I *am* one of the Chicken Littles, and here you are, engaging with me. :)

I would make a bet at fairly generous odds that no rattumb person who offered a negative opinion of Dragon Army will face social consequences they consider significant from having a negative opinion of Dragon Army.

comment by Vaniver · 2018-03-21T21:44:11.685Z · LW(p) · GW(p)
I would make a bet at fairly generous odds that no rattumb person who offered a negative opinion of Dragon Army will face social consequences they consider significant from having a negative opinion of Dragon Army.

My model of social consequences is that most of them are silent; someone who could have helped you doesn't, someone asked for a recommendation about you gives a negative one, you aren't informed of events because you don't get invited to them. This makes it difficult to adjudicate such bets; as you'd have to have people coming forward with silent disapprovals, which would have to be revealed to the person in question to determine their significance.

comment by plct · 2018-03-21T00:32:17.743Z · LW(p) · GW(p)

I disagree - that sort of things makes me feel like it's less likely to be worth it to be put in substantial effort to do good, legible criticism whereas my younger self, more inclined to trolling (a habit I try to avoid, these days), smells blood in the water.

I don't know if you would have considered my criticism to be good or bad (although there's a substantial chance you never saw it), but good criticism is a lot of work and therefore probably much easier to chill.

comment by Kaj_Sotala · 2018-03-19T22:14:15.802Z · LW(p) · GW(p)

FWIF, the feeling I got from those parts of your post wasn't "this is Duncan clearly discouraging future behavior of this type, by indicating that it's outside the Overton window" but rather something like "if you were the kind of a person who attacked Duncan's post in a bad way before, then this kind of defiance is going to annoy that kind of a person even more, prompting even further attacks on Duncan in the future".

(this comment has been edited to retract a part of it)

comment by Duncan_Sabien · 2018-03-19T22:24:35.479Z · LW(p) · GW(p)

"prompting even further attacks" ... there's now more flexible moderation on LW2.0, so those posts can simply be erased, with a simple explanation as to why. I don't think we should fear-to-defy those people, I think we should defy them and then win.

(this comment has been edited to remove parts that are irrelevant now that an honest misunderstanding between me and Kaj has been pinpointed. Virtue points accrue to Kaj for embodying the norm of edits and updates and, through example, prompting me to do the same.)

comment by Kaj_Sotala · 2018-03-19T22:39:38.270Z · LW(p) · GW(p)
there's now more flexible moderation on LW2.0, so those posts can simply be erased, with a simple explanation as to why.

That may be, but didn't you say that your motivation was to get people to engage in less such behavior in the future, rather than causing more of it?

[EDIT: the second half of this comment has been retracted]

comment by Duncan_Sabien · 2018-03-19T22:45:48.316Z · LW(p) · GW(p)

(this comment has been significantly edited in response to the above retraction)

That's my primary motivation, yeah. I agree that there's a (significant) risk that I'm going about things the wrong way, and I do appreciate people weighing in on that side, but at the moment I'm trusting my intuitions and looking for concrete, gears-y, model-based arguments to update, because I am wary of the confusion of incentives and biases in the mix. The below HPMOR quotes feel really relevant to this question, to me:

"One answer is that you shouldn't ever use violence except to stop violence," Harry said. "You shouldn't risk anyone's life except to save even more lives. It sounds good when you say it like that. Only the problem is that if a police officer sees a burglar robbing a house, the police officer should try to stop the burglar, even though the burglar might fight back and someone might get hurt or even killed. Even if the burglar is only trying to steal jewelry, which is just a thing. Because if nobody so much as inconveniences burglars, there will be more burglars, and more burglars. And even if they only ever stole things each time, it would - the fabric of society -" Harry stopped. His thoughts weren't as ordered as they usually pretended to be, in this room. He should have been able to give some perfectly logical exposition in terms of game theory, should have at least been able to see it that way, but it was eluding him. Hawks and doves - "Don't you see, if evil people are willing to risk violence to get what they want, and good people always back down because violence is too terrible to risk, it's - it's not a good society to live in, Headmaster! Don't you realize what all this bullying is doing to Hogwarts, to Slytherin House most of all?"

...

"There was a Muggle once named Mohandas Gandhi," Harry said to the floor. "He thought the government of Muggle Britain shouldn't rule over his country. And he refused to fight. He convinced his whole country not to fight. Instead he told his people to walk up to the British soldiers and let themselves be struck down, without resisting, and when Britain couldn't stand doing that any more, we freed his country. I thought it was a very beautiful thing, when I read about it, I thought it was something higher than all the wars that anyone had ever fought with guns or swords. That they'd really done that, and that it had actually worked." Harry drew another breath. "Only then I found out that Gandhi told his people, during World War II, that if the Nazis invaded they should use nonviolent resistance against them, too. But the Nazis would've just shot everyone in sight. And maybe Winston Churchill always felt that there should've been a better way, some clever way to win without having to hurt anyone; but he never found it, and so he had to fight." Harry looked up at the Headmaster, who was staring at him. "Winston Churchill was the one who tried to convince the British government not to give Czechoslovakia to Hitler in exchange for a peace treaty, that they should fight right away -"
"I recognize the name, Harry," said Dumbledore. The old wizard's lips twitched upward. "Although honesty compels me to say that dear Winston was never one for pangs of conscience, even after a dozen shots of Firewhiskey."
"The point is," Harry said, after a brief pause to remember exactly who he was talking to, and fight down the suddenly returning sense that he was an ignorant child gone insane with audacity who had no right to be in this room and no right to question Albus Dumbledore about anything, "the point is, saying violence is evil isn't an answer. It doesn't say when to fight and when not to fight. It's a hard question and Gandhi refused to deal with it, and that's why I lost some of my respect for him."

My primary disagreement with the moderation of LW2.0 has consistently been "at what threshold is 'violence' in this metaphorical sense correct? When is too soon, versus too late?"

comment by Kaj_Sotala · 2018-03-19T22:55:58.189Z · LW(p) · GW(p)

I'm sorry. I re-read what you'd actually written and you're right.

I'd been reading some discussion about something else slightly earlier, and then the temporal proximity of that-other-thing caused my impression of that-other-thing and my impression of what-Duncan-wrote to get mixed up. And then I didn't re-read what you'd written before writing my comment, because I was intending to just briefly report on my initial impression rather than get into any detailed discussion, and I didn't want my report of my initial impression to get contaminated by a re-read - not realizing that it had been contaminated already.

Will edit my previous comments.

comment by Duncan_Sabien · 2018-03-20T00:13:24.972Z · LW(p) · GW(p)

I have tremendous respect for the fact that you're the type of person who could make a comment like the one above. That you a) sought out the source of confusion and conflict, b) took direct action to address it, and c) let me and others know what was going on.

I feel like saying something like "you get a million points," but it's more like, you already earned the points, out there in the territory, and me saying so is just writing it down on the map. I've edited my own comments as well, to remove the parts that are no longer relevant.

comment by Duncan_Sabien · 2018-03-20T02:50:15.931Z · LW(p) · GW(p)

One other HPMOR quote that feels relevant:

"Wolves, dogs, even chickens, fight for dominance among themselves. What I finally understood, from that clerk's mind, was that to him Lucius Malfoy had dominance, Lord Voldemort had dominance, and David Monroe and Albus Dumbledore did not. By taking the side of good, by professing to abide in the light, they had made themselves unthreatening. In Britain, Lucius Malfoy has dominance, for he can call in your loans, or send Ministry bureaucrats against your shop, or crucify you in the Daily Prophet, if you go openly against his will. And the most powerful wizard in the world has no dominance, because everyone knows that he is a hero out of stories, relentlessly self-effacing and too humble for vengeance ... In Hogwarts, Dumbledore does punish certain transgressions against his will, so he is feared to some degree, though the students still make free to mock him in more than whispers. Outside this castle, Dumbledore is sneered at; they began to call him mad, and he aped the part like a fool. Step into the role of a savior out of plays, and people see you as a slave to whose services they are entitled and whom it is their enjoyment to criticize; for it is the privilege of masters to sit back and call forth helpful corrections while the slaves labor ... I understood that day in the Ministry that by envying Dumbledore, I had shown myself as deluded as Dumbledore himself. I understood that I had been trying for the wrong place all along. You should know this to be true, boy, for you have made freer to speak ill of Dumbledore than you ever dared speak ill of me. Even in your own thoughts, I wager, for instinct runs deep. You knew that it might be to your cost to mock the strong and vengeful Professor Quirrell, but that there was no cost in disrespecting the weak and harmless Dumbledore."

... in at least some ways, it's important to have Quirrells and Lucius Malfoys around on the side of LW's culture, and not just David Monroes and Dumbledores.

comment by bogus · 2018-03-20T10:04:49.998Z · LW(p) · GW(p)

... in at least some ways, it's important to have Quirrells and Lucius Malfoys around on the side of LW's culture, and not just David Monroes and Dumbledores.

This is an interesting point - and, ISTM, a reason not to be too demanding about people coming to LW itself with a purely "good faith" attitude! To some extent, "bad faith" and even fights for dominance just come with the territory of Hobbesian social and political struggle - and if you care about "hav[ing] Quirrells and Lucius Malfoys" on our side, you're clearly making a point about politics as well, at least in the very broadest sense.

comment by Ben Pace (Benito) · 2018-03-19T22:54:28.076Z · LW(p) · GW(p)

We totally have private messaging now! Please err on the side of using it for norm disputes so that participants don’t feel the eyes of the world on them so much.

(Not that norm disputes shouldn’t be public - absolutely we should have public discussion of norms - just it can be great to get past tense parts of it sometimes.)

comment by bogus · 2018-03-19T21:33:11.336Z · LW(p) · GW(p)

I think it's important to have public, common-knowledge deterrence of that sort of behavior. I think that part of what allowed it to flourish on LessWrong 1.0 is the absence of comments like my parenthetical, making it clear that that sort of thing is outside the Overton window

There is a very important distinction to be made here, between criticism of an online project like LessWrong itself or even LessWrong 2, where the natural focus is on loosely coordinating useful work to be performed "IRL" (the 'think globally, act locally' strategy) and people 'criticizing' a real-world, physical community where people are naturally defending against shared threats of bodily harm, and striving to foster a nurturing 'ecology' or environment. To put it as pithily as possible, the somewhat uncomfortable reality is that, psychologically, a real-world, physical community is _always_ a "safe space", no matter whether it is explicitly connoted as such or not, or whether its members intend it as such or not; and yes, this "safe space" characterization comes with all the usual 'political' implications about the acceptability of criticism - except that these implications are actually a lot more cogent here than in your average social club on Tumblr or whatever! I do apologize for resorting to contentious "political" or even "tribal" language which seems to be frowned upon by the new moderation guidelines, but no "guidelines" or rules of politeness could possibly help us escape the obvious fact that doing something physically, in the real world always comes with very real political consequences which, as such, need to be addressed via an attitude that's properly mindful and inclined to basic values such as compromise and adaptability-- no matter what the context!

comment by Duncan_Sabien · 2018-03-19T21:52:40.529Z · LW(p) · GW(p)

By the time I got to the end of this, I realized I wasn't quite sure what you were trying to say.

Given that, I'm sort of shooting-in-the-dark, and may not actually be responding to your point ...

1) I don't think the difference between "talking about internet stuff" and "talking about stuff that's happening IRL" has any meaningful relevance when it comes to standards of discourse. I imagined you feeling sympathy for people's panic and tribalism and irrationality because they were looking at a real-life project with commensurately higher stakes; I don't feel such sympathy myself. I don't want to carve out an exception that says "intellectual dishonesty and immature discourse are okay if it's a situation where you really care about something important, tho."

2) I'm uncertain what you're pointing at with your references to political dynamics, except possibly the thing where people feel pressure to object or defend not only because of their own beliefs, but also because of second-order effects (wanting to be seen to object or defend, wanting to embolden other objectors or defenders, not wanting to be held accountable for failing to object or defend and thereby become vulnerable to accusations of tacit support).

I reiterate that there was a lot of excellent, productive, and useful discourse from people who were unambiguously opposed to the idea. There were people who raised cogent objections politely, with models to back those objections up and concrete suggestions for next actions to ameliorate them.

Then there were the rationalists-in-name-only (I can think of a few specific ones in the Charter thread, and a few on Tumblr whose rants were forwarded to me) whose whole way of approaching intellectual disagreement is fundamentally wrong and corrosive and should be consistently, firmly, and unambiguously rejected. It's like the thing where people say "we shouldn't be so tolerant that we endorse wildly toxic and intolerant ranting that itself threatens the norm of tolerance," only it's even worse in this case because what's at stake is whether our culture trends toward truthseeking at all.

------------------------------------------------------------------------------------

I would appreciate it if you're willing to restate what you wanted to convey, as I don't think I ended up catching it.

comment by bogus · 2018-03-19T22:21:02.477Z · LW(p) · GW(p)

I don't think the difference between "talking about internet stuff" and "talking about stuff that's happening IRL" has any meaningful relevance when it comes to standards of discourse.

Well, human psychology says that "stuff that's happening IRL" kinda has to play by its own rules. Online social clubs simply aren't treated the same by common sense 'etiquette' (much less common-sense morality!) as actual communities where people naturally have far higher stakes.

I don't want to carve out an exception that says "intellectual dishonesty and immature discourse are okay if it's a situation where you really care about something important,

If you think I'm advocating for willful dishonesty and immaturity, than you completely missed the point of what I was saying. Perhaps you are among those who intuitively associate "politics" or even "tribalism" with such vices (ignoring the obvious fact that a 'group house' itself is literally, inherently tribal - as in, defining a human "tribe"!) You may want to reference e.g. Bernard Crick's short work In Defense of Politics (often assigned in intro poli-sci courses as required reading!) for a very different POV indeed of what "political" even means. Far beyond the usual 'virtues of rationality', other virtues such as adaptability, compromise, creativity etc. --even humor! are inherently political.

The flip side of this, though, is that people will often disagree about what's intellectually dishonest or immature in the first place! Part of a productive attitude to contentious debate is an ability and inclination to look beyond these shallow attributions, to a more charitable view of even "very bad" arguments. Truth-seeking is OK and should always be a basic value, but it simply can't be any sort of all-encompassing goal, when we're dealing with real-world conmunities with all the attendant issues of those.

comment by Duncan_Sabien · 2018-03-19T22:29:27.734Z · LW(p) · GW(p)

I still don't follow what you're actually advocating, though, or what specific thing you're criticizing. Would you mind explaining to me like I'm five? Or, like, boiling it down into the kinds of short, concrete statements from which one could construct a symbolic logic argument?

comment by LawChan · 2018-03-20T05:07:13.610Z · LW(p) · GW(p)

I skimmed some of Crick and read some commentary on him, and Crick seems to take the Hobbesian "politics as a necessary compromise" viewpoint. (I wasn't convinced by his definition of the word politics, which seemed not to point at what I would point at as politics.)

My best guess: I think they're arguing not that immature discourse is okay, but that we need to be more polite toward people's views in general for political reasons, as long as the people are acting somewhat in good faith (I suspect they think that you're not being sufficiently polite toward those you're trying to throw out of the overton window). As a result, we need to engage less in harsh criticism when it might be seen as threatening.

That being said, I also suspect that Duncan would agree that we need to be charitable. I suspect the actual disagreement is whether the behavior of the critics Duncan is replying to are actually the sort of behavior we want/need to accept in our community.

(Personally, I think we need to be more willing to do real-life experiments, even if they risk going somewhat wrong. And I think some of the tumblr criticism definitely fell out of what I would want in the overton window. So I'm okay with Duncan's paranthetical, though it would have been nicer if it was more explicit who it was responding to.)

comment by bogus · 2018-03-20T08:56:27.594Z · LW(p) · GW(p)

I suspect they think that you're not being sufficiently polite toward those you're trying to throw out of the overton window

Actually, what I would say here is that "politeness" itself (and that's actually a pretty misleading term since we're dealing with fairly important issues of morality and ethics, not just shallow etiquette-- but whatever, let's go with it) entails that we should seek a clear understanding of what attitudes we're throwing out of the Overton window, and why, or out of what sort of specific concerns. There's nothing wrong whatsoever with considering "harsh criticism [that] might be seen as threatening" as being outside the Overton window, but whereas this makes a lot of sense when dealing with real-world based efforts like the Dragon Army group, or the various "rationalist Baugruppes" that seem to be springing up in some places, it feels quite silly to let the same attitude infect our response to "criticism" of Less Wrong as an online site, or of LessWrong 2 for that matter, or even of the "rationalist" community not as an actual community that might be physically manifested in some place, but as a general shared mindset.

When we say that "the behavior of the critics Duncan is replying to are [not] the sort of behavior we want/need to accept in our community", what do we actually mean by "behavior" and "community" here? Are we actually pointing out the real-world concerns inherent in "criticizing" an effort like Dragon Army in a harsh, unpolite and perhaps even threatening (if perhaps only in a political sense, such as by 'threatening' a loss of valued real-world allies!) way? Or are we using these terms in a metaphorical sense that could in some sense encompass everything we might "do" on the Internet as folks with a rationalist mindset? I see the very fact that it's not really "explicit who (or what) [we're] responding to" as a problem that needs to be addressed in some way, at least wrt. its broadest plausible implications-- even though I definitely understand the political benefits of understating such things!

comment by Qiaochu_Yuan · 2018-03-19T19:56:24.446Z · LW(p) · GW(p)

Oli, I'm not sure if you saw the Tumblr criticism, but it was really bad, in some ways even worse than the numbers guy.

comment by habryka (habryka4) · 2018-03-19T20:36:19.484Z · LW(p) · GW(p)

Ah, I think I now remember some of that stuff, though I think I only skimmed a small subset of it. What I remember did seem quite bad. I did not think of those when writing the above comment (though I don't think it would have changed the general direction of my concerns, though it does change the magnitude of my worries).

comment by Duncan_Sabien · 2018-03-19T20:01:22.313Z · LW(p) · GW(p)

Including some from rationality community members with relatively high social standing.

comment by Zvi · 2018-03-20T13:07:58.902Z · LW(p) · GW(p)

I think this falls under the category 'risks one must be willing to take, and which are much less bad than they look.' If it turns out he was thinking of you, him saying this in writing doesn't make things any worse. Damage, such as it is, is already done.

Yes, there's the possible downside of 'only the people he's not actually talking about feel bad about him saying it' but a little common knowledge-enabled smackdown seems necessary and helpful, and Duncan has a right to be upset and frustrated by the reception.

comment by Kaj_Sotala · 2018-03-20T14:25:59.545Z · LW(p) · GW(p)
If it turns out he was thinking of you, him saying this in writing doesn't make things any worse. Damage, such as it is, is already done.

This seems to imply that if person A thinks bad things of person B and says this out loud, then the only effect is that person B becomes aware of person A thinking bad things of them. But that only works if it's a private conversation: person C finding out about this may make them also like B less, or make them like A less, or any number of other consequences.

comment by habryka (habryka4) · 2018-03-20T17:48:18.199Z · LW(p) · GW(p)

Yep, the chilling effect comes from the public ridicule, not Duncan’s individual judgement.

comment by Zvi · 2018-03-20T19:06:40.792Z · LW(p) · GW(p)

My presumption was that person A thinks bad things about class of people D, which B may or may not belong to and is worried that B belongs to, but when others think of D they don't think of B, so C's opinion of B seems unlikely to change. If people assume B is in D, then that would be different (although likely still far less bad than it would feel like it was).

comment by Kaj_Sotala · 2018-03-20T19:35:35.350Z · LW(p) · GW(p)

There seems to be a common thing where statements about a class of people D, will associate person B with class D by re-centering the category of D towards including B, even if it's obvious that the original statement doesn't refer to B. This seems like the kind of a case where that effect could plausibly apply (here, in case it's not clear, B is a reasonable critic and D is the class of unreasonable critics).

comment by DanielFilan · 2018-03-19T21:09:00.591Z · LW(p) · GW(p)

As another resident of Dragon Army, the coming end of the experiment and this post have made me interested in looking over the commentary on the original Dragon Army Charter post. Out of it all, it seemed to me that this comment by a_different_face was probably the best prediction for how things panned out, despite not being super on-the-mark. Kudos!

comment by adifferentface · 2018-03-20T16:52:54.834Z · LW(p) · GW(p)

That's me! Username difference because I can't register the handle I used there over here at the moment.

Thanks for the kudos. I do have another handle, though I'm going to continue not to associate these two.

I also think I got things about right, but I think anyone else taking an outside view would've expected roughly the same thing. I'm now noticing that in fact this post doesn't give me enough information to assess how accurate my specific predictions were: this list of advice is suggestive but mostly not actually descriptive of what outcomes were not achieved as much as hoped, and how. I get the impression it's mostly just a sort of failure to coalesce, which isn't quite the outcome I expected but is not that far off from "someone will [...] fail to keep up with their chores".

comment by Thrasymachus · 2018-03-20T21:34:15.429Z · LW(p) · GW(p)
I also think I got things about right, but I think anyone else taking an outside view would've expected roughly the same thing.

I think you might be doing yourself a disservice. I took the majority of contemporary critcism was more directed towards (in caricature) 'this is going to turn into a nasty cult!' than (what I took your key insight to be) 'it will peter out because the commander won't actually have the required authority'.

So perhaps the typical 'anyone else' would have alighted on the wrong outside view, or at least the wrong question to apply it to ('How likely would a group of rationalists end up sustaining a 'dictator' structure?', rather than 'Conditional on having this structure, what would happen next?')

comment by habryka (habryka4) · 2018-03-20T18:41:11.600Z · LW(p) · GW(p)

Meta note: We ported over the old LW passwords, so you can probably login again with that account. Let me know if it doesn't work, so I can debug it.

comment by adifferentface · 2018-03-20T21:08:03.133Z · LW(p) · GW(p)

It doesn't work. It just gives "unknown error". Trying to register the account gave "internal server error".

comment by Duncan_Sabien · 2018-03-19T21:54:41.102Z · LW(p) · GW(p)

I second the idea of kudos. I don't know if a_different_face is around here, but I've assigned them a correspondingly higher weight in my aggregating-public-opinion function, on future questions.

comment by adifferentface · 2018-03-22T13:42:28.554Z · LW(p) · GW(p)

Given this comment, I guess I should actually bother to write up my opinion on the callout aspect of your post, which is this:

The callout aspect of your post is bad, and with moderately high confidence will work against your explicitly stated goals.

I basically endorse habryka's comment here and Kaj_Sotala's here, but to make specific predictions:

  • This will not make most people significantly more reflective.
  • This kind of callout will not discourage trolls, that is, people who are being intentionally callous and who see themselves as mocking you and/or rationalists in general. I expect instead it will encourage them.
  • This may discourage non-troll people who have a viscerally negative reaction which they can't put into words (or lack the energy to), and it may not. I claim it is not desirable to discourage such people if you can instead cultivate the ability to draw what evidence there is from them and not feel attacked; these reactions have signal even if they are also partly noise.
  • This will discourage sensitive people from engaging with you or the community in general, where here "sensitive" is the kind which is adjacent but not synonymous to "shy" and not the kind which is adjacent to "will lash out in response to perceived attacks" (though both share an excessively high prior on something being an attack). I claim this because for many years I was such a person myself, though I am not now, and this kind of callout would have discouraged that earlier me even if I had not commented on the original post. This is especially true if I had felt negatively or had reservations about your original post. That this is how you respond to critics would mean you would feel not safe to engage with or at least not worth engaging with, to many people. Yes, I know that you intended to be careful to only scorn a subset of your critics, but a.) that doesn't actually matter to System 1 feelings of safeness and b.) you weren't actually as careful as you seem to think: early in the post you say you were wrong to act as though "a particular subset... would respond to things like argument and evidence"; later you complain about "some of the Chicken Littles [who] were clamoring for an off-ramp"; there is an easy interpretation where you mean these to be the same set, that is, anyone who thought this was a bad idea and wanted an off-ramp was an unproductive non-evidence-responding person, despite your inner parenthetical (which in any case only would exclude people who had a concrete "valid" failure mode to point at, and not just "this is a bad idea and should have an off-ramp"). It does not matter whether this is strictly logically entailed by the words you wrote: there is more to communication than bare denotation. I think you are mistaken in your contention that merely encouraging attentiveness to words-as-written is sufficient to avoid this.

I think that last prediction is important and the outcome it describes bad, so I want to point at another thing which might make it more convincing: I know a fair number of CFAR alums and staff, and they are among the bottom decile of sensitive-in-this-sense people in the world, as are many but not all rationalists in general. You spend a lot of time around such people. I think you may have forgotten, or failed to learn, what more sensitive people need in conversations in order to be able to engage productively. I hope you have not forgotten or failed to learn that such people may be worth having around.

I agree that LW, and Our Kind in general, often has the problem of not sufficiently discouraging bad actors. But beware the politician's syllogism.

Finally: your behavior here reads very precisely as lashing out in response to attacks and then rationalizing that lashing out as being justified - even if the lashing out might in addition catch up some people who, to a disinterested observer, were not really attacking you. This is normal. Because you were in fact attacked, in some cases quite viciously, and because responding to attacks with attacks is a deeply ingrained part of many humans' psychology, you should have an extremely high prior that this is what you are doing. But it is bad. This is the easiest form of rationalization in the world. I do the same, and have spent a lot of time trying to catch myself doing this and stop, with some but not total success. Spending several hours trying to craft the right response doesn't mean you weren't doing this; I would say it is evidence to the contrary, because nothing focuses the attention like crafting a counterattack. I think if you did goal-factoring here, and have spent enough time noticing when you are rationalizing, you would end up concluding this was not the right way to achieve your actual endorsed goals.

I think you should apologize for the snark at the beginning of the essay and the footnote about Chicken Littles, and retract them, without putting any additional attacks in your apology. You should link to specific comments which you think were bad behaviour which you would like the community to discourage and with which you will try to avoid engaging in the future, to make it clear to everyone else that you did not intend to attack them in a way which merely asserting that you didn't intend to attack "those who pointed at valid failure modes" really does not accomplish. And then you should get on with your life.

comment by Duncan_Sabien · 2018-03-22T20:27:35.527Z · LW(p) · GW(p)

I disagree with the heavy implicit assumption running throughout your post that you know better than I do what's going on inside my head, that you have put more thought into this than I, and that you understand people in general better than I. I also note that you've made your model of me essentially unfalsifiable—you assert that X must be true of me because it's so normal and representative of How Humans Work, and you leave no path for me to get in contact with your uncertainty and demonstrate that you're wrong.

(These are rationality sins in my opinion.)

I temper that disagreement with the fact that I am crediting you with predictive power these days, but I notice that you and others are not writing and did not bother to write hundreds of words to chill or disincentivize or disendorse behavior that was an order of magnitude worse, and so I find the impassioned call for correction hollow.

If you find my one tiny parenthetical worth this entire long response, but none of the actual abuse that I received in the other thread and on Tumblr worth objecting to directly, then I cannot help but object to your priorities. You've put substantially more effort into a combination of reprimand and trying-to-change-Duncan's-models than you have into decrying calls for my suicide.

comment by adifferentface · 2018-03-22T23:28:22.636Z · LW(p) · GW(p)

The majority of my post consisted of specific empirical predictions which were not about the contents of your head. Whether I am mistaken about why you are doing what you're doing changes nothing about what the effects of your behaviour are likely to be. To the extent it was about the contents of your head it was addressing a conflict I perceived between the likely outcomes of your behavior and the goals you have elsewhere explicitly given for that behaviour. I concede that if in fact you intend the consequences I predict above then there is no point in my having written them down. Otherwise, I disagree that this is something I ought not do.

As to your complaints about who I choose to try to reason with: I don't expect to be able to reason with trolls, and don't expect the community to need me to*. Moreover I do not care to reason with them: I care to have them leave. Long and careful arguments are not what seems best to accomplish that goal. You, however, are a largely reasonable person I largely respect, and one who I would not like to drive out of the community, so when I see you doing something I think is a mistake, it might seem worth my time to engage with you and try to convince you not to. I am much more interested in engaging with my friends than my enemies, even if my friends are already better people than my enemies.

In any case, this is not a contest of who is a better person. Perhaps my priorities are indeed misaligned. This does not mean my assessment of the likely consequences of your actions is wrong.

*though as it happens I spent much of my next several comments on the original post saying things to that one troll like "I would hope the community would shun you" and "I still would not want you, personally, in any community I'm part of, because your behavior is bad." I deny that it is necessary that I have done this as a precondition for being worth listening to, but in fact I did. In the interests of keeping my handles separate I will not comment on whether or how I responded to anything on tumblr or elsewhere.

comment by Qiaochu_Yuan · 2018-03-22T23:05:46.990Z · LW(p) · GW(p)

Yeah, I really don't want to do this personally, but it may be time to pull out the big guns and actually find direct quotes from the Tumblrites. I think people who didn't see the Tumblr commentary are still underestimating how bad it was.

comment by adifferentface · 2018-03-22T23:38:10.602Z · LW(p) · GW(p)

I have seen a great deal of the tumblr commentary, and commentary elsewhere, and had seen it before writing my comment. I agree much of it was extremely vicious, and perhaps there is worse than the "he should kill himself" one which I have not seen. This changes nothing about the likely consequences of Duncan's snark. It makes that snark more forgivable, certainly; I don't think Duncan is a bad person for responding to attacks with moderate snark. But I do nevertheless think that said snark is harmful, and since Duncan is a person who I respect, I think it worth my time to suggest this to him.

comment by Benquo · 2018-03-22T23:24:26.863Z · LW(p) · GW(p)

How would that help resolve this particular conflict?

comment by DanielFilan · 2018-03-19T23:05:27.464Z · LW(p) · GW(p)

Unfortunately, they seem to only have comments on the Dragon Army Charter Draft post.

comment by handoflixue · 2018-03-22T07:24:37.741Z · LW(p) · GW(p)
There was a particular subset of LessWrong and Tumblr that objected rather ... stridently ... to even considering something like Dragon Army

Well, I feel called out :)

So, first off: Success should count for a lot and I have updated on how reliable and trust-worthy you are. Part of this is that you now have a reputation to me, whereas before you were just Anonymous Internet Dude.

I'm not going to be as loud about "being wrong" because success does not mean I was wrong about there *being* a risk, merely that you successfully navigated it. I do think drawing attentions to certain risks was more important than being polite. I think you and I disagree about that, and it makes sense - my audience was "people who might join this project", not you.

That said, I do think that if I had more spoons to spend, I could have communicated better AND more politely. I wish I had possessed the spoons to do your idea more justice, because it was a cool and ambitious idea that pushes the community forward.

I still think it's important to temper that ambition with more concern for safety than you're showing. I think dismissing the risks of abuse / the risks to emotional health as "chicken little" is a dangerous norm. I think it encourages dangerous experiments that can harm both the participants, and the community. I think having a norm of dangerous experiments expects far too much from the rationality of this community.

I think a norm of dismissing *assholes* and *rudeness*, on the other hand, is healthy. I think with a little effort, you could easily shift your tone from "dismissing safety concerns" to "holding people to a higher standard of etiquette." I personally prefer a very blunt environment which puts little stock in manners - I have a geek tact filter (http://www.mit.edu/~jcb/tact.html), but I realize not everyone thrives in that environment.

---

I myself was wrong to engage with them as if their beliefs had cruxes that would respond to things like argument and evidence.

I suspect I failed heavily at making this clear in the past, but my main objection was your lack of evidence. You said you'd seen the skulls, but you weren't providing *evidence*. Maybe you saw some of the skulls I saw, maybe you saw all of them, but I simply did not have the data to tell. That feels like an *important* observation, especially in a community all about evidence and rational decisions.

I may well be wrong about this, but I feel like you were asking commenters to put weight in your reputation. You did not seem happy to be held to the standard of Anonymous Internet Dude and expected to *show your work* regarding safety. I think it is, again, an *important* community standard that we hold people accountable to *demonstrate* safety instead of just asking us to assume it, especially when it's a high-visibility experiment that is actively using the community as a recruiting tool.

(I could say a lot more about this, but we start to wander back in to "I do not have the spoons to do this justice". If I ever find the spoons, expect a top-level post about the topic, though - I feel like Dragon Army should have sparked a discussion on community norms and whether we want to be a community that focuses on meeting Duncan or Lixue's needs. I think the two of us are genuinely looking for different things from this community, and the community would be better for drawing establishing a common knowledge instead of the muddled mess that the draft thread turned in to.)

(I'm hesitant to add this last bit, but I think it's important: I think you're assuming a norm that does not *yet* exist in this community. I think there's some good discussion to be had about conversational norms here. I very stridently disagree that petty parenthetical namecalling and insults is the way to do it, though. I think you have some strong points to make, and you undermine them with this behavior. Were it a more-established social norm here, I'd feel differently, but I don't feel like I violated the *existing* norms of the community with my responses)

---

As an aside: I really like the concepts you discussed in this post - Stag Hunts, the various archetypal roles, ways to do this better. It seems like the experiment was a solid success in gathering information. The archetypes strike me as a really useful interpersonal concept, and I appreciate you taking the time to share them, and to write this retrospective.

comment by Kaj_Sotala · 2018-03-22T21:27:36.947Z · LW(p) · GW(p)

Given how much criticism one specific, relatively minor aspect of this post has been getting from a lot of people - myself included - I feel like I should explicitly state that:

1) This post is awesome, and having a public postmortem of a project like this is really really valuable

2) While I stand by all of the criticism which I have not explicitly retracted, I want to be clear that my criticism was focused on a very minor part of the post, and, as I already stated in point #1, this post is awesome overall and I upvoted it

3) Many of the reactions that Duncan got on his original post were totally out of proportion, and many orders of magnitude worse than anything he said in this post

4) The fact that so much of the discussion here is focusing on this relatively minor issue is probably a reflection of the fact that there wasn't anything more substantial to disagree with in this post; so it feels to me like that a minor issue is getting a disproportionate amount of weight in contrast to this post's merits, and I feel slightly bad for having been one of the people to pile on Duncan and making his experience of reading the comments more punishing than it should be, given how good this post was overall.

(Someone may feel that the share of critical comments isn't that large, and that most of the top-level comments are about something else. This is true. Still, usually negative comments are experienced by the author as much more strongly than positive comments, so I'm taking that into account when I feel like there's been a somewhat disproportionate amount of attention on the criticisms. That said, I don't feel like anyone who contributed to them did anything wrong - after all, I contributed to them myself. Rather, I want to just clearly indicate that I think this was a great post overall, and that I really that the amount of criticism-about-a-minor-thing won't leave a bad taste in Duncan's mouth overall.)

comment by adifferentface · 2018-03-22T23:59:12.611Z · LW(p) · GW(p)

I endorse this, while still considering it important that criticism be given if warranted. As always, and unfortunately, there is less to say to someone on an issue where we already agree.

comment by Elizabeth (pktechgirl) · 2018-03-20T22:39:20.395Z · LW(p) · GW(p)
*I'll note that some of the Chicken Littles were clamoring for an off-ramp

I really dislike the dismissal of people who wanted to preserve easy exit as an abuse avoidance measure. I get that it can feel like an accusation, but being able to say "we should have this preventative measure" without implying anything negative about anyone is a critical to preventing abuse, because it lets you put the measure in place before the abuse is so bad it's obvious. I also (knowing basically nothing about Duncan or the way the suggestion was delivered, and having vague positive feelings about the project goals) think that "something intolerable happens" is a reasonable concern and "leaving" is a reasonable solution.

comment by Duncan_Sabien · 2018-03-20T23:06:59.531Z · LW(p) · GW(p)

Conflation.

All squares are rectangles, but not all rectangles are squares.

You are responding as if I said "all people who wanted to preserve an easy exit as an abuse avoidance measure were Chicken Littles."

I did not. I said "Some of the Chicken Littles were clamoring for an easy exit as an abuse avoidance measure."

This is an important distinction that I expect we should have a culture of not missing here on LessWrong. You can go back to the original post and see plenty of examples of me responding positively to people who were concerned about abuse risk. You can also see me publicly committing to specific changes, and publicly admitting specific updates. I was not dismissive in the way that your first sentence (correctly!) disagrees with being; I very much strongly agree with the sentence "being able to say 'we should have this preventative measure' without [that being taken as] implying anything negative about anyone is critical to preventing abuse.

(provided that being able to disagree with the call for Specific Preventative Measure X is not conflated with "doesn't care about preventing abuse.")

comment by Wei_Dai · 2018-03-21T17:52:07.486Z · LW(p) · GW(p)

You are responding as if I said “all people who wanted to preserve an easy exit as an abuse avoidance measure were Chicken Littles.”

I did not. I said “Some of the Chicken Littles were clamoring for an easy exit as an abuse avoidance measure.”

English sentences like the latter can have something like the former as a possible meaning. (I want to say “implicature” but I'm not totally sure that's the correct technical term.) So I think even putting aside politics and coalitions, the sentence is ambiguous as a matter of linguistics.

Consider this example:

  • Some nice people helped me look for my dog yesterday.

It seems clear that the meaning of this sentence is closer to "Some people helped look for my dog yesterday, and they were all nice people." or "Some people helped look for my dog yesterday, and that makes me think they are nice." than "The intersection of sets [nice people] and [people who helped me look for my dog yesterday] is non-empty."

Or this example:

  • Some of the undisciplined children in his class couldn't sit still for more than a few seconds at a time.

This one is more ambiguous than the one above. I can make my brain flip between perceiving two different meanings, one where there's a pre-identified group of undisciplined children, and the speaker observed some of them not being able to sit still, and another one where the speaker thinks that not being able to sit still is prima facie evidence for a child being insufficiently disciplined.

comment by Kaj_Sotala · 2018-03-21T07:27:04.432Z · LW(p) · GW(p)
This is an important distinction that I expect we should have a culture of not missing here on LessWrong.

I linked this elsewhere in this thread too, but seems particularly relevant here: http://slatestarcodex.com/2014/05/12/weak-men-are-superweapons/

... suppose the atheist posts on Tumblr: “I hate religious people who are rabidly certain that the world was created in seven days or that all their enemies will burn in Hell, and try to justify it through ‘faith’. You know, the sort of people who think that the Bible has all the answers and who hate anyone who tries to think for themselves.”
Now there’s practically no implication that these people are typical. So that’s fine, right?
On the other side of the world, a religious person is writing “I hate atheists who think morality is relative, and that this gives them the right to murder however many people stand between them and a world where no one is allowed to believe in God”.
Again, not a straw man. The Soviet Union contained several million of these people. But if you’re an atheist, would you just let this pass?
How about “I hate black thugs who rob people”?
What are the chances a black guy reads that and says “Well, good thing I’m not a thug who robs people, he’ll probably love me”?

Moral of the story being that, as that SSC post discusses in more detail, statements like 'the chicken littles made this stupid argument' line will end up rubbing off on everyone who made a similar-sounding argument, the same way that "I hate black thugs who rob people" still associates black people with thugs even though it's specifically stated that it's a subgroup of black people that's being talked about.

People objecting to those kinds of lines, in a way which arguably misses the distinction being drawn, is a natural immune reaction intended to prevent the larger group from being infected by the stigma of the smaller group. I don't know what the consequences of having a norm against making those objections would be, but given that it would be interfering with a natural immune reaction that seems to serve an important role in maintaining healthy social dynamics, it seems like the kind of thing that would be a bad idea to tamper with.

Or to put it differently, when you say

You are responding as if I said "all people who wanted to preserve an easy exit as an abuse avoidance measure were Chicken Littles."
I did not. I said "Some of the Chicken Littles were clamoring for an easy exit as an abuse avoidance measure."

Then while that is true, the fact that elizabeth is responding as if you had said the first thing, should be a hint of the fact that the first thing is how many people's brains will tend to interpret your statement on a System 1 level; which means that the first one is the message that this line is actually sending, regardless of what the authorial intent was.

comment by Duncan_Sabien · 2018-03-21T07:31:56.391Z · LW(p) · GW(p)

Literally the only reason I'm on LessWrong is because of the tiniest glimmer of a hope that this can be a place where people actually respond to what was said, rather than to their own knee-jerk stereotypes and rounded off assumptions. That this can be a place where people will actually put forth the effort to get the basic everywhere everyday flawed human communication bugs out of the picture, and do deliberate and intentional communication and collaborative truth seeking on a meaningfully higher level. That that's the actual goal—that when people stick the flag of "Less Wrong" in the ground, they mean it, and are willing to put their social capital on the line to grow it and defend it. That this can be a place where we don't just throw up our hands, give up, cave in, and cater to the lowest common denominator.

That this can be a place where the truth is ACTUALLY a defense against criticism. That if somebody here gets mad at you for what they think you said, and the record shows that you didn't say that, everyone will agree that it's the person who got mad who's in the wrong and should've done differently, not you.

Everything Scott said in that post rings true to me about people and populations in general. But the hope is that LessWrong is not just humans doing business as usual. The hope is that LessWrong is actually different. That the mods and individual members will make it different, on purpose, with deliberate effort and maintenance, according to solid principles consistently adhered to. That we'll put forth the effort to coordinate on this particular stag hunt, and not just keep cynically sliding back toward the same old boring rabbit rabbit rabbit and being apologists for our own irrationality. I can indeed give up on that hope, but the result will be me leaving and taking my content down and not coming back.

And yes, it'd be right in line with how "many people's brains" work to interpret that as a threat, or an attempt to hold the community hostage, or whatever. But it's not that—it's simply a statement that that's the value I see in LessWrong, and if that value isn't there, LessWrong isn't worth it. If LessWrong is a place where you can be punished for other people's failure to listen and think carefully, then it's not meaningfully different from the entire rest of the internet.

If this place isn't trying to be special in that way, then in what way is it trying to be special?

comment by Kaj_Sotala · 2018-03-21T08:32:16.748Z · LW(p) · GW(p)

I agree that we should strive for discussion norms which allow for more rational discussion, and which cause people to respond to what an author actually said, rather than responding to a stereotype in their heads. And that this is pretty much the whole point of Less Wrong.

At the same time, I think that something like "pick your battles" applies. Justified or not, there's already a relatively strong norm against political discussion on LW, arising in part from the assumption that politics is such a mind-killer that there's no point in even trying to discuss it in rational terms. That seems like a concession to the fact that we're still human and driven by human coalitional instincts, and that does need to be taken into account, even if we strive to overcome it.

Now, by its nature, LW already tends to attract the kinds of people who want to focus on a relatively literal interpretation of what was said, and don't need to be explicitly told so. Most of the implicit conversational norms here don't arise from anybody needing to be told to "interpret this more literally", but rather out of everyone naturally preferring that kind of conversation.

To me, this suggests that if there's something which causes even many people like us to instinctively and automatically react by reading something in a non-literal way, then that reaction is a very powerful force, and that special caution is required. If we are suggesting a norm for dealing with that reaction, then we should at least try to do some basic cost/benefit analysis for that proposed norm, keeping in mind its likely function.

I read you as proposing some kind of norm like "always read claims about subsets of specific groups as only referring to that subset, if the claim is worded in such a way as to make that the literal interpretation".

To me, a major cost would be that this feels kind of similar as a norm of "always stay rational when discussing politics"; leaving aside the fact that the politics norm is uselessly vague, there's also the fact that "rationality when discussing politics" isn't something that would be under people's conscious control. Unless they are very very good, they are going to go tribal even if they tried not to.

Similarly, I think that people's S1 associations of groups will be affected by claims where subgroups are lumped together with the larger group, regardless of how they're told to read different sentences. If we try, we might be successful in enforcing a norm against complaining about such claims, but we can't force people's intuitions not to be affected by the claims.

So, if we were successful in enforcing that norm, then that would in effect be incentivizing people to associate low-status subgroups with larger groups they don't like, since it couldn't be called out anymore. That seems bad.

On the other hand, it would make things somewhat easier on well-meaning authors, since they wouldn't need to worry so much about adding all kinds of explicit disclaimers when talking about subgroups who might be associated with larger groups. A lot of people have been complaining about LW being too hostile of an environment to post on, so reducing authorial stress seems good.

The alternative would be to not have such a norm, in which case authors will sometimes get challenged in the comments for subgroup claims, giving them the opportunity to clarify that yes, they only meant to refer to the subgroup not the whole group (as you've done). This seems like it would cause authors some occasional frustration, but given the general climate on LW, if they just clarify that "no I really did only mean the subgroup, the larger group is fine" to everyone who objects, then that should mostly settle it.

My current feeling is that, in large part due to the existing no-politics norm, it's currently very rare for authors to really discuss subgroups in a manner which would necessitate any objections. Thus the cost for authors of not-having-a-proposed-norm seems quite low to me; symmetrically, the extra value had from establishing the norm would be low also. I find it difficult to guess what the actual cost of having the norm would be, so based on my heuristic of "be careful about the magnitude of costs you don't have a good model for" and in light of the seemingly limited value, I would feel hesitant about endorsing the norm.

comment by Duncan_Sabien · 2018-03-21T08:38:19.017Z · LW(p) · GW(p)
I read you as proposing some kind of norm like "always read claims about subsets of specific groups as only referring to that subset, if the claim is worded in such a way as to make that the literal interpretation".

This is not at all what I'm proposing; your post is way more fixated on the particular example than I expected. The radical norm that I am proposing is simply "read the words that people say, and process them attentively, and respond to those words." The political subset doesn't need to be considered separately, because if you have a community that supports and reinforces actually reading the words and processing them and responding to them, that's sufficient.

To me, this suggests that if there's something which causes even many people like us to instinctively and automatically react by reading something in a non-literal way, then that reaction is a very powerful force, and that special caution is required. If we are suggesting a norm for dealing with that reaction, then we should at least try to do some basic cost/benefit analysis for that proposed norm, keeping in mind its likely function.

I don't think cost/benefit analysis is the appropriate frame, here. I think this is the sole purpose, the sole mission. You don't walk into a martial arts academy and say, let's do a cost/benefit analysis on whether this whole kicking and punching thing is even worthwhile in a world full of guns. The frame is set—if you don't like martial arts, don't show up. Similarly, we shouldn't be evaluating whether or not to hold ourselves to a standard of rationality, even if doing so is very difficult in some subsets of cases. That question should be answered before one decides whether or not to show up on LESS WRONG. If a person thinks it's too costly, they shouldn't be here.

Cost/benefit analyses can help us choose which of several different strategic paths toward the goal to take, and they can help us prioritize among multiple operationalizations of that goal, but they shouldn't be used to let ourselves off the hook in exactly the areas where rationality is most missing or difficult, and therefore improvements are most needed.

I'm not going to add any further responses to this subthread, because I've said all I have to say. Either LW will agree that this is something worth coordinating to all-choose-stag on, or it won't. It looks like, given the attitudes of most of the mods, it'll probably be "won't," but there's still room for hope.

comment by Kaj_Sotala · 2018-03-21T10:05:04.552Z · LW(p) · GW(p)
The political subset doesn't need to be considered separately, because if you have a community that supports and reinforces actually reading the words and processing them and responding to them, that's sufficient.

Since you expressed a desire to disengage from the conversation, I'll just briefly note for the benefit of others that this excerpt seems like the biggest crux and point of disagreement. To me, coalitional instincts are something that are always active in every group, and whose influence needs to be actively fought back, or they will subvert the goals of the group; just deciding to ignore the political aspects of things, without considering in detail the effect that this change will have on social dynamics, is never sufficient.

comment by bogus · 2018-03-21T10:23:05.722Z · LW(p) · GW(p)

... I'll just briefly note for the benefit of others that this excerpt seems like the biggest crux and point of disagreement. ...

In tne interest of the general norm of "trying to identify cruxes and make them explicit", I'd like to endorse this - except that to me, the issue goes well beyond "human coalitions" and also encompasses many other things that would generally fall under the rubric of 'politics' in a broad sense - or for that matter, of 'ethics' or 'morality'! When people, plausibly, were 'politically' mindkilled by Duncan's Dragon Army proposal, this was not necessarily due to their belonging to an "anti-Duncan", "anti-rationality" or whatever-coalition; instead, the proposal itself may have been aversive to them in a rather deep sense, involving what they regarded as their basic values. This impacts the proposed solution as well, of course; it may not be sufficient to "actively fight back" a narrow coalitional instinct, but a need may arise for addressing "the political [or for that matter, moral, ethical etc.] aspects of things" at a somewhat deeper level, that goes beyond a conventional "arguments and evidence" structure to seek for 'cruxes' in our far more fundamental attitudes, and addresses them with meaningful and creative compromises.

comment by Kaj_Sotala · 2018-03-21T10:49:53.581Z · LW(p) · GW(p)

Yeah, agreed. It's not just "political instincts", it's that humans are always operating in what's a fundamentally social reality, of which coalitional instincts are a very large part but not the entirety.

I kinda dislike the "actively fight back" framing too, since it feels like a "treating your own fundamental humanity as an enemy" kind of thing that's by itself something that we should be trying to get out of; but the easiest links that I had available that concisely expressed the point used that language, so I went with that.

comment by bogus · 2018-03-21T11:03:08.078Z · LW(p) · GW(p)

I actually thought the "coalitional" part did deserve a mention, precisely because it is one of the few facets of the problem that we can just fight (which is not to say that coalitions don't have a social and formal role to play in any actual political system!) Again, I think Crick would also agree with this, and ISTM that he did grapple with these issues at a pretty deep level. If we're going to go beyond our traditional "no politics!" attitude, I really have to wonder why he's not considered a trusted reference here, on a par w/ the Sequences and whatever the latest AI textbook is.

comment by Kaj_Sotala · 2018-03-21T11:05:55.688Z · LW(p) · GW(p)
I think Crick would also agree with this, and ISTM that he did grapple with these issues at a pretty deep level.

Do you have reading recommendations on him?

comment by Conor Moreton · 2018-03-21T19:51:58.046Z · LW(p) · GW(p)

Loren ipsum

comment by Kaj_Sotala · 2018-03-21T20:54:46.425Z · LW(p) · GW(p)

Yeah, you're probably correct; I don't feel like I'd have a good enough handle of your model to even attempt your ITT. (If this is your way of subtly pointing out that the thing I identified as a crux is likely wrong, correction accepted.)

comment by bogus · 2018-03-21T08:09:05.534Z · LW(p) · GW(p)

That this can be a place where people will actually put forth the effort to get the basic everywhere everyday flawed human communication bugs out of the picture, and do deliberate and intentional communication and collaborative truth seeking on a meaningfully higher level. ... Everything Scott said in that post rings true to me about people and populations in general. But the hope is that LessWrong is not just humans doing business as usual. The hope is that LessWrong is actually different.

Look, I hate to break the news to you, but just like Soylent Green, Less Wrong is people! Your goals and aspirations are extremely worthwhile and I entirely agree with them, but to whatever extent they succeed, it will NOT be because "LessWrong is not just humans doing business as usual"! Rather, it will be bdcause the very definition of "business as usual" - and, crucially, "politics as usual"! - will have been successfully modified and perfected to make it more in line with both human values (humaneness) as they actually exist out there, in the real world, and a general norm of truth seeking and deliberation (which is however, i claim, not a sufficient condition to achieving this goal, other 'norms of engagement' being just as important). This is what it actually means to "raise the sanity waterline"! Making us less human and perhaps more Clippy-like (that is, with the entirety of accepted discourse being "Hey, it looks like you might be having problem X! Would you like me to use advanced Bayesian inference techniques to help you assess this problem and provide you with a helpful, canned solution to it? [OK]/[CANCEL]") is not a sensible or feasible goal, and it is indeed somewhat puzzling that you as a CFAR instructor yourself do not immediately notice and engage with this important point.

comment by norswap · 2018-03-19T22:14:04.599Z · LW(p) · GW(p)

I didn't know about the experiment, so I went to read the charter and came back to read this.

What struck me after reading the charter was "How will these people find the time?"

I ran a quick computation, and it turns out that after work and sleep, I have 7 hours left in my day (which, given I sleep 8 hours might be conservative by US standards). Accounting that some stuff just has to be done (hygiene, ya know), the 3 hours that one should theoretically dedicate to house stuff every day is half of your remaining time. Everything else you care about must fit in that three other hours. Seems tiny to me.

So was it the case that people were starved for time? Was it a cause of "soft-defection"?

comment by Vaniver · 2018-03-19T23:20:06.221Z · LW(p) · GW(p)
So was it the case that people were starved for time? Was it a cause of "soft-defection"?

At least one person in the house had expressed interest, and then before the house started took on additional constraints on their time, and so found themselves in this awkward bind where they could make time for the mandatory things, but just the mandatory things, and this was not quite in the spirit of the thing. (Most of the time when people were in 'black knight' roles, it was because of this sort of thing.)

I think this was comparatively small; I think the red knights caused more trouble by adding friction to the system that slowed everything down (not just the stuff that they pushed back again, but also all the other things that didn't happen because Duncan or others were paying attention to the pushback instead of building other things).

comment by Ekphrasis · 2018-03-21T01:59:26.352Z · LW(p) · GW(p)

One idea to consider for the future: Select a Dragon whose primary responsibility (even at the expense of other house commitments) is to serve as the house Tracker. They would be responsible for keeping track of all ongoing experiments, gathering data, analyzing it, and recommending areas for improvement (as measured against prior commitments)... but leaving the actual strategy for improvement up to the Commander.

Their primary purpose would be to make the house more self-reflective, by consistently drawing attention to what is and isn't working. They could also check in regularly with other Dragons on the status of their individual goals, both to provide accountability and to incorporate data from those check-ins into their overall tracking.

I think the roles of Tracker and Commander are too much for one person to handle alone.

comment by Duncan_Sabien · 2018-03-21T03:20:53.511Z · LW(p) · GW(p)

Yeah. I thought they wouldn't be too much, and I was wrong.

comment by Zvi · 2018-03-19T17:18:11.713Z · LW(p) · GW(p)

This makes me wonder - what of the missing two Knights?

The Green Knight is spoken of in legend and it seems appropriate. If you strike the Green Knight, it adopts and strikes you back. You can think you've got it under control, but you're wrong - 'life can't be contained' and all that. Green Knight might seem to agree to things, but it doesn't take. This legend was also the origin of what, when I was trying to model fair values for things in various non-efficient markets, my partner called The Green Knight Test - you get to bet into the market (at its midpoint) but the market then gets to bet into your model. And you're fully ready only if you can beat that. Because that means you actually understand the whole of its nature. Until then all you have is a Zeroing Out.

What then of the Blue Knight? Presumably he is the rationalist sphere writ large that champions the pursuit of knowledge - and who also resists, in the form of wanting to constantly reason everything out rather than exist in the moment or allow the rituals to work. The hidden enemy of the experiment, who we have met, and the enemy is us.

comment by ESRogs · 2018-03-20T23:33:11.133Z · LW(p) · GW(p)

I suspect that this is a part that some people might not have understood:

This legend was also the origin of what, when I was trying to model fair values for things in various non-efficient markets, my partner called The Green Knight Test - you get to bet into the market (at its midpoint) but the market then gets to bet into your model. And you're fully ready only if you can beat that. Because that means you actually understand the whole of its nature.

My attempted paraphrase -- let's say you have a model that prices assets that you can bet on (stocks, odds on a boxing match), and you think the model is pretty good. But is it actually good?

Well, the obvious test is whether you can make money from your model, by placing bets when it disagrees with the market. But Zvi is pointing out that there's really two different versions of this test: the Level 1 test, and the Level 2 test. And he calls Level 2, The Green Knight Test.

For the Level 1 test, when your model disagrees with the market, you get to decide whether to bet or not, and you get to bet at the current market price. So maybe you only bet when the model strongly disagrees with the market, or when the model is highly confident.

For the Level 2 test, you don't get to decide when to bet. You state your price, and then the market gets to decide whether to bet with you or not. And as long as others are willing to trade at your price you have to keep buying or selling. (Zvi calls this the Green Knight Test, because you're not just on the offense placing bets -- you gotta handle whatever the market throws back at you, like the Green Knight who strikes back.)

To make money on the Level 1 test, you just have to sometimes be more right than the market (and be able to tell when that is). To make money on the Level 2 test, you have to be more right than the market on average, even when the market gets to pick its battles.

@Zvi, did I get it right?

comment by lionhearted · 2018-03-21T06:35:56.699Z · LW(p) · GW(p)

Whoa. Thanks for the clarification/elaboration. I'm a big Zvi enthusiast but was unable to follow the idea until you commented. Interesting. It's the difference between needing to be be able to spot occasional exploitable inefficiencies and to have full-on defenses against everything bad.

comment by PeterBorah · 2018-03-19T18:59:14.598Z · LW(p) · GW(p)

(This is an entirely meta post, which feels like it might not be helpful, but I'll post it anyway because I'm trying to have weaker babble filters. Feel free to ignore if it's useless.

I generally enjoy your writing style, and think it's evocative and clear-in-aggregate. But I find this comment entirely inscrutable. I think there's something about the interaction between your "gesturing" style and a short comment, that doesn't work as well for me as a reader compared to that style in a longer piece where the I can get into the flow of what you're saying and figure out your referents inductively.

Either that or you're referencing things I haven't read or don't remember.)

comment by Zvi · 2018-03-20T13:16:39.984Z · LW(p) · GW(p)

Thank you, good feedback. My feeling was that as a short comment it was 100% fine if it didn't make sense to non-MtG people (or at least, anyone without the color wheel), and I'd rather keep it short than give the necessary background. Plus, it was more of a 'fun exploration' thing than trying to make a serious point.

Likely I should have linked back to the color wheel post, though.

comment by azavoluk · 2018-03-22T04:48:07.199Z · LW(p) · GW(p)

For the record, I play Magic regularly and have for a long time, and I didn't get it. I'm still not sure to what extent the colors really align with the descriptions given by either Duncan or Zvi.

The red knight values individual sovereignty, yes, but is not risk-averse or cautious. Red is certainly capable of forming armies that follow a leader.

Black doesn't particularly care about "convincing arguments."

Green could be a plant-like thing that's hard to kill, but it could also be a very fragile plant that shrivels up and dies. Or judgy, racist elves, or whatever.

Perhaps these are not so much absolute representations, as the representation of each color that is most likely to appear in a rationalist house (basically, these are all X/u knights, not pure X knights).

comment by jsteinhardt · 2018-03-19T19:43:11.984Z · LW(p) · GW(p)

FWIW I understood Zvi's comment, but feel like I might not have understood it if I hadn't played Magic: The Gathering in the past.

EDIT: Although I don't understand the link to Sir Arthur's green knight, unless it was a reference to the fact that M:tG doesn't actually have a green knight card.

comment by CronoDAS · 2018-03-20T15:04:48.736Z · LW(p) · GW(p)

The Arthurian Green Knight lets Gawain cut off his head, then picks it up and puts it back on. Trying to use force on the Green Knight is useless.

comment by LawChan · 2018-03-20T04:45:55.494Z · LW(p) · GW(p)

I also think I wouldn't have understood his comments without MTG or at least having read Duncan's explanation to the MTG color wheel.

(Nitpicking) Though I'd add that MTG doesn't have a literal Blue Knight card either, so I doubt it's that reference. (There are knights that are blue and green, but none with the exact names "Blue Knight" or "Green Knight".)

comment by CronoDAS · 2018-03-20T15:03:07.292Z · LW(p) · GW(p)

Also the Red Knight card is named "Blood Knight".

comment by LawChan · 2018-03-19T04:10:39.434Z · LW(p) · GW(p)

Thanks for posting this. I found the framing of the different characters very insightful.

comment by Duncan_Sabien · 2018-03-19T04:51:23.394Z · LW(p) · GW(p)

Yeah. Other than stag hunts, watching out for and having strategies for interfacing with Black, White, and Red Knights is by far the most useful takeaway for me as a team leader. I feel like I can't emphasize enough how important those three are to my model of social dynamics; they're probably going to crop up in several other essays.

comment by gwillen · 2018-03-19T08:23:10.564Z · LW(p) · GW(p)

I was surprised to hear that you ended up with Red Knights in Dragon Army. I would tend to fit that stereotype myself, I think, but your original post made it very clear what to expect going in. I'm surprised that anybody made it past that filter with Red Knight tendencies, while not being consciously aware of needing to rein them in as part of the experiment.

comment by Duncan_Sabien · 2018-03-19T09:07:46.183Z · LW(p) · GW(p)

I, too, was surprised. That first night. Very much wrongfooted me. Was not expecting it.

comment by PeterBorah · 2018-03-19T18:53:11.119Z · LW(p) · GW(p)

Agreed. I strongly identify with the description of the Red Knight (and somewhat the description of both the other two knights as well), and was therefore Not Interested in Dragon Army. To the point that I posted some strong critiques of the idea, though hopefully in a constructive manner.

I would be interested in a retrospective of how the people who inhabited that role ended up joining Dragon Army. What was the bug there? I though Duncan was admirably clear about how much it was a non-Red Knight-friendly zone.

comment by Kaj_Sotala · 2018-03-19T13:10:17.520Z · LW(p) · GW(p)

Reading your post made me think of occasions when I've been a White Knight in some project, and also of occasions when I've become a Black Knight in response.

Interestingly, while the Red Knight description also resonated as true in the sense of "yeah I've been that person", my initial sense-of-recognition was a memory of occasions when I was annoyed by people who were Red Knighting while I was trying to White Knight.

comment by ChristianKl · 2018-03-19T09:01:05.670Z · LW(p) · GW(p)

I like that you gave both the grades compared to vision and compared to another rational house for yourself. I think it would be great to also have those grades for the other people and average grade.

comment by Duncan_Sabien · 2018-03-19T09:19:49.087Z · LW(p) · GW(p)

It's hard to do that for the other individuals, since it would be something like "empty set" or "mu" in the counterfactual? But basically all of the Dragons are at least A- and in many cases A+ roommates; those who are moving out come with my highest recommendation and any house is likely to benefit from their presence.

comment by Davis_Kingsley · 2018-03-21T07:46:24.607Z · LW(p) · GW(p)

Great post! I especially like your description of the different Knights and of the "stag hunt" dynamic in general. I think I've very likely been in the white role at times in other groups. However, there's a dynamic near to this one that I also see a lot - that's one where all the relevant scaffolding has been built, the systems are in place, the group can start choosing "stag" if it wants to, but this isn't common knowledge yet.

I have often encountered (what I believe to be) this second dynamic, and it strikes me as very important - if the situation really is such that a little push can tip the balance into the "stag" equilibrium, doing that is crucial! Distinguishing between these two states seems hard and I'd quite like to be better at it.

comment by lionhearted · 2018-03-21T06:30:01.024Z · LW(p) · GW(p)

Lots of interesting ideas here. Thanks and massive respect for taking the time to think this through and write it up.

>D-, D, C-, C, C, C, B-, B, B, B+

What would have been required for have someone score an "A" ?

Both from the participant themselves, and from you structurally? Did you need different people? Or could the top people who participated have graded out at the top with different or better structure? Etc.

Again, big thanks and regards for the honest reflection.

comment by Duncan_Sabien · 2018-03-21T07:07:59.499Z · LW(p) · GW(p)

I feel like there were combinations of people that could've produced an A. Like, I can imagine taking a liiiiiittle bit of the personality of one person and layering it in over the core motivation of another, and saying "yep, that's an A right there."

Some people had the right spirit, but had too few spoons.

Some people had the right ideas, but didn't follow through.

Some people had the right connections with the rest of the house, but didn't put forth energy.

Some people poured out their heart and soul, but didn't have good judgment.

Some people waited too long for leadership, and didn't step up; others stepped up too hard, and pulled in the wrong directions.

Some people were too chill, and others weren't chill enough.

I think in order for me to get an A, I would've needed to get more of the ops and logistics down prior to the start date, and I would've needed to have the confidence or moral authority to push through the Red Knight, rein in the White Knight, and send the Black Knight away. That, plus a little more consistency and follow-through on a couple of core experiments, and I would've given myself an A even if the overall project still didn't succeed.

As for the other Dragons, I think the pieces were there. But structurally, I didn't build the right tubes for people's individual magic mojo to flow into and infect one another. Instead, everyone continued to be their own sort of awesome, and nobody really got imbued with the awesome of their fellow peers.

comment by azavoluk · 2018-03-22T04:32:34.745Z · LW(p) · GW(p)

I'm not really familiar with DA beyond the 2 posts and knowing Vaniver. That being said, thoughts that came to mind as I read.

0. This just seems like a standard coordination problem that I would expect most rationalists to be familiar with. Is there some reason you needed an entirely new analogy? I would not have predicted that out of ~12 reasonably dedicated rationalists, more than 1 would not have this idea as a Thing That Exists In One's Mind.

1. Seems like a good suggestion that I would not have thought of ahead of time, but seems obvious in retrospect.

2. I guess I didn't realize how much you were supposed to still be doing at CFAR while this was going on. If I had, I probably would have been significantly more pessimistic. It certainly sounds like the most "obvious" thing out of this list, a priori, even more than the things in 5.

6. Have you read Superforecasters? Something the book goes into, that SSC's review does not, is a bit of military history and strategy, pioneered by a German named Moltke. The basic idea is that generals cannot anticipate everything that can happen on the ground, and so they should direct the objective of lower officers, rather than the means of attaining them. Perhaps, if you are going to form an "army," that is equally as valid a model to consider as infantry grunts? My guess would be that rationalists are naturally going to resist the sort of hyper-hive-assimilation of a low-level unit (having many Red Knights, in your terms). Under this model, setting goals, agreeing on serious consequences for not meeting them, encouraging working together on them, and providing communal resources for working on them, might be more effective.

(Allegedly these ideas largely explained the Germany Army's disproportionate effectiveness in WW2, and Moltke's ideas have been adopted by pretty much every modern military).

I believe this is also the model described in Ender's Game, which leads me to my next thought--is 12 too many? Or worded differently, is it likely the same group size is optimal for all the things you want to accomplish? Ender broke his army into many more pieces than usual, all the way down into groups of 4 or 5, to get each group to be very tight. Maybe you've already got this part covered; I only mention it because I didn't see any mention of doing this systematically, only the EE thing (which you said was very successful).

16. I really like your phrasing at the end of this, and will probably look for an opportunity to use it at some point.

edit: I totally forgot, I think 7) is the one that is basically in line with what I actually did predict at the start.

comment by Vaniver · 2018-03-22T19:49:51.195Z · LW(p) · GW(p)
0. This just seems like a standard coordination problem that I would expect most rationalists to be familiar with. Is there some reason you needed an entirely new analogy? I would not have predicted that out of ~12 reasonably dedicated rationalists, more than 1 would not have this idea as a Thing That Exists In One's Mind.

My impression is that while I might expect a median economist to have heard of a stag hunt, I don't expect that of a median rationalist (where I do expect a median rationalist to have heard of the prisoner's dilemma). Talking about choosing cooperate vs. defect triggers a different sort of social reasoning / mental movements than talking about choosing stag vs. rabbit.

(In particular, I remember several discussions about house culture early on, where a norm of "always cooperate" was proposed and got some pushback, when "choose stag" would have likely made more sense / gone over better.)

Something the book goes into, that SSC's review does not, is a bit of military history and strategy, pioneered by a German named Moltke.

I don't think auftragstaktik would've helped with the specific problems underlying 6. I think that was something more like--a major benefit of having a commander is to specialize cognitive labor and lower coordination costs (since everyone just needs to know the plan, not feel like they have justified the plan to their satisfaction), and instead we had more standard discussion and consensus forming that led to us not specializing cognitive labor or lowering coordination costs, and not having practiced following orders for when it was important to follow orders. (Mission-type tactics still relies on people following orders!)

comment by Ben Pace (Benito) · 2018-03-22T20:08:25.834Z · LW(p) · GW(p)

As I pointed out in my recent post, in econ the prisoner’s dilemma is not a coordination problem, and it seems useful to conceptually separate those two decision matrices.

comment by Vaniver · 2018-03-23T03:10:30.214Z · LW(p) · GW(p)
As I pointed out in my recent post, in econ the prisoner’s dilemma is not a coordination problem, and it seems useful to conceptually separate those two decision matrices.

Sure; did you read me as claiming that it was was?

I also note that the Stag Hunt is importantly different from the Coordination Game in that only one of the good outcomes requires coordination--if you choose Rabbit, you get the lower payoff regardless of what I do.

comment by Zera · 2018-03-21T18:45:08.916Z · LW(p) · GW(p)

I greatly appreciate this post mortem. It has me excited for DAB 2.0, which seems pretty likely to either succeed or provide further valuable insights, but it doesn't seem like DAB 2.0 is on the horizon. My current estimation of why from the post and comments is that too many of the participants in 2.0 are worn out, but I wonder if you could confirm or correct and elaborate. Thanks again, both for the effort itself and for the write-up.

comment by Vaniver · 2018-03-21T20:17:05.641Z · LW(p) · GW(p)

Many people are moving out, and we are looking for people to move in; my guess is that phase 2 is going to be a more relaxed setup where we have specific rules and ambitions, but not the general command structure. (This might be something like "come to Tuesday night dinner, be growing on at least 2 of these 4 axes in a way that's legible to the house / you want to be held accountable for.) I think we're open to a phase 3 where we try again for the full thing, but it'd be more like a year from now than a few months from now.

comment by 12Case · 2018-03-19T06:45:20.834Z · LW(p) · GW(p)

Thank you for trying this. I have been watching from afar, and I wanted it to succeed. Point #1 here seems extremely true.

comment by drethelin · 2018-03-19T04:44:08.692Z · LW(p) · GW(p)

The main thing I want to say is that it's a huge error to leave something as a coordination problem when it doesn't have to be. The way stag hunts work IRL is by people going "Hey, who wants to go on a Stag Hunt?" and if sufficient people to capture a stag sign up, then the hunt happens. If they don't, there's no situation where a bunch of people waste their time on an impossible stag hunt. You can also do this in the Dragon Army sense by commanding a stag hunt take place as the lord, but either option is better than there being some important, valuable task that has a chance of failing based entirely on individual judgments of whether other individuals will participate.

comment by Duncan_Sabien · 2018-03-19T04:47:22.628Z · LW(p) · GW(p)

Common knowledge, double illusion of transparency, typical mind fallacy stuff. It's not always clear what's even a "hunt" or not; often someone will try to choose stag on something like e.g. keeping a bathroom sink clean, and others won't disambiguate between a bid for reasonable effort and a bid for extraordinary effort. These things take place at all scales in a smooth gradient; there were plenty of places where there was clear coordination on stag vs. rabbit, but our eyes weren't opened to the commonality between a bunch of different Things That Went Kind Of Sideways until pretty late in the game.