Speaking of Stag Hunts

There's a vision here of what LessWrong could/should be, and what a rationalist community could/should be more generally. I want to push back against that vision, and offer a sketch of an alternative frame.

The post summarizes the vision I want to push back against as something like this:

What I really want from LessWrong is to make my own thinking better, moment to moment.  To be embedded in a context that evokes clearer thinking, the way being in a library evokes whispers.  To be embedded in a context that anti-evokes all those things my brain keeps trying to do, the way being in a church anti-evokes coarse language.

Now, I do think that's a great piece to have in a vision for the LessWrong or the rationalist community. But I don't think it's the central piece, at least not in my preferred vision.

What's missing? What is the central piece?

Fundamentally, the problem with this vision is that it isn't built for a high-dimensional world [LW · GW]. In a high-dimensional world, the hard part of reaching an optimum isn't going-uphill-rather-than-downhill; it's figuring out which direction is best, out of millions of possible directions. Half the directions are marginally-good, half are marginally-bad, but the more important fact is that the vast majority of directions matter very little.

In a high-dimensional world, getting buffeted in random directions mostly just doesn't matter. Only one-part-in-a-million of the random buffeting in a million-dimensional space will be along the one direction that matters; a push along the direction that matters can be one-hundred-thousandth as strong as the random noise and still overwhelm it.

Figuring out the right direction, and directing at least some of our effort that way, is vastly more important than directing 100% of our effort in that direction (rather than a random direction).

Moving from the abstraction back to the issue at hand... fundamentally, questionable epistemics in this episode of Drama just don't matter all that much. They're the random noise, buffeting us about on a high-dimensional landscape. Maybe finding and fixing organizational problems will lead to marginally more researcher time/effort on alignment, or maybe the drama itself will lead to a net loss of researcher attention to alignment. But these are both mechanisms of going marginally faster or marginally slower along the direction we're already pointed. In a high-dimensional world, that's not the sort of thing which matters much.

If we'd had higher standards for discussion around the Drama, maybe we'd have been more likely to figure out which way was "uphill" along the drama-salient directions - what the best changes were in response to the issues raised. But it seems wildly unlikely that any of the dimensions salient to that discussion were the actual most important dimensions. Even the best possible changes in response to the issues raised don't matter much, when the issues raised are not the actual most important issues.

And that's how Drama goes: rarely are the most important dimensions the most Drama-inducing. Raising site standards is the sort of thing which would help a lot in a high-drama discussions, but it wouldn't much help us figure out the most-important-dimensions.

Another framing: in a babble-and-prune model, obviously raising community standards corresponds to pruning more aggressively. But in a high-dimensional world, the performance of babble-and-prune depends mostly on how good the babble is - random babble will progress very slowly, no matter how good the pruning. It's all about figuring out the right direction in the first place, without having to try every random direction to do so. It fundamentally needs to be a positive process, figuring out techniques to systematically pursue better directions, not just a process of avoiding bad or useless directions. Nearly all the directions are useless; avoiding them is like sweeping sand from a beach.

Executive summary: I have no idea what you're talking about.

Standards are not really popular.  Most people don't like them.  Half the people here, I think, don't even see the problem that I'm trying to point at.  Or they see it, but they don't see it as a problem.

I gather that you're upset about how the Leverage conversation went, and also Cancel Culture, so I assume your chief proposition is that LessWrong is canceling Geoff Anders; but you haven't actually made that case, just vaguely gestured with words. 

I think that a certain kind of person is becoming less prevalent on LessWrong, and a certain other kind of person is becoming more prevalent, and while I have nothing against the other kind, I really thought LessWrong was for the first group.

What are the two kinds of persons? Really, I honestly do not know what you are claiming here. Repeat: I don't have even a foggy guess as to what your two kinds of person are. Am I "a certain other kind of person"? How can I know?

Distinguish feeling from fact.

This post has virtually no facts. It has short, punchy sentences with italics for emphasis. It is written like a hortatory sermon. Its primary tool is rhetoric. The first quarter of it is essentially restating parts of the sequences. Then you point to some comments and are upset that they got upvoted. Others, you're upset they haven't been upvoted. I have no idea whatsoever why you feel these things, and you don't elaborate. I am apparently one of the people who "don't even see the problem that [you're] trying to point at." 

This comment was much longer in draft, but I've deleted the remainder because I don't want to seem "impatient" or "sneering". I'm just confused: You wrote all these words intending to convince people of something, but you don't specify what it is, and you don't use the tools we typically use to convince (facts, reliable sources, syllogistic reasoning, math, game theory...). Am I just not part of the intended audience? If so, who are they?

I'll have more to say on this in the future, but for now I just want to ramble about something.

I've been reading through some of the early General Semantics works. Partially to see if there are any gaps in my understanding they can fill, partially as a historical curiosity (how much of rationality did they have figured out, and what could they do with it?), partially because it might be good fodder for posts on LW (write a thousand posts to Rome [LW · GW]).

And somehow a thing that keeps coming into my mind while reading them is the pre-rigorous/rigorous/post-rigorous split Terrence Tao talks about, where mathematicians start off just doing calculation and not really understanding proofs, and then they understand proofs through careful diligence, and then they intuitively understand proofs and discard many of the formalisms in their actions and speech.

Like, the early General Semantics writers pay careful attention to many things that I feel like I have intuitively incorporated; they're trying to be rigorous about scientific thinking (in the sense that they mean) in a way that I think I can be something closer to post-rigorous. Rather than this just being "I'm sloppier than they were", I think I see at least one place where they're tripping up (tho maybe they're just trying to bridge effectively to their audience?), of which the first that comes to mind is when an author, in a discussion of the faults of binarism, makes their case using a surprisingly binarist approach (instead of the more scientific quantitative language).

And so when I see something that seems to say "good math is all about correct deductions", there's a part of me that says "well, but... that's not actually where good math comes from, if you ask the good mathematicians." There's a disagreement going on at the moment between Zvi and Elizabeth [LW(p) · GW(p)] about what inferences to draw from the limited data we have about the container stacking story. It's easy for me to tell a story about how Zvi is confused and biased and following the wrong policies, and it's easy for me to tell a story about how Elizabeth is confused and biased and following the wrong policies. But, for me at least, doing either of those things would be about trying to Win instead trying to Understand.

And, like, I don't know; the reason this is a ramble instead of a clear point is because I think I'm saying "don't bother the professors who talk in intuitions" to someone who is saying "we really need to be telling undergraduates when they make math errors", and yet somehow I'm seeing in this a vision that's something more like "people become rational by carefully avoiding errors" instead of something that's more like "people become rational by trying to cleave as closely as possible to the Way".

You may try to name the highest principle with names such as “the map that reflects the territory” or “experience of success and failure” or “Bayesian decision theory.” But perhaps you describe incorrectly the nameless virtue. How will you discover your mistake? Not by comparing your description to itself, but by comparing it to that which you did not name.

This post makes me kind of uncomfortable and I feel like the locus is in... bad boundaries maybe? Maybe an orientation towards conflict, essentializing, and incentive design?

Here's an example where it jumped out at me:

Another metaphor is that of a garden.

You know what makes a garden?

Weeding.

Gardens aren't just about the thriving of the desired plants.  They're also about the non-thriving of the non-desired plants.

And weeding is hard work, and it's boring, and it's tedious, and it's unsexy.

Here's another:

But gardens aren't just about the thriving of the desired plants.  They're also about the non-thriving of the non-desired plants.

There's a difference between "there are many black ravens" and "we've successfully built an environment with no white ravens."  There's a difference between "this place substantially rewards black ravens" and "this place does not reward white ravens; it imposes costs upon them."

Like... this is literally black and white thinking? 

And why would a good and sane person ever want to impose costs on third parties ever except like in... revenge because we live in an anarchic horror world, or (better) as punishment after a wise and just proceeding where rehabilitation would probably fail but deterrence might work? 

And what the fuck with "weeds" and "weeding" where the bad species is locally genocided? 

Just because a plant is "non-desired" doesn't actually mean you need to make it not thrive. It might be mostly harmless. It might be non-obviously commensal. Maybe your initial desires are improper? Have some humility.

And like in real life agriculture the removal of weeds is often counter-productive. Weeding is the job you give the kids so that they can feel like they are contributing and learn to recognize plants. The real goal is to maximize yield while minimizing costs without causing too much damage to the soil (or preferably while building the soil up even better for next year), and the important parts are planting the right seeds at the right time in the right place, and making sure that adequate quantities of cheap nutrients and water are bio-available to your actual crop.

Just because voting is wrong, here and there...  like... so what? Some of my best comments have gotten negative votes and some of the ones I'm most ashamed of go to the top. This means that the voters are sometimes dumb. That's OK. That's life. Maybe educate them? Here are some heuristics I follow:

Scroll to the bottom of the comments first to find the N that have 1 point, first read them all, then upvote the better N/2. Then look at the M with 2 votes and upvote the better M/2. And so on. Anything lower but more useful to have read first should have relatively higher karma.

If something has a good link, that's better than no links. (Click through and verify, however.) If the link sucks, it is worse than no links.

Don't upvote things that are already on top unless there are other reasons to do so. If something really clever is written later on, it will make it harder for later voters to push it up where it properly belongs.

At least don't read the first comment then mash the upvote when it ends on an applause light. Check the next one below that and so on, and think about stuff.

If a comment is addressed to someone and they respond, even shitty responses often deserve an upvote because that person's response should usually come instantly after unless someone else has a short sweet and much better comment that works as a good prologue. Linear sequences of discussion by two good faith communicators is wonderful to read and anyone horning on the conversation should probably come later.

If a sequence of discussion leads to a super amazing insight 3 or 4 ply in... perhaps someone actually changed their mind... reward all the comments that lead to that (similarly to how direct two-person back and forth is interesting). There's an analogy here to backpropagation.

In an easy to read debate, if one person is making better points, you're allowed to vote them higher than their interlocutor to show who is "winning" but you should feel guilty about this, because it is dirty dirty fun <3

The more comments a comment accumulates, the lower the net utility of the entire collection. Writing things that stand alone is good. Writing things that attract quibbles is less good. The debates should be a ways down the page.

And so on. This is all common sense, right? <3

(It isn't common sense.) You can just look [LW · GW] and see that people don't understand this stuff, but that's why it is good to spell out, I think? Lesswrong never understood this stuff, and I once thought I could/should teach it but then I just drifted away instead. I feel bad about that. Please don't make this place worse again by caring about points for reasons other than making comments occur in the right order on the page.

We don't need to organize a stag hunt to exterminate the weeds. We need to plant good seeds and get them into the sunlight at the top of the trellis, so long as it isn't too much work to do so. The rest might be mulch, but mulch is good too <3

An important note, that I couldn't manage to fold into the essay proper:

I have direct knowledge of at least three people who would like to say positive things about their experience at Leverage Research, but feel they cannot.  People who are fully on board with the fact that their experiences do not erase Zoe's, but whose own very different stories are relevant to understanding what actually went wrong, so that it can be fixed for those who suffered, and prevented in the future.

And at least these three people are not speaking up, because the current state of affairs is such that they feel they can't do so without committing social suicide.

This is a fact about them, not a fact about LessWrong or what would actually happen.

But it sure is damning that they feel that way, and that I can't exactly tell them that they're wrong.

Personally, I think it's clear that there was some kind of disease here, and I'd like to properly diagnose it, and that means making it possible to collect all of the relevant info, not an impoverished subset.

We were missing Zoe's subset.  It's good that we have it now.

It's bad that we're still dealing with an incomplete dataset, though, and it's really bad that a lot of people seem to think that the dataset isn't meaningfully incomplete.

LW is likely currently on something like a Pareto frontier of several values, where it is difficult to promote one value better without sacrificing others. I think that this is true, and also think that this is probably what OP believes.

The above post renders one axis of that frontier particularly emotionally salient, then expresses willingness to sacrifice other axes for it.

I appreciate that the post explicitly points out that is willing to sacrifice these other axes. It nevertheless skims a little bit over what precisely might be sacrificed.

Let's name some things that might be sacrificed:

(1) LW is a place newcomers to rationality can come to ask questions, make posts, and participate in discussion, hopefully without enormous barrier to entry. Trivial inconveniences to this can have outsized effects.

(2) LW is a kind of bulletin board and coordination center for things of general interest to an actual historical communities. Trivial inconveniences to sharing such information can once again have an outsized effect.

(3) LW is a place to just generally post things of interest, including fiction, showerthoughts, and so on, to the kind of person who is interested in rationality, AI, cryonics, and so on.

All of these are also actual values. They impact things in the world.

Some of these could also have essays written about them, that would render them particularly salient, just like the above essay.

But the actual question here is not one of sacred values -- communities with rationality are great! -- but one of tradeoffs. I don't think I understand those tradeoffs even the slightest bit better after reading the above.

Strong upvote. Thank you for writing this, it articulates the problems better than I had them in my head and enhances my focus. This deserves a longer reply, but I'm not sure if I'll get to write it today, so I'll respond with my initial thoughts.

What I really want from LessWrong is to make my own thinking better, moment to moment. To be embedded in a context that evokes clearer thinking, the way being in a library evokes whispers. To be embedded in a context that anti-evokes all those things my brain keeps trying to do, the way being in a church anti-evokes coarse language.

I want this too.

In the big, important conversations, the ones with big stakes, the ones where emotions run high—
I don’t think LessWrong, as a community, does very well in those conversations at all.

Regarding the three threads you list, I, others involved in managing of LessWrong, and leading community figures who've spoken to me are all dissatisfied with how those conversations went and believe it calls for changes in LessWrong.

Solutions I am planning or considering:

• Technological solutions (i.e. UI changes). Currently, I think it's difficult to provide norm-enforcing feedback on comments (you are required to write another comment, which is actually quite costly). One is also torn between signalling agreement/disagreement with statement and approval/disapproval of the reasoning. These issues could be addressed by factoring karma into two axes (approve/disapprove, agree/disagree) and also possible something like "epistemic reacts" where you can easily tag a comment as exemplifying a virtue or vice. I think that would give standard-upholding users (including moderators) a tool to uphold the standards.
  • There's a major challenge in all of this in that I see any norms you introduce as being additional tools that can be abused to win–just selectively call out your opponents for alleged violations to discredit them. This can maybe worked around, say giving react abilities only to trusted users or something, but it's non-trivial.
  • Another thing is that new-users are currently on too-even footing with established users. You can make an account and your comments will look the same as a user who's proven themselves. This could be addressed by marking new users as such (Hacker News does this) or we can create spaces where new users cannot easily participate (more on this in a moment).
  • Not a solution, but a problem to be solved: when it comes to users, high karma can in part indicate highly valuable contributions, but also is just a measure of engagement. Someone with hundreds of low scoring comments can have a much higher score than someone of higher standards with only a few standout posts. This means that karma alone is inadequate to segment out better users from lower standard and quality adhering ones.
• I am interested in creating "Gardens within the Garden". As you say, counting up, LessWrong does well compared to the GenPop but far from sufficient. I think it would be good to have a place where people can level-up, and a further higher-quality space towards which people can strive to be admitted to. Admission could be granted by moderators (likely) or by passing an adequate test (if we are able to create such a one), I imagine you (Duncan) would actually be quite helpful in designing it.
• I think our new user system is woefully inadequate. The system needs to change so that admission to LessWrong as a commenter and poster is not something that is taken for granted, and that new users are made aware that many (most?) new users will be turned away (once their initial contributions seem low enough quality).
  • Standards need to be made clear to new users, and for that matter, they need to be clarified to everyone. This is hard because to to me at least, picking the right standards is not easy. Picking the wrong standards to enforce could kill LessWrong (which I think would be worse than living with the current students).
  • I think that starting by getting the standards for new users clear ("stopping the bleeding") we can then begin to extend that onto the existing user base. As a general approach we (the moderators) have a much higher bar for banning long-term users over new users [1].

This is just the cached list that I'm able to retrieve on the spot. There are surely more good things that I'm forgetting or haven't thought of.

I think it isn’t. I think that a certain kind of person is becoming less prevalent on LessWrong, and a certain other kind of person is becoming more prevalent, and while I have nothing against the other kind, I really thought LessWrong was for the first group.

It is the definitely the case that people who I want on LessWrong are not there because the discussion doesn't meet their standards. They have told me. I want to address this, although it's somewhat hard because the people I want tend to be opinionated about standards in ways that conflict, or at least whose intersection would be a norm-enforcement burden that neither moderators or users could tolerate. That said, I think there are improvements in quality that would be universally regarded as good and would shift the culture and userbase in good directions.

In no small part, the duty of the moderation team is to ensure that no LessWronger who’s trying to adhere to the site’s principles is ever alone, when standing their ground against another user (or a mob of users) who isn’t

I would really like this to be true.

Hire a team of well-paid moderators for a three-month high-effort experiment of responding to every bad comment with a fixed version of what a good comment making the same point would have looked like. Flood the site with training data.

If you can find me people capable of being these moderators, I will hire them. I think the number of people who have mastered the standards you propose and are also available is...smaller than I have been able to locate so far.

Timelines for things happening from LW team
Progress is a little slow at the moment. Since the restructuring into Lightcone Infrastructure [LW · GW], I'm the only full-time member of the LessWrong team. I still get help with various tasks from other Lightcone members, and jimrandomh independently does dev work as an open source contributor; however, I'm the only one able to drive large initiatives (like rescuing the site's norms) forward. Right now the bulk of my focus on hiring [2]. Additionally, I've begun doing some work on the new user process, and I hope to begin are the experiments with karma factorization. Those are smaller steps than what's required, unfortunately.

If you or someone you know is a highly capable software engineer with Rationalist virtue, please contact me. While the community does have many software developers, the number who are skilled enough and willing to live in Berkeley and work on LessWrong is not so high that it's trivial to hire.

--

[1] In the terminology of Raemon, I believe we have some Integrity Debt in disclosing how many new users we ban (and their content that we remove).
[2] It's plausible I should drop hiring and just focused on everything in the OP/I mention above, but I consider LessWrong "exposed" right now since I'm neither technically strong enough or productive enough to maintain the site alone, which makes me reliant people outside the team, which is a kind of brittle way for things to be.

I think that smart people can hack LW norms and propagandize / pointscore / accumulate power with relative ease. I think this post is pretty much an example of that:
- a lot of time is spent gesturing / sermoning about the importance of fighting biases etc. with no particularly informative or novel content (it is after all intended to "remind people of why they care".). I personally find it difficult to engage critically with this kind of high volume and low density. 
- ultimately the intent seems to be an effort to coordinate power against types of posters that Duncan doesn't like

I just don't see how most of this post is supposed to help me be more rational. The droning on makes it harder to engage as an adversary, than if the post were just "here are my terrible ideas", but it does so in an arational way.

I bring this up in part because Duncan seems to be advocating that his adherence to LW norms means he can't just propagandize etc.

If you read the OP and do not choose to let your brain project all over it, what you see is, straightforwardly, a mass of claims about how I feel, how I think, what I believe, and what I think should be the case.

I explicitly underscore that I think little details matter, and second-to-second stuff counts, so if you're going to dismiss all of the "I" statements as being mere window dressing or something (I'm not sure that's what you're doing, but it seems like something like that is necessary, to pretend that they weren't omnipresent in what I wrote), you need to do so explicitly.  You need to argue for them not-mattering; you can't just jump straight to ignoring them, and pretending that I was propagandizing.

If people here really think you can't propagandize or bad-faith accumulate points/power while adhering to LW norms, well, I think that's bad for rationality.

I am sure that Duncan will be dissatisfied with this response because it does not engage directly with his models or engage very thoroughly by providing examples from the text etc. I'm not doing this stuff because I just don't actually think it serves rationality to do so.

While I'm at it:

Duncan:

I'm not trying to cause appeals-to-emotion to disappear.  I'm not trying to cause strong feelings oriented on one's values to be outlawed.  I'm trying to cause people to run checks, and to not sacrifice their long-term goals for the sake of short-term point-scoring.

To me it seems really obvious that if I said to Duncan in response to something, "you are just sacrificing long-term goals for the sake of short-term point-scoring", (if he chose to respond) he would write about how I am making a bald assertion and blah blah blah. How I should retract it and instead say "it feels to me you are [...]" and blah blah blah. But look, in this quote there is a very clear and "uncited" / non-evidentiated claim that people are sacrifiing their long-term goals for the sake of short-term point-scoring. I am not saying it's bad to make such assertions, just saying that Duncan can and does make such assertions baldly while adhering to norms. 

To zoom out, I feel in the OP and in this thread [LW(p) · GW(p)] Duncan is enforcing norms that he is good at leveraging but that don't actually protect rationality. But these norms seem to have buy in. Pooey!

I continuously add more to this stupid post in part because I feel the norms here require a lot of ink gets spilled and that I substantiate everything I say. It's not enough to just say "you know it seems like you are doing [x thing I find obvious]". Duncan is really good at enforcing this norm and adhering to it. 

But the fact is that this post was a stupid usage of my time that I don't actually value having written, completely independent of how right I am about anything I am saying or how persuasive.

Again I submit:

I explicitly underscore that I think little details matter, and second-to-second stuff counts, so if you're going to dismiss all of the "I" statements as being mere window dressing or something (I'm not sure that's what you're doing, but it seems like something like that is necessary, to pretend that they weren't omnipresent in what I wrote), you need to do so explicitly.  You need to argue for them not-mattering; you can't just jump straight to ignoring them, and pretending that I was propagandizing.

Look, if I have to reply to every single attack on a certain premise, before I am allowed to use this premise, then I am not going to be allowed to use the premise ever. Because Duncan has more time than me allocated to this stuff, and seemingly more than most people who criticize this OP. But that seems like a really stupid norm. 

I made this top level because, even though I think the norm is stupid, among other norms I have pointed out, I also think that Duncan is right that all of them are in fact the norm here.

All of which is to say that I spend a decent chunk of the time being the guy in the room who is most aware of the fuckery swirling around me, and therefore the guy who is most bothered by it.

I feel like this claim is being used as evidence to bolster a call to action, without being well supported.  Unsupported evidence is always worse than supported, but it's especially grating in this case because providing counter-evidence can reasonably be predicted to be interpreted as a social attack, which people have inhibitions against making (and then when do they do get made, come out in really unproductive ways). I would prefer a norm where, if you're going to claim raw or relative intelligence as a reason to believe you, you need to provide at least a link to evidence.

Summary: I found this post persuasive, and only noticed after the fact that I wasn't clear on exactly what it had persuaded me of. I think it may do some of the very things it is arguing against, although my epistemic status is that I have not put in enough time to analyze it for a truly charitable reading. 

Disclaimer: I don't have the time (or energy) to put in the amount of thought I'd want to put in before writing this comment. But nevertheless, my model of Duncan wants me to write this comment, so I'm posting it anyway. Feel free to ignore it if it's wrong, useless, or confusing, and I'm sorry if it's offensive or poorly thought out! 

Object-level: I quite liked your two most recent posts on Concentration of Force and Stag Hunts. I liked them enough that I almost sent them to someone else saying "here's a good thing you should read!" It wasn't until I read the comment below [LW(p) · GW(p)] by SupposedlyFun that I realized something slightly odd is going on and I hadn't noticed it. I really should have noticed some twinge in the back of my mind on my own, but it took someone else pointing it out for me to catch it. I think you might be guilty of the very thing you're complaining about in this second essay. I'm not entirely sure. But if I'm right about what this second post is about, you'd want me to write this comment. 

Of course, this is tricky because the problem is I'm not sure I can accurately summarize what this second post is about. The first post was very clear - I can give a one-sentence summary I'd give an >80% you'd agree with as an accurate summary (you can win local battles by strategically outnumbering someone locally without outnumbering them in the war more generally, and since such victories can snowball in social realms, we should be careful to notice and leverage this where possible as it's a more effective use of force). Whereas I only give <30% you'd agree with any attempted summary of this post I could give. In your defense, I didn't click out to all of the comments in the other 3 posts that you give as examples of things going wrong. I also didn't read through the entire post a second time. Both of these should be done for a more charitable reading. On the other hand, I committed a decent amount of time to reading both of these essays all the way through, and I imagine anything more than that is a slightly unreasonable standard for effort to understand your core claim. 

I have something like the vague understanding that you think LW is doing something bad, you want less of it, and more of something better. Maybe you merely just want more Rationality and I'm not missing anything, but I think you're trying to make a more narrow point and I'm legitimately not sure what it is. I get that you think the recent Leverage drama is not a good example of Rationality. But without following a number of the linked comments, I can't say exactly what you think went wrong. I have my own views of this from having followed the Leverage drama, but I don't think this should be a prerequisite to understanding the claims in your post. 

Your comment below [LW(p) · GW(p)] provides some additional nuance by giving this example: "I have direct knowledge of at least three people who would like to say positive things about their experience at Leverage Research, but feel they cannot." Maybe the issue is you merely need to provide more examples? But that feels like a surface-level fix to a deeper problem, even though I'm not sure what the deeper problem is. All I can say is that I left the post with an emotion (of agreement), not a series of claims I feel like I can evaluate. Whereas your other posts feel more like a series of claims I can analyze and agree or disagree with. What's particularly interesting is I read the essay through and I was like "Yeah Duncan woo this is great, +1" and I didn't even notice I didn't know precisely the narrow thing you're arguing for until I read SupposdelyFun's comment saying the same. This suggests you might be doing the very thing (I think) you're arguing against: using rhetoric and well-written prose to convince me of something without my even knowing exactly what you've convinced me of. That the outgroup is bad (boo!) that the warriors for rationality are getting outnumbered (yikes!) and that we should rally to fix it (huzzah!).

I'm not entirely sure. My thinking around this post isn't clear enough to know precisely what I'm objecting to, but I'm noticing a vague sense of confusion, and I'm hoping that pointing it out is helpful. I do think that putting out thinking on this topic is good in general, and meta-discussion about what went wrong with the Leverage conversation seems sorely needed, so I'm glad that you're starting a conversation about it (despite my comments above).

My brain notes, in passing, that Eternal September, and the Septembers that preceded it, can be described in terms of concentration of force.  If a forum has a certain culture, and a bunch of noobs come in without it (instead exhibiting some kind of "mainstream lowest-common-denominator" culture, incompatible with that of the forum)... If they come in one by one, then they'll face negative reinforcement (downvotes and/or critical comments), pushing them to either adopt the forum's culture or leave; if they arrive in a clump, then they'll be in a position to positively support each other, reducing the negative-reinforcement effect.  If enough of them arrive in a group, then the forum's "immune system" may fail to stop them, and they may end up changing the forum's culture.

Today, a sudden influx of users tends to come from some big event.  Like when a hugely popular site drops a link to your forum, or when there's a huge story on your forum that draws a lot of interest from outsiders.  (The Leverage story is one of the latter; a story like Zoe Curzi's is fascinating to humans.  I told a non-rationalist friend about it, and he said it was very juicy gossip.  It also came up in discussions with rationalists, some of whom are more active than others on LW; I wonder if the posts were also linked to on other sites.  I wonder if LW admins could confirm patterns like "The recent huge threads about Leverage and about MIRI had a higher proportion of non-users, of new users, and of less-regular users than most other threads." HTTP referrer headers can give the proximate source of inbound links, while interpersonal gossip is harder to track.  Actually, regarding links, habryka posted an image [LW(p) · GW(p)] recently [incidentally, the top post, "I don't know how to count that low", is an example of getting linked to by Hacker News] ... but although it's interesting, I don't think it directly addresses my above question.)

So, it seems like sudden large influxes of users, attributable to big stories or big inbound links, are a danger to a forum's culture.  That seems to be "common/received wisdom" from some portions of the internet I've occupied.

This brings to mind a funny episode from Hacker News's history, where its creator posted something like "We've gotten written about by a major news site and will probably be flooded by mainstream people who want to talk about politics and don't care about technical subjects, so please make an extra effort to post and upvote links about Erlang internals and things like that to discourage those people", and existing literal-minded users took this to heart and filled the front page with nothing but Erlang links.

Anyway, combining one of the proposals with "concentration of force", this generates the following idea: If you have something like "dedicated volunteers or paid posters", then by far the best time to deploy them is when you have what looks like one of these big influxes.  (It seems an obvious enough combination that I'm mildly surprised that this wasn't on Duncan's list of terrible-idea suggestions.  Perhaps because it goes against the focus on small things?  Heh, Gunnar_Zarncke has the same suggestion [LW(p) · GW(p)].)  My impression is that existing moderators did put a bunch of time into following the huge threads; but, like, if you had a squad of "reserve forces" of extra moderators kept on retainer, who only get called in occasionally, that's probably more efficient and effective.

First of all, thank you, Duncan, for this post. I feel like it captures important perspectives that I've had, and problems that I can see and puts them together in a pretty good way. (I also share your perspective that the post Could Be Better in several ways, but I respect you not letting the perfect be the enemy of the good.)

I find myself irritated right now (bothered, not angry) that our community's primary method of highlighting quality writing is by karma-voting. It's a similar kind of feeling to living in a democracy--yes, there are lots of systems that are worse, but really? Is this really the best we can do? (No particular shade on Ruby or the Lightcone team--making things is hard and I'm certainly glad LW exists and is as good as it is.)

Like, I think I have an idea that might make things substantially better that's not terrible: make the standard signal for quality being a high price on a quality-arbitrated betting market. This is essentially applying the concept of Futarchy to internet forums (h/t ACX and Hanson). (If this is familiar to you, dear reader, feel free to skip to responses to this comment, where I talk about features of this proposal and other ideas.) Here's how I could see it working:

When a user makes a post or comment or whatever, they also name a number between 0 and 100. This number is essentially a self-assessment of quality, where 0 means "I know this is flagrant trolling" and 100 means "This is obviously something that any interested party should read". As an example, let's say that I assign this comment an 80.

Now let's say that you are reading and you see my comment and think "An 80? Bah! More like a 60!" You can then "downvote" the comment, which nudges the number down, or enter your own (numeric) estimate, which dramatically shifts the value towards your estimate (similar to a "strong" vote). Behind the scenes, the site tracks the disagreement. Each user is essentially making a bet around the true value of the post's quality. (The downvote is a bet that it's "less than 80".) What are they betting? Reputation as judges! New users start 0 judge-of-quality-reputation, unless they get existing users to vouch for them and donate a bit of reputation. (We can call this "karma," but I think it is very important to distinguish good-judge karma, from high-quality-writing karma!) When voting/betting on a post/comment, they stake some of that reputation (maybe 10% up to a cap of 50? (Just making up numbers here for the sake of clarity; I'd suggest actually running experiments)).

Then, you have the site randomly sample pieces of writing, weighting the sampling towards those that are most controversial (ie have the most reputation on the line). Have the site assign these pieces of writing to moderators whose sole job is to study that piece of writing and the surrounding context and to score its quality. (Perhaps you want multiple moderators. Perhaps there should be appeals, in the form of people betting against the value set by the moderator. Etc. More implementation details are needed.) That judgment then resolves all the bets, and results in users gaining/losing reputation.

Users who run out of reputation can't actually bet, and so lose the ability to influence the quality-indicator. However, all people who place bets (or try to place bets when at zero/negative reputation) are subsidized a small amount of reputation just for participating. (This inflation is a feature, encouraging participation in the site.) Thus, even a new user without any vouch can build up ability to influence the signal by participating and consistently being right.

First, I think promoting and encouraging higher standards is, if you'll pardon the idiom, doing God's work. 

Thank you. 

I'm so appreciative any time any member of a community looks to promote and encourage higher standards. It takes a lot of work and gets a lot of pushback and I'm always super appreciative when I see someone work at it.

Second, and on a much smaller note, if I might offer some......... stylistic feedback?

I'm only speaking here about my personal experience and heuristics. I'm not speaking for anyone else. One of my heuristics — which I darn well know isn't perfectly accurate, but it's nevertheless a heuristic I implicitly use all the time and which I know others use — is looking at language choices made when doing a quick skim of a piece as a first-pass filter of the writer's credibility.

It's often inaccurate. I know it. Still, I do it.

Your writing sometimes, when you care about an issue, seems to veer very slightly into resembling the writing of someone who is heated up about a topic in a way that leads to less productive and coherent thought.

This leads my default reaction to discounting the credibility of the message slightly.

I have to forcibly remind myself not to do that in your case, since you're actually taking pretty cohesive and intelligent positions. 

As a small example:

These are all terrible ideas.

These are all

terrible

ideas.

I'm going to say it a third time, because LessWrong is not yet a place where I can rely on my reputation for saying what I actually mean and then expect to be treated as if I meant the thing that I actually said: I recognize that these are terrible ideas.

I just — umm, in my personal... umm.... filters... it doesn't look good on a skim pass. I'm not saying emulate soul-less garbage at the expense of clarity. Certainly not. I like your ideas a lot. I loved Concentration of Force. 

I'm just saying that, on the margin, if you edited down some of the first-person language and strong expressions of affect a little bit in areas where you might be concerned about it being "not yet a place where I can rely on my reputation for saying what I actually mean"... it might help credibility.

I've written quite literally millions of words in my life, so I can say from firsthand experience that lines like that do successfully pre-empt stupid responses so you get less dumb comments.

That's true.

But I think it's likely you take anywhere from a 10% to 50% penalty to credibility to many casual skimmers of threads who do not ever bother to comment (which, incidentally, is both the majority of readers and me personally in 2021).

I see things like the excerpted part, and I have to consciously remind myself not to apply a credibility discount to what you're saying, because (in my experience and perhaps unfairly) I pattern match that style to less credible people and less credible writing.

Again, this is just a friendly stylistic note. I consider myself a fan. If I'm mistaken or it'd be expensive to implement an editing filter for toning that down, don't bother — it's not a huge deal in the grand scheme of things, and I'm really happy someone is working on this.

I suppose I'm just trying to improve the good guys' effectiveness for concentration of force reasons, you could say.

Salut and thanks again.

Some thoughts on resource bottlenecks and strategy.

There's a lot I like about the set of goals Duncan is aiming for here, and IMO the primary question is one of prioritization.

I do think some high-level things have changed since 2018-or-so. Back when I wrote Meta-tations on Moderation [LW · GW], the default outcome was that LW withered and died, and it was really important people move from FB to LW. Nowadays, LW seems broadly healthy, the team has more buy-in, and I think it's easier to do highly opinionated moderation more frequently for various reasons.

On the other hand, we did just recently refactor the LW team into Lightcone Infrastructure. Most of the team is now working on a broader project of "figure out the most important bottlenecks facing humanity's ability to coordinate on x-risk, and building things that fix that bottleneck" (involving lots of pivoting). Ruby is hiring more people to build more capacity on the LW team but hiring well is a slow process. And most of the plans that seem to accomplish (some version of) what Duncan is pointing to here seem really expensive.

The good news is that we're not money-constrained much these days. The biggest bottlenecked resource is team-attention. When I imagine the "hire a bunch of moderators to full-time respond to every single comment" (not an inherently crazy idea IMO), the bottleneck is vetting, hiring, training and managing those moderators. 

I do think "which standards exactly, and what are they aiming for?" is a key question. A subproblem is that rigidly pushing for slightly misaligned standards is really infuriating and IMO drives people away from the site for reasons I don't think are good. Part of reason I think hiring moderators is high effort is that I think a bad (or, "merely pretty good") moderator can be really annoying and unhelpful.

I am pretty optimistic about technological solutions that don't require scaling human attention (and I do think there's a lot of low-hanging fruit there). 

Brainstorming some ideas:

• Users get better automated messaging about site norms as they first start posting. Duncan and Ruby both mentioned variants of this.
  • One option is that in order to start growing in vote power, a user has to read some stuff and pass some kind of multiple choice test about the site norms. (I might even make it so everyone temporarily loses the ability to Strong Vote until they've taken the test)
• Checking if a user has gotten a moderate amount of net-downvotes recently (regardless of total karma), and some combo of flagging them for mod-attention, and giving them an automated "hey, it seems like something has been up with your commenting lately. You should reflect on that somehow [advice on how to do so, primary suggestion being to comment less frequently and more thoughtfully]. If it keeps up you may temporarily lose posting privileges."
• FB-style reacts that let people more easily give feedback about what's wrong with something without replying to it.

I think you are getting at something here, Duncan. I've become interested in the following question lately: "How should rationalists conduct themselves if their goal is to promote rationality [? · GW]?" Now, I understand that promoting rationality is not every rationalist's top priority, hence I stated that condition explicitly.

I've been thoroughly impressed by how Toby Ord conducts himself in his writings and interviews. He is kind, respectful, reassuring and most importantly, he doesn't engage in fear-mongering despite working on x-risks. In his EA interview, he said, "Let us not get into criticising each other for working on the second most important thing." I found this stunningly thoughtful and virtuous. I find this to be an excellent example of someone going about achieving their goals effectively.

As much as I like Dawkins and love his books, I will admit that his attitude is sometimes unhelpful towards his own goals. I recall hearing (forgot where) that before a debate on spirituality, Dawkins' interlocutor asked him to read some documents ahead of the discussion. Dawkins showed up having not read the papers and said, "I did not read your documents because I know they are wrong." [citation needed] Now, this attitude might have been amusing to some in the audience, but, on the whole, this is irrational given the goal is to promote science.

Whenever I engage in motivated reasoning, motivated scepticism or subtle ad hominem in an argument, I can feel it. It feels like I am making a mistake. I feel a vague sense of guilt and confusion in the back of my head and a lump in my throat upon engaging in such conduct. I like the idea of leaning into confusion, which I recall coming across somewhere in the sequences and elsewhere on LessWrong. Still, I would like to become more proficient at actively avoiding these mistakes in the first place.

Since it was posted, I have been closely following the My experience at and around MIRI and CFAR [LW · GW] post, but I didn't know who or what to believe. Anecdotes were being thrown like hotcakes and every which way. Given this confusion, I became more interested in learning a lesson from the situation than picking sides, debunking claims or pointing fingers at culprits.

Suggestions for LW features that could shape its culture by (dis)incentivizing certain behavior (without thinking about how hard they would be to implement):

On How LW Appears to Outside Readers

There's a certain kind of controversial post that inevitably generates meta-discussion of whether people should be allowed to post it here (most recently in this book review [LW · GW]). Crucially, the arguments I see there are not "I don't like this" but usually "I'm afraid of what will happen when people who don't like this see it, and associate LW with it". (I found this really tedious, and would prefer a culture where we stick our own heads out and stick to "I don't like this" rather than appealing to third parties.) Also, I wish there were a way to preempt this objection, so as to not fight the same battle over and over.

Other posts are in dispute (as in, have an unusually high fraction of downvotes), like jessicata's post [LW · GW], but a casual reader might only see the post (and maybe its positive karma score), with all the controversy and nuance happening in the utterly impenetrable comments section.

So what might one do about that?

• Some Reddit-style sites compute a controversy score (percentage of downvotes). Then one could sort by controversy, or by default filter controversial posts from the frontpage, or visibly flag controversial posts in a way people not familiar with LW would understand, or tag as "controversial", or something.
• If controversial posts are rare in number, this problem can also be tackled by mod intervention. For instance, mods could have the power to put a disclaimer (/ disclaimer template) above a post, or these could be triggered automatically by some specific metrics. Some (bad) examples:
  • "This post has a notable fraction of downvotes, and a high number of nested comment threads. This indicates that it's in dispute. Check the comments for details."
  • Or: a manual or automatic disclaimer on high-traffic non-frontpaged private blog posts (like the book review [LW · GW]) to indicate to readers who come from elsewhere: This post is not frontpaged. LW has less strict moderation standars for private blogs. The karma score (upvotes) of posts reflect positive sentiment towards the contribution by the poster (e.g. taking the time to write a review), they're not automatically an endorsement of <the reviewed item>. And so on.

Handling High-Stakes Controversies

Duncan's post notes various ways in which recent controversies were not handled optimally. Some (probably bad) suggestions for site features to help handle such situations better:

• The disclaimer thing, so new readers know that a post is controversial before they read it and take its claims at face value. This might also be warranted for some high-karma controversial comments.
• Moderation
  • Mods could try to "turn down the heat" by setting stricter commenting guidelines, or temporarily prevent users below some karma threshold from posting, or something.
  • Flagging important clarifying comment threads so they appear higher in the comment order, or with extra highlighting.
  • Mark comments by moderators-acting-as-moderators with a flair or other highlighting.
• On-site low-friction ways to post anonymously, with the option of leaking some non-identifying information like "I've been a LW user for >3 years with >1k karma", or to ping specific LW users with "user X can vouch for my identity", which user X could then confirm with a single click. Though I would not want this anonymity feature available on non-controversial posts.
• A feature for a user to "request moderation for this post", or "request stricter commenting guidelines" or to indicate "this post seems controversial to me" or something.

Of course such features don't solve a problem by themselves, but they can help, and I'm more optimisic of attempts to improve a culture if the site infrastructure supports those attempts and incentivizes that improved culture.

Rewarding Exceptional Content

As noted in the LW book review bounty program [LW · GW], exceptional content on LW is potentially very valuable, so it makes sense to incentivize it. The karma system helps, but it's not enough - as can be seen when extra incentives come into play, e.g. the extra reviews [LW · GW] generated by the bounty program.

Some features beyond the karma system that could help here:

• Reddit nowadays has a separate "awards" system which users can use to reward exceptional content. I don't like the specific implementation at all - it's full of one-upmanship of progressively more expensive awards, and posts with lots of awards just look cluttered - but one could imagine an implementation that would work here.
• For instance, there are a number of active bounties [? · GW] on LW, but one could imagine a smaller-scale version of setting and rewarding bounties that would work better if built into the site, e.g. for Question posts (to reward the best answer), or just to gift someone money for writing a particularly important post or comment.
  • That said, this is the kind of thing that, if implemented suboptimally, could easily incentivize detrimental behavior, instead.
• Or (like mentioned in the Controversies section), what about a system for users or mods to flag particularly high-quality or high-importance comments so they get some extra highlighting or something?

Related:

Because comments are much less discoverable than posts, lots of high-effort high-value comments, even if they're very-high-karma, get lost in the masses of LW comments and are hard to find or refer to later on.

What could be done about that?

• For instance, if users or mods see a comment thread of exceptional and enduring value, they could flag it as such (I already occasionally see follow-up comments of the form "This is good enough for a top-level post!"), and then others (volunteers or paid contributors) could turn the best ones of those into top-level posts, with karma going to the original posters.

(To end on a meta comment: I spent >2.5h on three high-effort comments in this thread, and would be disappointed if they got lost in the shuffle. Conversely, I'm more likely to make the effort in the future if I have a sense that it paid off in some way.)

For what it's worth, I think some of those terrible ideas are great or close to great.

In particular:

• Hire a team of well-paid moderators for a three-month high-effort experiment of responding to every bad comment with a fixed version of what a good comment making the same point would have looked like.  Flood the site with training data.
• Make a fork of LessWrong run by me, or some other hopeless idealist that still thinks that there might be something actually good that we can get if we actually do the thing (but not if we don't).
• Create an anonymous account with special powers called TheCultureCurators or something, and secretly give the login credentials to a small cadre of 3-12 people with good judgment and mutual faith in one another's good judgment.  Give TheCultureCurators the ability to make upvotes and downvotes of arbitrary strength, or to add notes to any comment or post à la Google Docs, or to put a number on any comment or post that indicates what karma TheCultureCurators believe that post should have.

Rob Bensinger wants me to note that he agrees.

The first one would be costly and annoying to lots of people but also time boxed and super interesting. Training data is really good, and very pedagogically valuable.

The second one just seems low cost to everyone except the idealist, so if they're willing, great!

The third would be controversial and complicated, but for instance putting a number for what karma they think it should have wouldn't change the current voting system and add information and could be time boxed like the first one.

Mostly I appreciate just the generation of lots of ideas to give my brain more to chew on and a sense that bigger things are possible.

Also more generally I really resonate with "dear God, I need the other people around me to be good at this to be my best self."

I'm curious what you and others think of Raelfin's post about the karma system: https://www.lesswrong.com/posts/xN2sHnLupWe4Tn5we/improving-on-the-karma-system
 

↑ comment by Duncan Sabien (Deactivated) (Duncan_Sabien) · 2021-11-19T21:53:12.907Z · LW(p) · GW(p)

I like it, but not as much as I like two-axis proposals, which I think can be done with a smooth enough UI that they don't impose a burden.

With something like the below, you can click to weak vote and hold to strong vote, just like we currently do, and can in one click express each of the four following positions:

• I like/agree with this point, and furthermore think it's being expressed correctly/is in line with norms of reasoning and discourse I want to see more of on LW (dark blue)
• I like/agree with this point, but I want to note objection with how it's being expressed/have reservations about whether it's good rationality or good discourse (pale orange)
• I dislike/disagree with this point, and want to note objection with how it's being expressed (dark orange)
• I dislike/disagree with this point, but want to endorse/support the way it was arrived at and expressed (pale blue)

Replies from: cmessinger

↑ comment by chanamessinger (cmessinger) · 2021-11-19T21:56:10.451Z · LW(p) · GW(p)

Oh yeah, I've seen you post this before, I liked it!

Replies from: Duncan_Sabien

↑ comment by Duncan Sabien (Deactivated) (Duncan_Sabien) · 2021-11-19T22:15:12.256Z · LW(p) · GW(p)

Nuance is the cost of precision and the bane of clarity. I think it's an error to feel positively about nuance (or something more specific like degrees of uncertainty), when it's a serious problem clogging up productive discourse, that should be burned with fire whenever it's not absolutely vital and impossible to avoid.

There is a part of Sequences which I am too lazy to find now, which goes approximately like this: "If you make five maps of the same city, and you make those maps correctly, then the maps should be the same. So if you make five maps of the same city, and you find differences between them (for example, some streets A and B intersect on one map, but run parallel on another map), it means that you made a mistake somewhere, and the maps are not as good as you wish them to be. Nonetheless, you cannot fix this mistake by merely adjusting some of the maps to fit the other ones. The sameness of the maps is a desired outcome... but it must happen naturally, as a result of all maps correctly representing the same city... not artificially, as a result of adjusting the maps to fit each other."

I get a similar feeling from some of your posts (including this one, also the punch bug). It seems to me that you care a lot about being right; and that is a good thing, and it's kinda what this community is trying to be about. And you seem strongly frustrated by people coming to conclusions dramatically different from yours; which indeed means that someone is wrong about something. And I agree that this is frustrating, and it is something that wouldn't happen if we succeeded at our stated goal.

But... it happens. And I worry that if we push against this, we may be "Goodharting" our search for truth. The common agreement should happen as a result of everyone examining the facts carefully and rationally. Not as a result of peer pressure that coming to the common agreement is what rationalists should do. Like, they "should" in the sense that "this is what would automatically happen to perfectly rational agents, per Aumann's theorem", but not "should" in the sense that "they should be directly trying to achieve this".

So, disagreeing with a specific thing seems okay to me. ("You guys said X, but I am convinced that non-X, here is my evidence.") But this kind of meta commentary feels to me like pressure to do the wrong thing. ("You guys disagree with me on X, Y, Z. As aspiring rationalists, we should not be regularly disagreeing on so many things.") Even if I agree that disagreeing on too many things too often is evidence that something went wrong. Still, the only correct way to agree on X, Y, Z, is to separately discuss X and come to a conclusion, discuss Y and come to a conclusion, and discuss Z and come to a conclusion. Not some bulk update like: "shit, it's really bad to disagree on so many things, and Duncan seems to have a lot of support so I guess the wisdom of the crowd is on his side, so from now on I am going to automatically switch my opinion on everything to whatever Duncan says". Convince me by arguments, not by meta-arguments about how disagreement is wrong.

And this is not intended as a defense of any specific things I said that you might disagree with. I am just a stupid human, with limited time and attention, often posting past midnight when I should be sleeping instead. It is plausible that I am wrong at a horribly large amount of things. Sorry for that. But there is also a chance that you might be wrong about some things, or maybe we just misunderstand each other, so my updating to your (perceived) position might also be a mistake. I am also unhappy about us not being better synchronized in our perspectives, but I see it as an inevitable consequence of our imperfections, and I hope it gets better over time... but I am not going to force it.

If you convince me about specific mistakes, I hope I have the ability to change my mind. If you convince me about an embarassing number of mistakes, I might even conclude that it is better for humanity if I simply stop writing, at least until my game improves significantly. But the meta-argument itself is not actionable for me (other than the general exhortation to be more careful, which I believe I am already trying).

I agree that it is important to defend and concentrate the force of rationality.

However, I think the difficulty in the original context that this post is about might be that community drama is hard to address in a good way. One major constraint in understanding any topic is having a high flow of representative or unusually useful information from the topic. But for community drama, this is really hard, because it interferes in people's privacy, it involves events that happened in the past, it involves adding up many small interactions, it may be dependent on social relationships, etc..

As such, I think it's going to be really hard for anyone to prove or disprove the validity of complaints about community dynamics in online writing. The best approach I can think of right now is to just allow people to speak up about their impressions without too much burden of proof, to test if others have the same impressions. Maybe there are other, better approaches, but I don't think standards of argumentation can solve them without doing something about the information bottleneck.

Reading this post I kinda feel like you are failing to take your own advice about gardening in a certain way.

Like it feels good to call people out on their BS and get a conversation going, but also I think there's some alternative version of these two posts that's not about building up a model and trying to convince people that something is wrong that must be done about it, and instead a version that just fights the fight at the object level against particular posts, comments, etc. as part of a long slog to change the culture through direct action that others will see and emulate through a shift in the culture.

My belief here is that you can't really cause much effective, lasting change by just telling people they're doing something that sucks and is causing a problem. Rarely will they get excited about it and take up the fight. Instead you just have to fight it out, one weed at a time, until some corner of the garden is plucked and you have a small team of folks helping you with the gardening, then expanding from there.

If LW is not that place and folks don't seem to be doing the work, then maybe LW is simply not the sort of thing that can be what you'd like it to be. Heck, maybe the thing you'd like to exist can't even exist for a bunch of reasons that aren't currently obvious (I'm not sure about this myself; haven't thought about it much).

My own experience was discovering several years ago that, yeah, rationalists the community actually kinda suck at the basic skills of rationality in really obvious ways on a day to day basis. Like forget big stuff that matters like AI alignment or community norms. They just suck at like figuring out how not to constantly have untied shoelaces and other borning, minor drags on their ability to be effective agents. And that's because they're just human, with no real special powers, only they intellectually know a thing or two about how in theory they could be more effective if only they could do it on a day to day basis and not have to resort to a bunch of cheap hacks that work but also cause constant self harm by repeatedly giving oneself negative feedback in order to beat oneself into the desired shape.

Now of course not all rationalists, there's plenty of folks doing great stuff. But it's for this reason I basically just see LW as one small part that can play a role in helping a person become more fully realized in the world. And a part of that is arguably a place where folks can just kinda suck at being rationalists and others can notice this or not or not notice it for a long time and others go off in anger. Maybe this seems kinda weird, but I hold this sense that LW can only be the thing it's capable of being, because that's all any group can do. For example, I don't expect my Zen community to help me be better at making calibrated bets; it's just not designed for that and any attempt to shove it in probably wouldn't work. So maybe the thing is that the LW community just isn't designed to do the things you'd like it to do, and maybe it just can't do those things, and that's why it seems so impossible to get it to be otherwise.

Likely there is some community that could do those things, but it's hard to see how you could pull that out of the existing rationalist culture by a straight line. Seems more likely to me to require a project on the order of founding LW than on the order of, say, creating LW 2.0.

I'd like to throw out some more bad ideas, with fewer disclaimers about how terrible they are because I have less reputation to hedge against.

Inline Commenting

I very strongly endorse the point that it seems bad that someone can make bad claims in a post, which are then refuted in comments which only get read by people who get all the way to the bottom and read comments. To me the obvious (wrong) solution is to let people make inline comments. If nothing else, having a good way within comments to point to what part of the post you want to address feels like a strict win, and given that we already have pingbacks I think letting sufficiently good comments exist alongside the post would also be good. This could also be the kind of thing that a poster can enable or disable, and that a reader can toggle visibility on.

Personal Reputation

I don't have great models for how reputation should or does work on LessWrong. The second of these is testable though - I'd be curious to see what happened if prominent accounts, before commenting, flipped a coin, and in half of all cases posted through a random alt. Of course it may not be a bad thing if respected community figures get more consideration, but it would just be interesting to know how much of an effect it had. There are loads of obvious ways to hedge against this, all themed around anonymisation at different levels, but I think here it's less of a 'can' and more of a 'should', so I'd be curious to hear anyone else's thoughts on that.

Curated Comments

I agree that there are comments that are epistemically not so great. There's some underlying, very complicated question about 'who gets to decide what comments people should read', and I have some democratic instinct which resists any centralisation. But it does feel like some comments are notably higher-effort, or particularly at risk of brigading. I reckon a full prediction market-style moderation system would be a mess, but it seems like it wouldn't be that hard if, when someone made a comment, they could submit it for curation as 'a particularly carefully considered, relevant, and epistemically hygienic response' which, if approved, would be bumped above non-curated comments, with some suitable minor penalty for failed attempts or notes of feedback.

Debate as a model

In formal debate (or at least the kind I did) you distinguish between a point of information and a point of order. When you try to lodge a point of information, the opposing speaker can choose whether they'd like to be interrupted, and you're just interjecting some relevant facts. A point of order, though, is made to the chair, when there's a procedural violation, and it can very much interrupt you. I'm not sure how you'd extend this to lesswrong but it feels like a useful distinction in a similar context.

On existing LW systems, and on how they shape and incentivize discussions here:

• Moderation-by-moderator is expensive and doesn't scale. So sites like Reddit or Less Wrong use a karma system. Some impacts of this system:
• Whether a post or comment is good or bad is determined by a single scalar, i.e. karma.
  • So how do you vote on a comment you consider important and valuable for the most part, but which contains one sentence you consider very wrong? Maybe upvote it while explaining your disagreement?
  • Or what do you do with comments that seem high-effort but very wrongheaded? I want to incentivize effort, even if it occasionally produces wrongheaded results; but upvoting would suggest agreement.
  • What if a comment looks correct and receives lots of upvotes, but over time new info indicates that it's substantially incorrect? Past readers might no longer endorse their upvote, but you can't exactly ask them to rescind their upvotes, when they might have long since moved on from the discussion.
  • How much karma a comment gets depends to a significant extent on how many views it gets, i.e. on a) how early it's posted, b) on how much traffic the post will get overall, and c) on whether it's a top-level comment or a nested comment. So you can't 100% distinguish a comment of high quality vs. one which was just seen a lot. Conversely, if a comment has little karma, that may be because people haven't seen it (e.g. because it's new), or because people have seen it and not considered it valuable. How could one tell the difference?
• There are various tensions between posting one big comprehensive post or comment (e.g. so all discussion is in the same place) vs. several small ones (e.g. so people can vote on or respond to separate parts separately). This is related to both the karma system and the threaded-comments system.
• Posts vs. comments: Posts are much easier to discover and reference than comments, and conversely some valuable site meta discussions get lost in sub-sub-sub comment threads.
• Effortful comments just take a ton of time. For example, my comments in this thread have so far easily taken >2h to write.
• Everywhere on the Internet, New is considered Better (e.g. LW posts get traffic from Google and Hacker News, both of which prefer new content). This has various consequences, like most discussion on a post happening in the first few hours or days.

And so on. Will post suggestions in a separate thread.

A few related thoughts: if we do push farther upon the epistemic hygiene axis, it may be worth writing a Sequence on "Living Epistemically Clean" or something along those lines, so that the standards for discussion are clear and we have a guide to upholding those standards. Specification and implementation, if you will. Such a sequence could potentially also cover, say, implementing TAPs for epistemic hygiene in the outside world. It would probably be useful to have more info on how to address your list of "things your brain tries to do" as a community.

I notice that I value some way of allowing new users to grow accustomed to writing on LW. I imagine that my first post may not be epistemically clean enough to pass the standards you're pointing towards (which isn't a point against your standards) and that others new to the site may be discouraged by, say, a ban. The help reviewing drafts that LW currently offers helps, and that's probably a useful place to focus efforts towards helping people uphold the standards you want to put up. Potentially have multiple levels of users--specific karma amounts or moderator approval would be required to comment in some areas?

Slightly meta: I notice that this post seems to come off as somewhat finger-pointing-towards-moon-like. It works for me, because I see what you're pointing towards, but it looks like some [LW(p) · GW(p)] others don't, and therefore the post comes off as filled with applause lights and no substance. Or I might just be failing their ITT. I'm not entirely sure how to resolve the missed communication, but I figured it might be worth mentioning.

Misc. thoughts on the object level:

• I understand that the community should (and hopefully could) do better, to optimize for a certain culture, that the status quo is Not Good Enough. However, I'm not clear on how widespread you consider this problem to be. My personal impression is that there are <5 controversial posts per month, posts which demand a ton from LW's culture. These are high-stakes events where we noticeably fall short of what is asked of us, and so such occasions could hence benefit a lot from e.g. better culture, or more explicit moderation. The rest of the site, it seems to me, is handled decently well by the karma system. What's your take on that?
  • E.g. regarding several of your explicitly-acknowledged-as-terrible suggestions: I'm not clear on what kind of comment you have in mind that would warrant a temporary ban but which would not get downvoted into oblivion. And similarly, the "team of well-paid moderators" part suggests to me you see a site-wide problem, not just a "we fall short in the high-stakes cases" problem.
  • In any case, if the problem is specifically the high-stakes controversial posts, that opens up a different class of solutions: e.g. a mod could flag a post as controversial, which would also trigger a stricter commenting policy or something.
• To expand on this impression of mine: I was surprised by this [LW(p) · GW(p)] illustration of how big LW is nowadays.
  • To put this into perspective for a non-representative but popular post: this [LW · GW] post was briefly on Hacker News, got >17k unique pageviews, and now has 93 karma with 55 votes, and 10 comments by 8 unique commenters. So there are orders of magnitude between the number of passive readers, the number of votes, and the number of comments.
  • Conversely, jessicata's post on MIRI [LW · GW] has >8k unique pageviews, 61 karma with 171 votes, and 925 comments by I-will-*not*-count-those unique commenters.
  • I file these not-randomly-chosen examples in the to-me-natural-seeming categories "controversial posts in need of better culture", versus "uncontroversial posts with little active engagement".
• Regarding your fully-acknowledged-as-terrible suggestions: There is a tradeoff between LW being inviting for newcomers, versus being the best it could be for veterans. Any community desperately needs new blood to survive. And... I don't really think we can demand much more from newcomers (who after all haven't yet had a chance to absorb the culture here)? I occasionally see comments of the form "I'm too intimidated by the high-quality discussions on this site, so I never post myself". I'm a veteran myself, so-to-speak, having registered on the site 8 years ago, and yet I only wrote my first effortful post this year (partly incentivized by the new feedback system [LW · GW]). I am in favor of attempts that make participating here easier, not harder, and would not wish to trade that off for a better culture. (So I'd e.g. much prefer incentives which reward good behavior, rather than punish poor behavior, so as not to discourage newcomers.)

I have more thoughts, including suggestions of my own, but will post those separately.

$ pbpaste | egrep '\[-\]' | sed -E 's/[0-9]+[a-z]+$//' | sort | uniq -c | sort -nr
  62 [-]jessicata
  46 [-]Benquo
  41 [-]ChristianKl
  36 [-]Unreal
  35 [-]Duncan_Sabien
  32 [-]Viliam
  30 [-]habryka
  22 [-]TekhneMakre
  22 [-]AnnaSalamon
  20 [-]Rob Bensinger
  ...

Recap:  Concentration of force is

At each relevant moment, you want to project locally superior or overwhelming force. having the most [resources] actually present, or perhaps this just means having the right [resources] pointed in the right directions.

Taking this analogy to forum moderation, this seems to mean the ability 

• to call in a sufficient number of participants
• of suitable qualification (e.g., moderator or specific skill)
• on short notice
• to police a particular post.

Policing in the civilian sense of e.g., the original Metropolitan Police)  

Additionally, information is needed: The ability to detect posts requiring policing, but I understand that the LW moderation team already has this ability. 

Typo?

In fact, we're honestly well past the 80/20—LessWrong is at least an 85/37 by this point

One thing we are working on in the Guild of the ROSE is a sort of accreditation or ranking system, which we informally call the "belt system" because it has many but not all of the right connotations. It is possible to have expertise in how to think better and it's desirable to have a way of recognizing people who demonstrate their expertise, for a variety of reasons. Currently the ranking system is planned to be a partly based on performance within the courses we are providing, and party based in objective tests of skill ("belt tests"). But we are still experimenting with various ideas and haven't rolled it out.

I'm confused.

In the counterfactual where lesswrong had the epistemic and moderation standards you desire, what would have been the result of the three posts in question, say three days after they were first posted? Can you explain why, using the standards you elucidated here?

(If you've answered this elsewhere, I apologize).

Full disclosure: I read all three of those posts, and downvoted the third post (and only that one), influenced in part by some of the comments to that post.

I guess long-time lurkers/new posters like me are part of the problem(though obviously I assume most online only LW members didn't engage with a California drama post). I still think LW is a great place for discussion and just being exposed to new ideas and good feedback, but I'm probably dragging down the sanity level.

Re fear: I think the SSC situation made it clear that LW and rationalist adjacent spaces are more public than users might think, maybe people are hesitant because they don't want to get twitter blasted or show up as a snap in an NYTimes article two years down the line.

Re concentration of force: I would imagine raw censorship would be really hard and contentious to enforce. Probably attempting to aristocratize/oligarchize the site might work better. Maybe increase the visibility of old-time users and posters, tweak the karma needed to post/vote, highlight high karma account comments over low karma comments. 

There must be some blog post somewhere documenting every attempted antidote to Eternal September syndrome to pick and choose from. Disclaimer, norm and law changes have chaotically unpredictable effects on communities, so who knows what the outcome would be.

A democratic version of this is people being more meta in comments and replies, addressing structural concerns with what people are commenting, as you mention in the post rewriting the original comment in a more rigorous/ironman form. Upside, this is a way to acculturate people into the community in an emotionally positive manner, rather than just by punishment. It's also much more legible and learnable than a comment deletion, which might have all sorts of reasons. Downside, this can make actual discussion really difficult and encourages pedantry which can also be taken too far. It also requires some degree of critical mass of users willing to engage in it.

The utopian version of this to me would be people looking at a post or comment they disagree with, suspending their own opinion on it, and attempting to help the commenter improve their argument in the direction the OP was going.

It's a Trojan horse.  It's just such good-thoughts-wrapped-in-bad-forms that people give a pass to, which has the net effect of normalizing bad forms.

Should this be "bad thoughts wrapped in good forms ... normalizing bad thoughts"?

I commend your vision of LessWrong.

I expect that if something like it is someday achieved, it'll mostly be done the hard way through moderation, example-setting and simply trying as hard as possible to do the right thing until most people do the right thing most of the time.

But I also expect that the design of LessWrong on a software level will go a long way towards enabling, enforcing and encouraging the kinds of cultural norms you describe. There are plenty of examples of a website's culture being heavily influenced by its design choices - Twitter's 280-character limit and resulting punishment of nuance comes to mind. It seems probable that LessWrong's design could be improved in ways that improve its culture.

So here are some of my own Terrible Ideas to improve LessWrong that I wouldn't implement as they are but might be worth tweaking or prototyping in some form.
(Having scanned the comments section, it seems that most of the changes I thought of have already been suggested, but I've decided to outline them alongside my reasoning anyway.)

• Inline commenting. If a comment responds to a specific part of a post, it may be worth having it presented alongside that part of the post, so that vital context isn't missed by the kinds of people who don't read comments, or if a comment is buried deep under many others.  These could be automatic, chosen by the author, vetoed by the author, chosen by voting, etc. Possibly allow different types of responses, such as verification, relevant evidence/counterevidence, missed context, counterclaims, etc.
• Multiple voting axes. Having a 1D positive-negative scale obviously loses some information - some upvote to say "this post/comment is well-written", "I agree with this post/comment", "this post/comment contains valuable information", "this post/comment should be ranked higher relative to others", and pretty much any other form of positive feedback. Downvotes might be given for different reasons again - few would upvote a comment merely for being civil, but downvoting comments for being uncivil is about as common as uncivil comments.
  Aggregating these into a total score isn't terrible, but it does lead to behaviour like "upvoting then commenting to point out specific problems with the comment so as to avoid a social motte-and-bailey" like you describe. Commenting will always be necessary to point out specific flaws, but more general feedback like "this comment makes a valuable point but is poorly written and somewhat misleading" could be expressed more easily if 'value', 'writing quality' and 'clarity' were voted on separately.
• Add a new metadata field to posts and comments for expressing epistemic status. Ideally, require it to be filled. Have a dropdown menu with a few preset options (the ones in the CFAR manual are probably a good start), but let people fill in what they like.
• Allow people to designate particular sentences in posts/comments as being a "claim", "hypothesis", "conjecture", "conclusion" (possibly linking to supporting claims), "crux", "meta", "assumption", etc., integrating epistemic status into a post's formatting. In my mind's eye, this looks something like Medium's 'highlight' feature, where a part of a post is shown in yellow if enough readers highlight it, except that different kinds of statements have different formatting/signposting. Pressing the "mark as assumption" button would be easier to do and to remember than typing "this is an assumption, not a statement of fact", and I also expect it'd be easier to read.
  These could have a probability or probability distribution attached, if appropriate. 

Most of these would make interacting with the site require extra effort, but (if done right) that's a feature, not a bug. Sticking to solid cultural norms takes effort, while writing destructive posts and comments is easy if due process isn't enforced.

Still, making these kinds of changes right is very difficult and would require extensive testing to ensure that the costs and incentives encourage cultural norms that are worth encouraging. 

But it’s not actually doing the thing, and as far as I can tell it’s not really trying to do the thing, either—not in the way that blue-tribe Americans are actually trying to do something about racism, from pushing for institutional change all the way down to individuals taking personal responsibility for aiding strangers-in-need.

Is this really what you want? The way that blue-tribe Americans are “actually trying to do something about racism” seems to have almost no chance of actually doing anything about racism, nor even to be designed or intended to have any chance of actually doing anything about racism. Mapping this dynamic to your project seems to invalidate the entirety of your post.

Yet the fact that (in the “racism” case) the stated goal, the intended goal, and the actual goal do not coincide, appears not to be an incidental or contingent fact, but rather seems to be integral to the effectiveness and persistence of the actual efforts…

Please note that this is not a nitpick. It seems to me that this point substantively undermines your thesis, and your hopes for solving the problem you describe. (Which is unfortunate, because I largely agree with your diagnosis of the problem.)

Meta comments on this post:

I appreciate your commitment to precision, specificity, and disclaimers, to rule out what you don't mean [LW · GW] and to pinpoint exactly what you do mean. I aspire to clear communication, too, and it's obvious to me that you've put orders of magnitude more effort and skillpoints into it than I could ever imagine doing in my own attempts to communicate. (The best I can personally do is write in a longwinded manner, since my attempts at concise language are too easy to misunderstand.)

That said, this has downsides, too. For instance, I'm not a native speaker and my reading speed is slow. By the time I reached the object-level points in this post (which was already split up! originally it was 3/4 meta or setup stuff!) I'd forgotten some of the meta points or didn't understand how they related to the object level.

Also, LW is high context, and sometimes we expect people to know things that they should know but don't, or did know but forgot. For example, only by the last sentence of this post did I vaguely understand what the stag hunt stuff was about. So I'd recommend adding one sentence at the start to explain what the title and Stag Hunt refer to.

(Another minor problem here is that without the concept handle "stag hunt", I could not necessarily find this post in the future, even if I remembered you wrote it and browsed just your essays.)

point at small things as if they are important

Taking unimportant things seriously is important. It's often unknown that something is important, or known that it isn't, and that doesn't matter for the way in which it's appropriate to work on details of what's going on with it. General principles of reasoning should work well for all examples, important or not. Ignoring details is a matter of curiosity, allocating attention, it shouldn't impact how the attention that happens to fall on a topic treats it.

general enthusiasm for even rather dull and tedious and unsexy work

This is the distinction between "outer enthusiasm", considering a topic important, and "inner enthusiasm", integrity in working on a topic for however long you decide to do so, even if you don't consider the topic important. Inner enthusiasm is always worthwhile, and equivocation with outer enthusiasm makes it harder to notice that. Or that there should be less outer enthusiasm.

Where bad commentary is not highly upvoted just because our monkey brains are cheering, and good commentary is not downvoted or ignored just because our monkey brains boo or are bored.

Suggestion: give our monkey brains a thing to do that lets them follow incentives while supporting (or at least not interfering with) the goal. Some ideas:

• split upvotes into "this comment has the Right effect on tribal incentives" and "after separating out its impact on what side the reader updates towards, this comment is still worth reading"
• split upvotes into flair (a la basecamp), letting people indicate whether the upvote is "go team!" or "this made me think" or "good point" or " good point but bad technique", etc

Strong upvote. Also, I first noticed these types of dynamics at a large scale in the comments on Duncan's Dragon Army proposal (I'm linking to the Medium version since the LW version seems to be gone).

The post has a lot of things going on. I am chipping on a it somewhat one spoonful at a time.

I didn't engage with the main drama directly as I am not around there.

It feels like a story about the a stag hunting party that encountered a bear. A bit meatier but a different beast and dangerous. Then some hunters go "nope, I didn't sign up for this" or just flee in abject horror. Those that stay get mauled because a small group can't deal with a full bear.

With a regular stag hunt it elicts frustration. Repeated failures might make starvation creep closer but usually dominate doing nothing. Getting mauled by a bear is actively harmful. And it is more harmful the less people are there to share the hurt. With a group you can carry wounded, or have diluted chance of getting hit by the next swing. Each additional desserter means the ones that stay get actively hurt more.

Maybe the hunting group didn't prepare to actually defend from the prey. Maybe they could have won if everybody suppressed their horror and just followed standard pack hunting guides. Somebody on serious mode gets super annoyed when they thought they went in with a group and end up dueling a bear. And there can be a sense of "hunting is exactly what hunting packs do", "we train up on smaller prey so that we can go after larger ones". Then there is the issue if instead of the village going out to look for the bear the bear visits the village.

I spend a lot of time around people who are not as smart as me, and I also spend a lot of time around people who are as smart as me (or smarter), but who are not as conscientious, and I also spend a lot of time around people who are as smart or smarter and as conscientious or conscientiouser, but who do not have my particular pseudo-autistic special interest and have therefore not spent the better part of the past two decades enthusiastically gathering observations and spinning up models of what happens...
...
All of which is to say that I spend a decent chunk of the time being the guy in the room who is most aware of the fuckery swirling around me, and therefore the guy who is most bothered by it... I spend a lot of time wincing, and I spend a lot of time not being able to fix The Thing That's Happening because the inferential gaps are so large that I'd have to lay down an hour's worth of context just to give the other people the capacity to notice that something is going sideways.

This thought came to me recently and I wanted to commend you for an excellent job at articulating it. Having the "wincing" experience too many times has damaged my optimistic expectations of others, the institutions they belong to, and society as a whole. It has also conjured feelings of intellectual loneliness. Having this experience and the thoughts that follow from it constitute what might be the greatest emotional challenge that I struggle with today.

Make wildly overconfident assertions that it doesn't even believe (that it will e.g. abandon immediately if forced to make a bet).

This seems like an important failure mode that I think a ton of people, including myself, fall victim to. I can't actually think of a reference post for it though. Does it exist? Really, I think it deserves it's own tag [? · GW]. I recall hearing it discussed in the rationality community, but moreso in passing, not as a focal point of something.

Hire a team of well-paid moderators for a three-month high-effort experiment of responding to every bad comment with a fixed version of what a good comment making the same point would have looked like.  Flood the site with training data.

What's so terrible about this idea? I imagine the main way it could go wrong is not being able to find enough people willing to do it / accidentally having too low a bar and being overwhelmed by moderators who don't know what they are doing and promote the wrong norms. But I feel like there are probably enough people on LW that if you put out a call for applications for a very lucrative position (maybe it would be a part-time position for three months, so people don't have to quit their jobs) and you had a handful of people you trusted (e.g. Lightcone?) runing the show, it would probably work.

Standards are not really popular. Most people don’t like them. Half the people here, I think, don’t even see the problem that I’m trying to point at. Or they see it, but they don’t see it as a problem.

I think a good chunk of LW’s current membership would leave or go quiet if we actually succeeded at ratcheting the standards up.

I don’t think that’s a bad thing. I’d like to be surrounded by people who are actually trying. And if LW isn’t going to be that place, and it knows that it isn’t, I’d like to know that, so I can go off and found it (or just give up).

I have expressed more or less this view several times in the past, and have, each time, been told that the policy of Less Wrong, and goals of its moderation team, are directly and explicitly opposed to it.

Do you have some reason to believe that this has changed?

(This is not a rhetorical question. It’s entirely possible that you’ve got information about shifting views of the Less Wrong team, which I am not privy to; but if so, then I think it is necessary to “lay the cards on the table”, as it were. Otherwise, it seems to me that this exercise is pointless, yes?)

Would the StackExchange model work? Granting ranks on the basis of productive contribution, along with privileges, ultimately recruiting moderators from the highest ranks.

Hire a team of well-paid moderators for a three-month high-effort experiment of responding to every bad comment with a fixed version of what a good comment making the same point would have looked like. Flood the site with training data.

Maybe we can start with a smaller experiment, like a group of (paid or volunteer) moderators do this for just one post? I sometimes wish that someone would point out all the flaws in my comments so I can tell what I can improve on, but I'm not sure if that won't be so unpleasant that I'd stop wanting to participate (or there would be some other negative consequence). Doing a small experiment seems like a good first step to finding out.

Assuming such experiments go well, however, I'm still worried about possible longer term unintended consequences to having a "high standards" culture. One that I think is fairly likely is that the standards will be selectively/unevenly enforced, against comments/posts that the "standards enforcers" disagree with, making it even more costly to make posts/comments that go against the consensus beliefs around here than it already is. I frequently see such selective enforcement/moderation in other "high standards" spaces, and am worried about the same thing happening here.