In my experience, writing full-fledged, thoroughly researched material is pretty time-consuming, and if you push that out to the audience immediately, (1) you've sunk a lot of time and effort that the audience may not appreciate or care about, and (2) you might have too large an inferential gap with the audience for them to meaningfully engage.

The alternative I've been toying with is something like this: when I'm roughly halfway through an investigation, I publish a short post that describes my tentative conclusions, without fully rigorous backing, but with (a) clearly stated conclusions, and (b) enough citations and other signals that there's decent research backing my process. Then I ask people what they think of the thesis, which parts they are interested in, and what they are skeptical of. Then after I finish the rest of the investigation I push a polished writeup only for those parts (for the rest, it's just informal notes + general pointers).

For examples, see https://www.lesserwrong.com/posts/ghBZDavgywxXeqWSe/wikipedia-pageviews-still-in-decline and http://effective-altruism.com/ea/1f9/the_aidsmalaria_puzzle_bleg/ (both are just the first respective steps for their projects).

I feel like this both makes comments more valuable to me and gives more incentive to commenters to share their thoughts, but the jury is still out.

This feels like a classic Goodhart's Law problem. More comments are good for multiple reasons (feedback, positive reinforcement, more ideas, keeping up the conversation, and so on). It would be good to generate more comments, and do things that do this (as long as we avoid obvious provocations like #1 and #2 above), but if we start maximizing comments with our posts, then if you're not very careful about it, that path goes south quickly.

One thing I think is both 'safe' and effective over time is to reliably engage with your commentators. If you establish a norm that you're going to clearly read most or all the comments on your own posts, and reply to them when there's something to say (comments on comments are reinforcement the same way comments are) that sends a clear signal that comments are welcome and valued. I think that worked well on my own blog, but it's hard to disentangle with many things including 'I kept writing and got less awful at it in other ways' or with 'I kept writing and people finally started reading.'

I'd also note that from my Magic writing I developed the heuristic of a 1000:1 page view to comment ratio as the default outcome, so compared to that we're doing astoundingly well. My few examples of my blog being linked to by Scott and Marginal Revolution tell me that the 1000:1 remains about right in those contexts.

What about asking your audience questions?

For example, you could ask questions:
* Seeking criticism, such as "I think section x is the weakest part, what are some alternative arguments?"
* Promoting understanding, such as "Can you think of 2 more examples of <concept I just introduced>?"
* Stimulating research, such as "I think this model can be applied to area y, does anyone have suggestions for how to do this?"

This might help get readers out of passive consumption mode, and into thinking about something they could comment about. It would also make the writing more useful.

The situation you find yourself in is called Warnock's Dilemma, named after Bryan Warnock, one of the early Usenet pioneers. He said that silence can be interpreted in one of five different ways:

1. The post is obviously correct, and so well researched that nothing more needs to be said.

2. The post is complete and utter nonsense but no one wants to waste the time or energy to point it out.

3. No one read the post.

4. No one understood the post and thought it would be worth their time to ask for further clarification.

5. No one cares about the post.

After reading Ryan Holiday's book, Trust Me I'm Lying, I'm not sure that it is possible to incentivize comments without falling into the Buzzfeed trap of writing content with high emotional valence in order to drive engagement. I think that the way to improve one's writing is to seek out people and ask them to give comments (and incentivize this, either through social means or even via financial means, if you can afford it). Relying on one's writing alone to draw commenters sets up an incentive gradient that leads straight to Buzzfeed, and I'm not sure that there's really any way out of it.

After reading this, I tried to figure out what the desired end goals were. I think we can all agree that commenting for the sake of commenting is meaningless, so I looked for what would fill in the blank "commenting for the sake of ______".

At the beginning of the post I saw:
1) I like getting comments (I see this as a bid for attention)
2) Staying in the public view (more comments = more staying power)
3) Learning on the side of the reader (commenting = engagement = better learning)

At the end of the post I saw:
4) Public attention incentivizes efforts to research and learn (I see this as attention as well)
5) People writing for comments write more engagingly
6) Learn how to comment better (what's the goal of a comment? To learn better?)
7) Engaging writing decreases the chance that good thoughts don't get lost (public view again)

With that, I'm seeing two main ideas: bids for attention and learning on behalf of the public. Bids for attention aren't bad and I think a community has to respond to them for the community to work well. If a community never tells its members that they are seen and noticed, the community isn't going to last very long. How does it change the writing and responding if we acknowledge these bids?

On public learning, I think it might help to look into the research on teaching. Additionally, the assertion of teacher and student will have to be addressed. The approach of "I have things you need to learn" is a bit off putting and likely to turn people away. However, this community seems to want to learn, so maybe a call for teachers on topics the community is interested in, and that can establish who is teacher and who is student.

On specifically the teaching/learning part, research in teaching shows that lecturing is very ineffective for learning. Many times, long written posts are simply data dumps that are similar to lectures. Effective teaching puts effort on the learner, tolerates and corrects mistakes, and encourages trying. The teacher may have the knowledge but doesn't give the knowledge and instead helps the learner find the knowledge. I have no idea how well this would work in a web forum.

I think there are more ways that writing and commenting can be used, but these were the themes I picked up from this post. Perhaps by bringing them forward like this, the other uses of writing can be drawn out.

Be Wrong - It has been said: if google fails you, the fastest way to get a question answered is to post a wrong answer on reddit. This will result in a lot of flood of people explaining things to you.

It strikes me, in light of Scott Alexander's recent post on predictive processing, that this might not be quite as "evil" as it seems. Maybe seeing a wrong answer triggers a prediction error which brings more relevant information into conscious awareness, actually making the original question easier to think about.

What if there were different sections for different types of comments? Ex. a section for:

1. "I agree"-type comments

2. "Stupid question"-type comments

3. Very low barrier "open thread/random" comments

4. Typical high barrier "meaty" comments

Right now, I suspect that people with "I agree", "Stupid question" or "open thread/random" comments don't post them because they're "not good enough". But if there was a section explicitly asking for these types of comments, I suspect that people would be very willing to provide them.

Another benefit is that comment readers could skim for the types of comments they're interested in reading.

I'm not sure if the sections I proposed are the best ones to have, just thought a concrete example would make things easier to describe.

I don't think the UI for comment sections would make things overly complicated. Something like Bootstrap's nav tabs should do.

Writing this comment to provide social validation!

Like you, I have this norm in my head that I should only comment if I have something concrete to add, rather than to just encourage or discourage. But I wonder if this comes from a different context? I don't remember how I learned this norm, but many places on the Internet get enough posts and comments at a low enough average quality that I don't really value having another marginal post or comment unless it's raising the average quality. By contrast, the marginal post or comment on LW is interesting to me even if it's not raising the average quality, because the average quality is pretty high and I literally run out of posts and comments to read. So maybe the norm is not serving its original purpose.

As a blogger, I found (a few years ago) I reliably got more comments on what I thought were my *worst* posts. Now that's more or less no longer true; I can write stuff that's reasonably "meaty" for a blog post and still spark debate, though I'll never get a ton of attention as long as I stay out of the Petty Bullshit Gravity Well. Full-fledged theories are bad for getting comments (and also more likely to be wrong in retrospect). Semi-formed ideas fleshed out with facts and references and imagery are better.

You mention wanting to be incentivized to research things, and also that a particular danger to the community is writers optimizing for engagement at the expense of other things.

It seems like a possible partial remedy for this would be a mechanism for the readership to make their desires known in a centralized place. Right now, a hypothetical writer William, if they want to craft content the community wants, would be best served by doing a review of past posts in search of things which are consistently popular. If they are lucky and clever, they may even be able to infer a niche which hasn't been filled - a gap in some topic, or perhaps a missing topic altogether - that would be welcome. This is time-consuming and presumably error-prone. Perhaps more importantly, it seems likely to produce content which is similar to what has come before. (Disclaimer: I have never made a top-level post, so if there is a different, quicker, more accurate process, please let me know.)

If instead the community could vote to say "We would like more posts on Machine Learning" or "We want posts on what Dr. X's latest psychology research means about consciousness", this would create an easily visible incentive to research and an incentive to explore specific topics (which may steer writers away from optimizing for comments, although the comments will likely follow). It adds complexity to the site, but I think it may be worth an experiment.

Venue also matters a lot through the social context it brings. Individual Wordpress blogs often feel like you're saying "this is where my writing lives; by commenting, you're coming into my house", which can be challenging to take lightly -- especially when you're talking about a neighborhood of individual blogs, few of which get regular comments. Meanwhile social media is a weird mix of jokes and personal content with discussion-oriented ideas, where there's an uncertain rudeness in potentially burying someone with attention or notifications by Starting Discourse. And in both of these, if it's not controversy or gossip or dilettantism, then posting the most makes you king.

So I was/am hopeful about posting more to LW 2.0 largely for the sake of better defaults around "this is for having a conversation" -- both "formally" in responding directly to or building on the OP, and more "socially" or indirectly by contributing thoughts on the same subject, and in a venue with moderation and karma where things can bubble up without the speculative/social/everyone's-an-expert elements (or sheer consistent quantity).

I find that my writing seems to actively repel comments compared to stuff that gets comparably received by other metrics. I do try to go out of my way to write mostly on the rare occasions I have something unambiguously sensible or useful to contribute; it earns me a high upvote/downvote ratio, but little sense of how people are engaging with what I have to say.

At the same time, maybe this makes me part of the problem of silence on the best writing. I'm also interested in learning to be a better commenter, but I'm not someone who thinks they can or should always have something to say. For my part, I think this mostly indicates that I should comment more with thoughtful questions, but I'm very interested in you or anyone else fleshing out your "being a better commenter" open problem -- I think this is potentially more important for success here than writing the right kinds of posts.

I wonder if it would be useful to have some kind of informal peer review for boring important posts.

My solution is that I'm motivated by the process of writing, and so if anything am less likely to write high-comment-count articles, because it's stressful when lots of people disagree with me. (Although I end up writing a lot of social reality posts anyway because that is what I have opinions about.)

In fandom, people tend to leave comments that talk about particular emotions they had or lines they thought were particularly funny or insightful, so you get a lot of comments on widely read things even if there is nothing wrong with them. I find myself reluctant to leave such comments on lesswrong 2.0 because I don't want to clutter everyone's recent-comments feed.

Promoted to featured for writing up thoughts from over a long period, about an important problem for the community, and provoking great, helpful comments.

1. Be Wrong - It has been said: if google fails you, the fastest way to get a question answered is to post a wrong answer on reddit. This will result in a lot of flood of people explaining things to you.

Reminds me of the old joke: if you want to find out how to do X in Linux, asking about it is useless. But going to the #linux IRC channel to say "Linux sucks because it doesn't even let me do X", will give you the answer of how to do X in no time.

I am working on plans for a sequence that I think has a shot at being 6, though I worry about my writing quality. (edit clarification: I will probably be ready sometime early next year. I've been pondering how to do this for a while, and I'm starting practicing writing skill I'll need.)

This post has a good quality of meta. The advice mostly applies internally.

It would be nice if there was a magic number in the font attributes, supplementing size/color/etc, which modulated people's internal quality bar, ranging from "facebook" to "academic journal". A website could dynamically set this based on the difference between the ideal amount of engagement and the amount observed.

This seems to be a problem which crops up again and again in different forms: a discussion/location/thing gets good, and implicitly, the quality bar rises. Maybe it is because an audience showed up, or maybe it's just because you start worrying about ruining the quality of what has gone on so far. This takes away the original sense of freedom which often produced the quality in the first place. You start a tumblr to put writing that's not good enough for the blog. Your Twitter account gets popular and you retreat to an alt account, only to find that it gets popular, making you retreat to another alt... That's a bit different from what you're pointing at, but it seems related.

Realistically, how does one hang a little sign that says "community space, feel welcome to engage"? I suppose a sense that people know each other is important to this. It's part of the reason Facebook feels easy; you are talking to your friends.

I suppose the size of the audience is very important here. If there is a large enough audience then good posts will still receive a reasonable number of comments even if they get less than posts which have more of the factors that you described. On the other hand, when the size of the audience falls below a critical mass this kicks off a Negative Feedback Loop where less commenters means less posters and hence less commenters and so on and so forth. I'll be curious to see whether LW2.0 had gotten us above this critical mass.

Maybe part of the problem is that some topics lend themselves to short-form discussion, while other topics lend themselves to long-form discussion? There's lots of great posts I read where I don't feel the urge to comment, but do feel the urge to research a little and then write a long-form response - heck, that's how I feel about probably a third the stuff Scott writes. But the current standard interface encourages a long-form post followed by short-form comments.

Maybe one solution would be encourage "comments" which are links to a long-form response, along with a short summary? Maybe make such responses more discoverable in comment feeds?