Should you write longer comments? (Statistical analysis of the relationship between comment length and ratings)

In the "top 10" aggregate, you are at risk of the following Simpsonian problem: you have two posters A and B; one writes longer comments than the other and also happens to be cleverer / more interesting / funnier / better at appealing to the prejudices of the LW crowd. So in the whole group there is a positive correlation between length and quality, but actually everyone likes A's shorter comments better and everyone likes B's shorter comments better. (Or, of course, likewise but with "longer" and "shorter" switched.)

One story is that people are more willing to read long comments from Yvain or Eliezer because they have reputations for being insightful.

I think these plots would by much improved by adding error bars. In particular, I suspect that the number of short posts is greater than the number of long posts and so the average-karma estimates for long posts are more uncertain.

Also, did you bucketize the word counts? What do specific points on your plots correspond to?

You cannot use observed dependences in the data to suggest decision changes because p(y | x) is not in general equal to p(y | do(x)).

Upvotes are in my opinion a poor metric to measure the quality of a post. You're confusing information on how insightful, thoughtful or useful your writing is with information on how pleasing it is due to the upvoter due to providing social confirmation of their beliefs or entertaining them for other reasons.

A much more useful way to measure the quality of your own writing is to look at how interesting or thoughtful the replies you get are: this shows that people find your ideas worth engaging with. This is a subjective assessment however that can't be captured by the real line.

Naturally, even if your longer comments are rated higher than the short ones, this does not mean that inflating comments would always produce positive results.

Here's where I got really hopeful that you would address the part about correlation and causation, Goodhart's law, etc.

Would statistical feedback on the style and content of your posts be useful to you?

[pollid:1010]

Since I critiqued the graphs in the OP and offered to make them better, cleonid was gracious enough to provide me with the dataset for Eliezer (EY) and Yvain (Scott Alexander) (SA) posts and invite me to play with it. For convenience, I'll split my comment into two parts -- the preliminaries and the analysis itself.

First, two caveats.

I'm looking at data for posts by EY and SA. They, being superstars, are not representative of LW members. This analysis will not tell you, gentlereader, whether you would get more karma by making your posts longer or shorter (unless, of course, you are EY or SA :-D).

Next, the data that I have lacks timestamps. Therefore I'm forced to make the assumption (almost certainly false) that nothing changed with time and treat the data as a blended uniform mass. The temporal dimension is, sadly, lacking.

A brief description of the data: we have two tables (EY and SA) which list the number of characters in a post, as well as the number of upvotes and downvotes that the post received. There are about 3100 data points for EY and about 1400 for SA.

The distribution of post lengths for EY and for SA is rather different: EY posts tend to be much shorter. This is visible in the following graph which plots the empirical distributon of post lengths (cut off at 2000 chars, but there is a long thin tail beyond it).

In numbers, the median length of an EY post is only about 200 characters -- half of his posts are less than that. In fact, a quarter of his posts are less than 87 characters. On the other side, 10% of his posts are longer than 950 characters, 5% are longer than about 1500 characters. The mode of the distribution -- the most frequent length of the post -- is around 70 characters.

SA writes more: his median post length is 450 characters, more than double that of EY's. The shortest quarter of the posts is below 165 characters, the longest 10% are over 1900 characters and the longest 5% -- over 2700 characters. The mode for SA's posts is 180 characters, two and a half times as much as EY's mode.

Continued in post 2.

My prior expectation would be: A long comment from a specific user has more potential to be interesting than a short one because it has more content. But, A concise commenter has more potential to write interesting comments of a given length than a verbose commenter.

So while long comments might on average be rated higher, shorter versions of the same comment may well rate higher than longer versions of the same comment would have. It seems like this result does nothing to contradict that view but in the process seems to suggest people should write longer comments. The problem is that verbosity is per-person while information content is per-comment. Also verbosity in general can't be separated from other personal traits that lead to better comments.

You could test this by having people write both long and short versions of comments that appear to different pools of readers and comparing the ratings.

Continued from part 1.

The gist of part 2 is four graphs.

The graphs plot most of the data (except for outliers) in the following form. Each post is represented by two points with the same X coordinate: the number of characters. The Y coordinate for one point is the number of upvotes the post received, the Y coordinate of the other point is the number of downvotes for the same post. Upvotes are light green and downvotes are pink.

The upvotes and the downvotes are modeled separately by two loess (local regression) curves. The difference between two graphs for each of the posters is in the details of the fit. Specifically, one fit assumes gaussian errors and so the loess curve tends to approximate the local mean. The other fit assumes heavy-tailed errors and its loess curve tends to approximate the local median. Since the distribution of votes is skewed, the mean and the median are noticeably different.

Each plot has four vertical lines at four quantiles: 25%, 50%, 75%, and 95%. The lower numbers represent the loess estimate of the number of downvotes for this particular post length. The upper numbers represent the loess estimate of the number of upvotes.

We will start with the robust fit which approximates the median. Here is the plot for EY

and here is the plot for SA

As you can see, longer posts pay off though not in a particularly spectacular manner for EY -- long posts work better for SA. The downvotes also increase, but insignificantly. If we treat the loess estimate as the median, in all cases half of the posts has zero downvotes.

Since the votes are positively skewed, the means should be higher than the medians and we can see it in the second set of graphs with non-robust loess fits. EY

and SA

The overall pattern is very much the same, but the numbers are higher. Again, longer posts bring much more karma for SA, not so much but still some for EY.

I would love to see mine, considering I have two very different styles of post which have different average lengths.

I would be interested to see the results of some Clustering Algorithm on the comment data. It may be, that long comments can be classified into high karma and low karma and we can then analyze what the differences between them are. If it is possible to extract features of high-quality posts, then those features can be the goal, instead of just the length.

I also think it's dangerous to focus too strongly on karma, because karma score is only a rough approximation of actual quality. For example, I believe many short comments, that only ask for some clarification are generally more important than is reflected by their karma.

Some comments are very insightful and are not particularly long. I recently have been reading Roissy's site, and I searched a few terms at LW. I discovered the user Vladimir_M who had some very interesting comments, not only on that front. HughRistik was also someone who stood out. Both of their commenting were very insightful. I am disappointed those people were not on the list.

My other problem with this is that if karma is a good indicator. That pretty much depends on the userbase; I guess we have another thing to test here.

I think there may be a collective action problem here. Optimizing length to maximize upvotes seems like a way to encourage others' biases. Then there are prisoner's dilemma possibilities. If everybody increases their comment length, won't we have to increase our comment length even more to stand out?

I wonder if the value of a post is not correlated with upvotes. i.e. a post that is 1.3* more valuable than another might have to be 100 words longer but only get 10% more upvotes.

I feel like even if we could encourage only the posts who are below 500 characters that seem to have downvotes to consider increasing their length in order to roughly correlate with sharing more words = being clearer = providing more value. Even at the worst effects of the results of such a strategy, we would probably see the garden get a little nicer.

And continuing the metaphor, its not like we are chopping down trees, just clearing out a few weed around the roses.

I propose a character count on the comment boxes so that people know how many characters they are writing, then possibly a popup (similar to the one about comments on downvoted posts), that says,

"we noticed that comments are more meaningful, helpful and thought out when they are at a minimum 500 characters, you can still post a shorter comment but you have N characters to go to cross the arbitrary threshold we decided on. You can still post but it will cost -1 karma. If you think that you will get at least one upvote more than usual then its certainly worthwhile posting as is; otherwise can you add more useful characters to your post?"

[pollid:1011]