Please Critique Things for the Review!

And yeah, the whole thing feels mostly like work, which can’t help.

This is partly why I haven't done any reviews, despite feeling a vague moral obligation to do so. Another reason is that I wasn't super engaged with LW throughout most of 2018 and few of the nominated posts jumped out at me (as something that I have a strong opinion about) from a skim of the titles, and the ones that did jump out at me I think I already commented on back when they were first posted and don't feel motivated to review them now. Maybe that's because I don't like to pass judgment (I don't think I've written a review for anything before) and when I first commented it was in the spirit of "here are some tentative thoughts I'm bringing up for discussion".

Also, I haven't voted yet because I don't remember the details of the vast majority of the posts, and don't feel comfortable just voting based on my current general feeling about each post (which is probably most strongly influenced by how much I currently agree with the main points it tried to make), and I also don't feel like taking the time to re-read all of the posts. (I think for this reason perhaps whoever's selecting the final posts to go into the book should consider post karma as much or even more than the votes?)

I think if there was a period where every few days a mod would post a few nominated posts and ask people to re-read and re-discuss them, that might have helped to engage people like me more. (Although honestly there's so much new content on LW competing for attention now that I might not have participated much even in that process.)

Nod. Something perhaps worth saying explicitly was that I was expecting / hoping for each longtime user to review a smallish number of things (like, 1-5) over the course of the monthlong review process, focusing on posts that they had some kind of strong opinion about.

(Some people have done lots of smaller reviews, which I also think is good but for different reasons, and not something I think people should be feeling pressure to do if they’re not finding it worthwhile.)

Raw numbers to go with Bendini's comment:

As of the time of writing this comment, there've been 82 reviews on the 75 qualified (i.e., twice-nominated) posts by 32 different reviewers. 24 reviews were by 18 different authors on their own posts. 

Whether this counts as a shortage, is puzzling, or is concerning is a harder question to answer. 

My quick thoughts:

• Personally, I was significantly surprised by the level of contribution to the 2018 Review. It's really hard to get people to do things (especially thing that are New and Work) and I wouldn't have been puzzled at all if the actual numbers had been 20% of what they actually are. Even the more optimistic LW team members had planned for a world where the team hunkered down and wrote all the reviews ourselves.
• If we consider the relevant population of of potential reviewers to be the same as those eligible to vote, i.e., users with 1000+ karma, then there are ~130 [1] such users who view at least one post on the site each week (~150 at the monthly timescale). That gives us 20-25% of active eligible voters writing reviews.
  • If you look at all users above 100 karma, the number is 8-10% of candidate reviewer engaging in the Review. People below 100 karma won't have written many comments and/or probably haven't been around for that long so aren't likely candidates.

Relative to the people who could reasonably be expected to review, I think we're doing decently, if something like 10-20% of people who could do something are doing it. Of course, there's another question of why there aren't more people with 100+ or 1000+ karma around to begin with, but it's probably not to do with the incentives or mechanics of the review.

[1] For reference, there are 430 users in the LessWrong database with more than 1000 karma.
 

I also buy the econ story here (and, per Ruby, I'm somewhat pleasantly surprised by the amount of reviewing activity given this).

General observation suggests that people won't find writing reviews that intrinsically motivating (compare to just writing posts, which all the authors are doing 'for free' with scant chance of reward, also compare to academia - I don't think many academics find peer review/refereeing one of the highlights of their job). With apologies for the classic classical econ joke, if reviewing was so valuable, how come people weren't doing it already? [It also looks like ~25%? of reviews, especially the most extensive, are done by the author on their own work].

If we assume there's little intrinsic motivation (I'm comfortably in the 'you'd have to pay me' camp), the money doesn't offer that much incentive. Given Rudy's numbers suppose each of the 82 reviews takes an average of 45 minutes or so (factoring in (re)reading time and similar). If the nomination money is ~roughly allocated by person time spent, the marginal expected return of me taking an hour to review is something like $40. Facially, this isn't too bad an hourly rate, but the real value is significantly lower:

• The 'person-time lottery' model should not be denominated by observed person-time so far, but one's expectation how much will be spent in total once reviewing finishes, which will be higher (especially conditioned on posts like this).
• It's very unlikely the reward is going to allocated proportionately to time spent (/some crude proxy thereof like word count). Thus the EV would be discounted by whatever degree of risk aversion one has (I expect the modal 'payout' for a review to be $0).
• Opaque allocation also incurs further EV-reducing uncertainty, but best guesses suggest there will be Pareto-principle/tournament dynamic game dynamics, so those with (e.g.) reasons to believe they're less likely to impress the mod team's evaluation of their 'pruning' have strong reasons to select themselves out.

Agree with these reasons this is hard. A few thoughts (this is all assuming you're the sort of person who basically thinks the Review makes sense as a concept and want to participate, obviously this may not apply to Mark)

Re: Prestige: I don't know if this helps, but to be clear, I expect to include good reviews in the Best of 2018 book itself. I'm personally hoping that each post comes with at least one review, and in the event that there are deeply substantive reviews those may be given top-billing equivalent. I'm not 100% sure what will happen with reviews in the online seqeunce.

(In fact, I expect reviews to be an potentially easier way to end up in the book than by writing posts, since the target area is more clearly specified.)

"It's Hard to Review Posts"

This is definitely true. Often what needs reviewing is less like "author made an unsubstantiated claim or logical error" and more like "is the entire worldview that generated the post, and the connections the post made to the rest of the world, reasonable? Does it contain subtle flaws? Are there better frames for carving up the world than the one in the post?"

This is a hard problem, and doing a good job is honestly harder than one month work of work. But, this seems like a quite important problem for LessWrong to be able to solve. I think a lot of this site's value comes from people crystallizing ideas that shift one's frame, in domains where evidence is hard to come by. "How to evaluate that?" feels like an essential question for us to figure out how to answer.

My best guess for now is for reviews to not try to fully answer "does this post check out?" (in cases where that depends on a lot of empirical questions that are hard to check, or where "is this the right ontology?" are hard to check). But, instead, to try to map out "what are the questions I would want answered, that would help me figure out if this post checked out?"

(Example of this includes Eli Tyre's "Has there been a memetic collapse [LW · GW]?" question, relating to Eliezer's claims in Local Validity [LW · GW])

FWIW from a karma perspective I've found writing reviews to be significantly more profitable than most comments. IDK how this translates into social prestige though.

Not all of us agree with the project. I disagree with the entire concept of "pruning" output in this way. I wouldn't participate on principle.

This insight density is what makes them interesting, but it also makes it very hard to evaluate the theories within them.

So posts should be (pre-)processed for theory/experimentation? (Or distilled?)

Other possible factors:

Maybe people read newer posts instead of (re-)reading older posts

the time of the year (in which reviews occurred)

the length of time open for reviews

the set of users reviews are open to

the set of posts open to review. For example, these are from long ago. (Perhaps if there was a 1 year retrospective, and 2 year, and so on up to 5 years, that could capture engagement earlier, and get ideas for short term and longer term effects.)

some trivial inconveniences around reading the posts to be reviewed (probably already addressed, but did that affect things a lot?)