The 2021 Review Phase

post by Raemon · 2023-01-05T07:12:24.251Z · LW · GW · 7 comments

Contents

  Key Info:
  How much Progress Overall?
  What kinds of reviews are most valuable?
  Go Forth, Review, and Get Prizes
None
7 comments

(Apologies for this post being quite late – I got sick just prior to the holidays, and then had a surprisingly eventful holidays)

We've wrapped up the nominations phase for the 2021 Review. 139 people have participated so far, casting 1815 votes and writing 72 reviews. 

Now we're in the Review Phase, where we ask "Were these posts actually good? How could they be improved? How do they fit together?". 

Key Info:

How much Progress Overall?

One of my original goals for the Review was to provide a clearer feedback mechanism to the Lightcone team on "how is LessWrong doing?". It's one thing to track karma and site activity. But did our infrastructure maintenance, UI improvements, research retreats and other interventions actually seem to translate into more good, quality intellectual progress?

This year some people have commented [LW · GW] that there are fewer exciting-feeling posts than they remember from previous years. I've also had a bit of this feeling – despite feeling at the time like in 2021, LessWrong was a pretty happening place with lots of good content. I'm not sure I endorse the feeling. For example: I think important progress happened in the MIRI Dialogues, but in a diffuse way, which didn't result in many concrete posts I can point at to that I learned discrete concepts from.  Progress != posts, necessarily. 

But, it still seems worth paying attention to the sensation. Yes requires the possibility of no, [LW · GW] and it's fairly important for the Review to be able to return the answer "less overall progress this year." 

Now, right now there isn't much of a mechanism for tracking "how much progress" there was between years. I'm still thinking about how to seriously answer this question. But for now, I've update the /reviewVoting page to be able to show previous years, so you can go eyeball the results from the previous 2020 Review [? · GW], see how many posts you voted as a "9", etc. 

What kinds of reviews are most valuable?

Briefly noting what you liked or didn't like about a post is good. But here are some specific things I'm excited for:

New Information. My favorite reviews are ones that give new information to the reader. This could be a concrete fact, or a rebuttal to a key argument in a post, or a new way the post is relevant that you hadn't considered. It could be a new frame about how to even think about the post.

Concrete use cases. I think reviews that say "this concretely helped me, here's how" are also great, especially if they give specifics. I think it's both helpful and rewarding to individual authors to hear how their post was useful, so they can do more of that. (I think this is also helpful on a broader level, so that the collective LW userbase can see what kinds of effects are realistic)

i.e. rather than saying "this post has helped me a bunch over the years", say "here are two specific times that it helped me, and how."

Thoughts on how the post could be improved. Two years later, if a post still seems like good reference material, but is confusing or poorly argued, give advice on how to improve it. Discuss the use-case for the readership you have in mind.

Reflect on the Big Picture. How do various posts fit together into something greater than the sum of their parts? What major conversations have happened on LessWrong and what have you taken from them?

Brevity/clarity. Bonus points if they are concise and relevant enough that it's fairly easy to print them in the 2021 book. (I wanted to print some reviews in the 2020 books but found it'd require a fair amount of editing work to get them into a shape that made sense). Note, I'm not saying "optimize for shortness at the expense of saying anything substantive", just, note that all-else-being-equal, taking less space to convey the key information is helpful.

Go Forth, Review, and Get Prizes

Remember, I'll be awarding prizes to reviews that I think add substantial new information, critique, or useful frames. Prizes will be announced in the comments here.

7 comments

Comments sorted by top scores.

comment by Raemon · 2023-01-13T22:43:15.340Z · LW(p) · GW(p)

I've made a few updates to the Review UI over the past week. (I apologize to people who keep having the UI shift a little bit underneath them, although I think it's overall an improvement).

Quick Review Page

I originally designed the Review Dashboard as basically a powertool for powerusers. The Review is kinda intrinsically a complex operation, and for my own use case, I think I needed a view that showed a lot of information. I went and tried to simplify it recently, and found that, nope, all the complexity was actually serving an important purpose. 

But, it sure was overwhelming. And having thought through the ideal poweruser use-case, I finally had an easier time figuring out a more simplified UI that tries to just get you to do one thing, reasonably easily.

So, behold the Quick Review Page [? · GW]. This basically tells you "please try to review 3 posts if you can", and shows you the top-10 unreviewed posts (based on the votes from 1000+ karma users in the Nomination Voting phase). 

It doesn't cover everything you might possibly want. There's a lot of posts I think would benefit from multiple reviews (Quick Review only shows un-reviewed posts). So I do recommend still checking out the Advanced Review Dashboard [? · GW], which provides more customized sorting options. But, if you look at the advanced dashboard and go "aaaaah", or "uuugggh why does it take so long to load?", the Quick Review page loads much faster and probably gets you most of the value.

3 Review "checkboxes" 

I think it's a reasonable "be a good citizen of LessWrong" ask to do at least 3 quick reviews, to help provide some longterm signal. I'm most interested in this from longterm users so on the frontpage it only shows from 1000+ karma users, although they appear for all users on the dashboard pages.

This is just a UI nudge, obviously a lot of people are busy, or aren't that invested in LessWrong, and that's fine.

Reviews page

You can now easily see reviews from previous years on the /reviews [? · GW] page. You can filter by year, or see all reviews from all time.

Longform Reviews

Some people have historically written up longform, top-level-post reviews, that either go into a lot more detail evaluating one post, or review a lot of posts at once and maybe look at some high level themes that emerge. I wanted to more explicitly encourage people to do that. So there is now a button on the dashboard pages 

You can find it at the bottom-right of the Quick Review [? · GW] or Advanced Dashboard [? · GW]. Functionally it's only slightly different from the normal New Post edit form. In future years I might incorporate Top Level Review posts into the rest of the Review UI.

comment by Raemon · 2023-01-16T08:26:39.786Z · LW(p) · GW(p)

We are now on to the final voting phase for the review. I may write up a post about that later, although the main takeaway is, well "cast your final ballots." The UI for the voting phase integrates the reviews into the voting ballots, so you can read a bit about what others found valuable or wrong/confused about each post to help inform your votes.

Meanwhile, I do want to note: you can still review as-yet-unreviewed posts, and get them onto the ballot.  The Voting Dashboard won't show posts that haven't been reviewed, so you'll have to remember the individual posts and navigate to their post page to find the "review" button. (This wasn't originally intentional behavior, I realized it was possible to continue reviewing things into the Voting Phase a couple years ago, but, I decided it's basically fine for the Voting Phase to be a "soft deadline", and left it as a "feature" rather than a "bug". The longer you wait to review things the fewer people will see it and it's less likely to get upvoted, so, I do encourage you to get any last minute reviews in soon)

Also meanwhile, I still think it's valuable to write in-depth reviews, especially for posts you expect to do well in the final vote. In my ideal world, all the top 10-20 posts would have gotten subjected to a significant critical analysis.

Meanwhile meanwhile meanwhile, a couple days ago I coded up a review leaderboard [? · GW]. This was kinda last minute and mostly I'm excited to have be more integrated into next year's 2022 Review, but, feel free to get Extrinsically Motivated to do some additional reviews. :P [also remember sock puppet voting will get your account banned :p :/]. Congrats to Elizabeth, Daniel Kokotajlo, la3orn, Drake Morrison and A Ray for being the non-LessWrong-team-members who topped the chart so far.

comment by Raemon · 2023-01-09T16:54:58.523Z · LW(p) · GW(p)

After talking things over with the LessWrong team, I decided to re-run the preliminary-vote totals. So, during the final week of the Review Phase, people deciding what to prioritize have a more accurate picture of which posts are likely to actually win, and therefore are worth more effort writing detailed reviews of. 

It's important not to be continuously re-running the vote-totals because it shifts voting behavior to be too strategic, but I think it's fine to re-run them once in the middle of the review phase each year.

comment by Raemon · 2023-01-05T20:51:50.734Z · LW(p) · GW(p)

This thread will be for aggregating reviews that I awarded prizes to.

Replies from: Raemon, Raemon, Raemon
↑ comment by Raemon · 2023-01-20T20:54:46.189Z · LW(p) · GW(p)

This comment is incomplete, and I will likely edit some of the prizes slightly. But, I'd fallen behind on awarding prizes for reviews and I want to highlight Yes, the Lightcone team will give you money for reviewing stuff. So, I wanted to ship this rough version for now to give a sense of what sort of reviews I found valuable.

Thanks to the large number of people who've stepped up to review so far. (I'd still be excited for EffortReview that explore the details of some of the higher ranked posts)

Next round of prize announcements:

Honorable Mentions:

These reviews didn't go into much detail, but added at least a couple new arguments or frames that I found useful:

↑ comment by Raemon · 2023-01-23T19:40:02.979Z · LW(p) · GW(p)

Next set of review prizes:

↑ comment by Raemon · 2023-01-05T21:07:24.421Z · LW(p) · GW(p)

First set of review prizes:

If he were eligible for a prize, I'd be awarding $50 to Ben Pace's review of Yudkowsky and Christiano discussing takeoff speeds. (Like iceman's review of Unnatural Categories, I didn't quite feel like I learned from it per se, but still was glad to get a detailed breakdown of what Ben found valuable).

If you want to receive money, please update your payment contact info and PayPal email at https://www.lesswrong.com/payments/account [? · GW]