The LessWrong 2022 Review

post by habryka (habryka4) · 2023-12-05T04:00:00.000Z · LW · GW · 43 comments

Contents

    Getting Started
    No books this year, sorry folks
  How does the review work?
    Phase 1: Preliminary Voting
      Writing a short review
      Why preliminary voting? Why two voting phases?
      How is preliminary voting calculated?
    Phase 2: Reviews
    Phase 3: Final Voting
None
43 comments

The snow is falling, the carols are starting, and we all know it's time for our favorite winter holiday tradition. It's LessWrong review time!

Each year we come together and review posts that are at least one year old. That means for the next two months we are reviewing all posts from 2022.

While our everyday lives are filled with fads and chasing the sweet taste of karma and social approval, the LessWrong review is the time to take a step back and ask ourselves "did this actually help me think better?", "did this actually turn out to be valuable?" and "which things withstood further and extended scrutiny?".

We've done this 4 times so far (2018 [LW · GW], 2019 [LW · GW], 2020 [LW · GW], 2021 [LW · GW]).

The full technical details of how the Annual Review works are in the final section [LW · GW] of this post, but it's basically the same as the past few years. There are three phases:

1. Preliminary Voting Phase (2 weeks, Dec 4 — 17): We identify posts especially worthy of consideration in the review casting preliminary votes. Posts with 2 preliminary votes move into the Discussion Phase.
2. Discussion Phase (4 weeks, Dec  17 — Jan 14): We review and debate posts. Posts that receive at least one written review move to the final voting phase.
3. Final Voting (2 weeks, Jan 14 — Jan 28): We do a full voting pass, using quadratic voting. The outcome determines the Annual Review results.

For more of the philosophy of the Annual Review, see the previous announcement posts here [LW · GW], here [LW · GW], here [LW · GW], and here [LW · GW].

Getting Started

At the top of any posts eligible for the review, you will see this: 

These will be your preliminary votes for the 2022 review. Posts need to get at least 2 preliminary votes (positive or negative) in order to move to the next phase of the review.

To start perusing posts, I recommend going to the All 2022 Posts page [? · GW], or the View Your Past Upvotes [? · GW] page. Note: only users with accounts registered before January 2022 are eligible to vote.

No books this year, sorry folks

For 2018, 2019, and 2020 we printed books of the results of the review. We have sold many thousands of them, I am very proud of them, and many people told me that these are among the favorite things that they own:

Sadly, there won't be a book this year (and also not of the 2021 review). The effort involved in making them is hard to justify with increasing demands from many of our other projects (as well as reduced funding, since if you take into account the 4-5 staff months these cost to make each year, we net lost money on these).

I am thinking about other ways to create an easy to reference artifact that captures the results of this year's and last year's review. I think the minimum I want to do is to create a good ebook and maybe an audible version using our machine narration (or doing human narration). Additional suggestions are welcome.

We are going to be doing a Christmas sale of all of the previous years' books in the next few days, and hopefully before Christmas we will also have a good ebook (and maybe even an audiobook version) available of last year's review results.

How does the review work?

Phase 1: Preliminary Voting

To nominate a post, cast a preliminary vote for it. Eligible voters will see this UI:

If you think a post was an important intellectual contribution, you can cast a vote indicating roughly how important it was. For some rough guidance:

• A vote of 1 means “it was good.” 
• A vote of 4 means “it was quite important”. 
• A vote of 9 means it was "a crucial piece of intellectual progress."

You can vote at the top of a post page, or anywhere the post appears in a list (like the All Posts page [? · GW], or the new View Your Past Upvotes [? · GW] page).

Posts that get at least one positive vote go to the Voting Dashboard, where other users can vote on it. You’re encouraged to give at least a rough vote based on what you remember from last year. It's okay (encouraged!) to change your mind later. 

Writing a short review

If you feel a post was important, you’re also encouraged to write up at least a short review of it saying what stands out about the post and why it matters. (You’re welcome to write multiple reviews of a post, if you want to start by jotting down your quick impressions, and later review it in more detail)

Posts with at least one review get sorted to the top of the list of posts to vote on [? · GW], so if you'd like a post to get more attention it's helpful to review it.

Why preliminary voting? Why two voting phases?

Each year, more posts get written on LessWrong. The first Review of 2018 considered 1,500 posts. In 2021, there were 4,250. Processing that many posts is a lot of work. 

Preliminary voting is designed to help handle the increased number of posts. Instead of simply nominating posts, we start directly with a vote. Those preliminary votes will then be published, and only posts that at least two people voted on go to the next round.

In the review phase this allows individual site members to notice if something seems particularly inaccurate in its placing. If you think a post was inaccurately ranked low, you can write a positive review arguing it should be higher, which other people can take into account for the final vote. Posts which received lots of middling votes can get deprioritized in the review phase, allowing us to focus on the conversations that are most likely to matter for the final result.

How is preliminary voting calculated?

You can cast an unlimited number of votes, but after a certain threshold, the greater the total score of your votes, the less influential each of your votes will be. On the back end, we use a modified quadratic voting system [LW · GW], which allocates a fixed number of points across your votes based on how strong they are.

Fine details: A vote of 1 costs 1 point. A vote of 4 costs 10 points. A vote of 9 costs 45 points. If you spend more than 500 points, your votes start to become proportionally weaker.

Phase 2: Reviews

The second phase is a month long, and focuses entirely on writing reviews. Reviews are special comments that evaluate a post. Good questions to answer in a review include:

• What does this post add to the conversation?
• How did this post affect you, your thinking, and your actions?
• Does it make accurate claims? Does it carve reality at the joints? How do you know?
• Is there a subclaim of this post that you can test?
• What followup work would you like to see building on this post?

Phase 3: Final Voting

Posts that receive at least one review move on the Final Voting Phase. 

The UI will require voters to at least briefly skim reviews before finalizing their vote for each post, so arguments about each post can be considered. 

As in previous years, we'll publish the voting results for users with 1000+ karma, as well as all users. The LessWrong moderation team will take the voting results as a strong indicator of which posts to include in the Best of 2022 sequence.

To get started, you can View Your Past Upvotes [? · GW] and start voting on some posts.

43 comments

Comments sorted by top scores.