The 2020 Review

post by Raemon · 2021-12-02T00:39:19.191Z · LW · GW · 39 comments

Contents

      A few important announcements about this year’s review:
  How does the review work?
    Preliminary Voting
      Why did we switch to preliminary voting?
      How is preliminary voting calculated?
    Final Voting
    Donation Buttons
    The 2019 Books
    Voting on Important Intellectual Progress
    Go Forth and Review!
None
39 comments

Today is the first day of the LessWrong 2020 Review. At the end of each year, we take a look at the best posts from the previous year, and reflect on which of them have stood the test of time. 

As we navigate the 21st century, a key issue is that we’re pretty confused about which intellectual work is valuable. We have very sparse reward signals when it comes to “did we actually figure out things that matter?”

The Review has a few goals. It improves our incentives, feedback, and rewards for contributing to LessWrong. It creates common knowledge about the LW community's collective epistemic state about the most important posts of 2020. And it turns all of that into a highly curated sequence that people can read. You can read more about the philosophy behind the Review in last year’s announcement post [LW · GW].

A few important announcements about this year’s review:

• We’ve replaced the nomination process with Preliminary Voting [LW · GW].
• Winning posts will get Donation Buttons [LW · GW].
• There’s a new View My Past Upvotes [? · GW] page to help you find posts to vote on.
• 2019 Books will be shipping in a couple weeks. We’re still evaluating whether and how to do books for 2020.

How does the review work?

The review has three phases:

1. Preliminary Voting Phase (Dec 1- 14)
2. Discussion Phase (Dec  14 - Jan 11)
3. Final Voting (Jan 11 to Jan 25)

Users who registered before January 1st 2020 can vote. The LessWrong moderation team will take the results of the vote as input for a curated sequence of posts, and award prizes. We’ll be giving more weight to the votes of users with 1000+ karma.

Preliminary Voting

The first big change this year is changing the Nomination Phase to the Preliminary Vote Phase. Eligible voters voters will see this UI:

If you think a post was an important intellectual contribution, you can cast a vote indicating roughly how important it was. A vote of 1 means “it was good.” A vote of 4 means “it was quite important”, and is weighted 4x a vote of 1. A vote of 9x means it was a crucial piece of intellectual progress.

You can vote at the top of a post, or anywhere the post appears in a list (like the All Posts page [? · GW], or the new View Your Past Upvotes [? · GW] page).

Posts that get at least one positive vote go to the Voting Dashboard, where other users can vote on it. You’re encouraged to give at least a rough vote based on what you remember from last year.

If you feel a post was important, you’re also encouraged to write up at least a short review of it saying what stands out about the post and why it matters. (This is essentially the same as writing a nomination comment from the 2018 and 2019 Reviews. In practice nominations and reviews were fairly similar and it didn’t seem worth separating them out in the UI). You’re allowed to write multiple reviews of a post, if you want to start by jotting down your quick impressions, and later review it in more detail.

Why did we switch to preliminary voting?

Each year, more posts get written on LessWrong. The first Review of 2018 considered 1,500 posts. In 2020, there were 3,000. Processing that many posts is a lot of work. 

Preliminary voting is designed to help handle the increased number of posts. Instead of simply nominating posts, we start directly with a votes. At the end of the Preliminary Voting phase, the results of the vote will be published. This will help the LessWrong community prioritize reviews. Posts that are highly ranked can invite more investigation of how they stand the tests of time. If you think a post was (unfairly) ranked low, you are welcome to write a positive review arguing it should be considered more strongly.

Posts which everyone agrees are “meh” can get deprioritized, making more time for more interesting posts.

How is preliminary voting calculated?

You can cast an unlimited number of votes. However, the more votes you cast, and the higher your total “score” (where a “9” vote counts for 9x the score of a “1” vote), the less influential each of your votes will be. We normalize voting strength so that all users who are past a certain “score” threshold exert roughly the same amount of total influence.

On the back end, we use a modified quadratic voting system [LW · GW], which allocates a fixed number of “points” across your votes based on how strong they are.

Final Voting

Posts that receive at least one review move on the Final Voting Phase. The UI will require voters to at least briefly skim reviews before finalizing their vote for each post, so arguments about each post can be considered. 

As with last year, we'll publish the voting results for users with 1000+ karma, as well as all users. The LessWrong moderation team will take the voting results as a strong indicator of which posts to include in the Best of 2020 sequence.

(Note: I am currently uncertain whether Final Voting will use the fine-tuned quadratic system from last year. I plan to take last year's voting data, round each vote to the nearest "1, 4, or 9", and see if the results are significantly different from the original vote. If they aren't very different, I suspect it may not make sense to encourage everyone to spend a bunch of time fine-tuning their quadratic points. I'm open to arguments in either direction)

Donation Buttons

Something I’d like LessWrong to do better is to allow authors to transition from hobbyists, to professionals that get paid to research and write full time.

Earlier this year, I was thinking about whether LessWrong should become more like substack, where there’s an easy affordance to start supporting financially supporting authors you like. I liked the idea but wasn’t sure it’d be healthy for LessWrong – the sorts of posts that make people excited to donate are often more tribal/political. But this seemed less worrisome during The Review. It’s a time when people are thinking holistically about the LessWrong intellectual world, comparing many different posts against each other and reflecting on which ones were truly valuable. 

So, after the Final Vote this year, all posts above some[1] threshold will get a donation button interface, which makes it easier people to just give the author money. I encourage everyone to donate in proportion to how much value you got from a post. If it slightly improved your life, maybe donate $20-$50 as a thank you. If you think a post was a crucial insight for helping the entire world, maybe donate as if it were an effective altruism target. (i.e. if you’re the sort of person who donates 10% of your income, consider if any LessWrong posts are competitive with the other causes you might give to). LessWrong posts are a public good, and I think at least some are worth supporting in this way.

Lightcone Infrastructure [LW · GW] will be allocating our own prizes. We have not decided the total amount we’ll give, but it will most likely be substantially more than the $2000 we awarded last year.

[1] I’m not yet sure exactly what threshold to set. I’m expecting a lot of mediocre posts to get at least one positive vote, which shouldn’t automatically warrant inclusion in donations list.

The 2019 Books

The 2018 books [LW · GW] were well received last year, selling out almost the entire 4000 sets we printed (though there are still some 300+ copies in Australia, available on Amazon there).

The 2019 year’s books are a week or two away from launch. They include 59 essays, each of which has a unique customized illustration generated by machine learning. They’ll be eligible again for Amazon Prime, so shipping will be fast in North America, and likely in time for Christmas. A little later we’ll be supplying books to Amazon UK, which is where European readers can order from (with slightly longer shipping times and prices). 

We will not ourselves be shipping to every other country – last year we attempted to ship to ~25 countries, most of which sold very few copies while requiring a lot of setup work. Alas, we are LessWrong, not Santa Claus – we unfortunately exist and are subject to logistical constraints. :P

At this time we’re not committed to doing another anthology set next year. We’re going to wait until after the launch of this year’s books to see whether there’s demand for annual anthologies. We have some different book projects in mind for the community, including a book of The Core Sequences [? · GW], or entire sequences by other authors that fare well in the review, or books dedicated to a single topic drawing from the full history of LessWrong (covering topics such as Coordination or AI Alignment).

Meanwhile, we’ll definitely be collating the winning 2020 essays into a proper LessWrong sequence, prominently displayed in the site library. (I expect to have the 2018 and 2019 sequences released later this week). And again, we’ll be awarding significantly more financial prizes this year, and facilitating donation buttons to make it easier to reward authors who have done good work.

Here’s a sneak peak of the spines of the upcoming books, which includes this year’s volume titles (Book 1 is on the bottom). This year’s books are notably bigger than last year’s, 60% bigger in terms of page size.

Voting on Important Intellectual Progress

In past years, the vote was officially for creating a published book. This made it easier to reason about what exactly you were voting for, but also meant that some types of posts were harder conceptually to reward. Some important progress isn’t very fun to read. Some important posts are massively long, and couldn’t possibly fit in a book.

So this year, I’d like to formally ask that you vote based on how important an intellectual contribution a post made, rather than whether you think it makes sense to publish.

The LessWrong moderation team will take stock of the top-rated posts, and make judgment calls on how to best reward them. Some may fit best into anthology style books. Some may be more appropriate for (eventual) textbooks. Some might be important-but-tedious empirical work that makes more sense to give an honorable mention to in the books, while primarily rewarding them with prize money.

In practice, this is not that different from how we’ve been assembling the books in previous years. But it had been a bit ambiguous, and I thought it best to make it official.

You are welcome to use your own taste in what you consider important intellectual progress. But some questions that might inform your vote include:

• Does this post introduce a concept that helps you understand the world?
• Does the post provide useful and accurate empirical data?
• Does this post teach a skill that has helped you?
• Does this post summarize or distill information that makes it easier to grasp?
• Do the central arguments of the post make sense?
• Does this post promote an important and interesting hypothesis?

While writing reviews, it’s also worth exploring questions like:

• How does this fit into the broader intellectual landscape?
• What further work would you like to see?

Go Forth and Review!

I have more ideas for how to improve the Review this year, which I’ll be posting about as they reach fruition. Meanwhile, let the LessWrong 2020 Review commence! 

Head over to your past upvotes page [? · GW] and start voting!

 

 

 

39 comments

Comments sorted by top scores.