The 2023 LessWrong Review: The Basic Ask

post by Raemon · 2024-12-04T19:52:40.435Z · LW · GW · 8 comments

Contents

  The Ask
    The concrete, minimal Civic Duty actions
  What's new?
  Reviews are more prominent
      Possible addition: Converting old "Review-like" comments. 
  Linkpost Submission UI
  Nominations Page
    Improved Best of LessWrong browsing
  Sliding Scale of Voting Power
  Nuts and Bolts: How does the review work?
    Phase 1: Preliminary Voting
      Writing a short review
      Why preliminary voting? Why two voting phases?
    Phase 2: Discussion
    Phase 3: Final Voting
None
8 comments

Each December, the LessWrong community reflects on the best blogposts of yesteryear, to decide which posts stood the tests of time. 

In this post, I aim to:

• Explain some subtleties of what I'm hoping we get out of the Annual Review
• Make some explicit asks (even of busy people), which I think have reasonable cost/benefit ratios.
• Describe the new UI features. 

The most important changes are:

• Reviews are much more prominent than they used to be
• You can nominate posts from elsewhere on the internet using the Linkpost Importer. (Use this to nominate posts that contribute to the overall LessWrong or Alignment Forum conversation). It can import PDFs from arXiV, and blogposts from most urls.

For the Nomination Voting Phase, the main ask I'm making is: "spend ~30 minutes casting nomination votes, and write 2 short reviews explaining how you got value from the most important posts."

 

The Review is a complex endeavor, serving many purposes. 

Most people understand the overt purpose, which is: "assemble a list of the best posts from 1-2 years ago. Highlight those posts at the top of LessWrong and the Alignment Forum. Help new or occasional readers keep up with important concepts."

I think most people have some sense of the secondary purpose, which is "shape the incentives of LessWrong to reward more timeless and important content." 

These are good things. But they're only ~half of the purpose of the Annual Review, according to me. The voting on the top posts is the stakes of the Review, but what I'm most excited about is the side effect of those stakes.

The side effect is: 

By voting and reviewing and arguing over two months, the LessWrong community becomes more of a place that engages deeply with multifaceted, longterm conversations, reflecting on how ideas fit together, synthesizing a bigger picture understanding. And in particular, they do this as a group, with a kind of collective orienting process.

The "side effect" is sort of the main point, but you can't legibly build a system to incentive people to "synthesize a deep understanding." Ultimately, they either do it, or they don't.

Every November, I wonder if this is all actually worth it. We could just have everyone do the initial vote, spend about a week on it, and them call it a day. But every year, once I get started reviewing posts, I find the process quite rewarding on its own merits. The Review helps me with my big picture thinking. And, I believe, the fact that many of LessWrong members participate creates something deeper than the (naive^[1]) sum of it's parts.

The Ask

If you're the sort of longterm member who's judgment would be valuable, but, because you're a smart person with good judgement, you're busy... here is what I ask:

First, do some minimal actions to contribute your share of judgment for "what were the most important, timeless posts of 2023?". (See next section)

Then, in proportion to how valuable they seem, spend at least some time this month reflecting...

• ...on the big picture of what intellectual progress seems important to you. Do it whatever way is most valuable to you. But, do it publicly, this month, such that it helps encourage other people to do so as well. And ideally, do it with some degree of "looking back" – either of your own past work and how your views have changed, or how the overall intellectual landscape has changed.
• ...on how you wish incentives were different on LessWrong. Write up your thoughts on this post. (I suggest including both "what the impossible ideal" would be, as well as some practical ideas for how to improve them on current margins)
• ...on how the LessWrong and X-risk communities could make some group epistemic progress on the longstanding questions that have been most controversial. (We won't resolve the big questions firmly, and I don't want to just rehash old arguments. But, I believe we can make some chunks of incremental progress each year, and the Review is a good time to do so)

In a future post, I'll share more models about why these are valuable, and suggestions on how to go about it.

The concrete, minimal Civic Duty actions

It's pretty costly to declare something "civic duty". The LessWrong team gets to do it basically in proportion to how much people trust us and believe in our visions. 

Here's what I'm asking of people, to get your metaphorical^[2] "I voted and helped the Group Reflection Process" sticker:

Phase I:  Nomination Voting 2 weeks	We identify posts especially worthy of consideration in the review, by casting preliminary votes. Posts with 2 positive votes move into the Discussion Phase. Asks: Spend ~30 minutes looking at the Nominate Posts [? · GW] page and vote on ones that seem important to you. Write 2 short reviews[3] explaining why posts were valuable.
Phase II: Discussion 4 weeks	We review and debate posts. Posts that receive at least 1 written review move to the final voting phase.  Ask: Write 3 informational reviews[4] that aim to convey new/non-obvious information, to help inform voters. Summarize that info in the first sentence.
Phase III: Final Voting 2 weeks	We do a full voting pass, using quadratic voting. The outcome determines the Best of LessWrong results. Ask: Cast a final vote on at least 6 posts.

Note: Anyone can write reviews. You're eligible to vote if your account was created before January 1st of 2023. More details in the appendix.

What's new?

In addition to general simplifying the UI, here are some major updates this year:

• Reviews get more prominence. Reviews with 10+ karma are now displayed on the Best of LessWrong [? · GW] page, and on spotlights at the top the LessWrong and Alignment Forum homepage.
• Linkpost Submission UI. It's now much easier to submit posts from other corners of the rationalsphere (i.e. alignment-relevant papers, Astral Codex Ten posts, etc).
• Streamlined Nominations Page. See all posts you voted on, commented on, or read.
• Improved Best of LessWrong Page. If you want to browse the winners of previous Review Votes, it's not easier to find posts relevant to your interests (previously you only had the title and illustrations to go off – now you can read a short summary of each post)
• Voting power scales smoothly. Previously, 1000+ karma users got 3x the voting power. This time there is a smoother curve with exponential decay.
• [Coming Soon] Best of Alignment Forum Page. Make it more straightforward to see the best Technical AI Safety work.

I will likely attempt to build some additional tools for the Discussion phase, but haven't yet made commitments about that. (I'm particularly interested in trying to build "power tools for using the Review to help you think big picture," such as making it easier to briefly skim lots of posts and remember the key points of each one.)

Reviews are more prominent

I think the biggest problem with past Annual Reviews is there wasn't that great an incentive to write thoughtful, effortful reviews. We patched this by manually paying out prizes, but I'd like the system to more naturally encourage it.

An issue is that reviews kinda disappear-into-the-void. People might pay a bit of attention when you first write them, but years later when people get linked the post, they won't necessarily see your extensive critique.

Now reviews with 10+ karma will appear wherever Best of LessWrong Spotlights appear (including on the /bestoflesswrong page, and at the top of the home page^[5]). So if a post was generally upvoted, but, you wrote a review highlighting an important flaw, then your comment will be listed along side it if your review gets upvoted.

This is meant to incentivize both writing and voting on reviews.

(Corollary: Try to convey the most important information in the first sentence, so that people skimming the /bestof page can see it without having to click-to-expand)

Possible addition: Converting old "Review-like" comments. 

Some posts received a fairly in-depth "review" like comment back when it was first published. I think it's probably reasonable to highlight those as well. I haven't yet worked out a principled way of converting them, the but the LessWrong team might manually convert some, and I meanwhile encourage people to reply to older comments that seem like a good fit to help us notice them.

Linkpost Submission UI

A lot of the major conversations on LessWrong partially happen in other places. 

Scott Alexander doesn't crosspost all Astral Codex Ten posts, but it seems like important ones should get featured here. And there are various other bloggers in that reference class where, a discussion of the latest on coordination, decision-theory, or rationality training would be incomplete without them.

In particular, a lot of AI Existential Safety research is done in places other than LessWrong. I think it's important for LessWrong and the Alignment Forum to be orienting to the entire literature, not just the parts that were shaped like blogposts and posted to LW/AIAF. In the broader world, some alignment-ish research is deeply important, some is irrelevant, and some is mostly some kind of political statement. Having a conversation about all of that seems worthwhile.

To submit a linkpost, you enter a URL on the Nominations page, and it'll automatically import the post, attempt to infer the post's title and date of publication. 

It'll be saved as a draft, with a rough attempt at importing the page's content, which you'll need to clean up. 

So: I highly encourage you to submit all writing that you think has been important to the progress of rationality, AI alignment, or other important threads from the rationalsphere.

You'll need to write a short review explaining why the post is relevant. 

By default the linkposts will list you as a coauthor (so you can edit to have useful highlights), but the LessWrong team will review linkposts as they're submitted and (in some cases) attempt to connect them to the correct author.

Nominations Page

We've consolidated various tools to help you find posts worth nominating into the Nominations Page. Previously we showed people posts-they-voted on, but not everyone votes that much. So now we also include posts you commented on, or read. (In addition to providing the new "submit linkpost" UI)

Improved Best of LessWrong browsing

You might recall the Best of LessWrong [? · GW] page looking like this:

Some folk complained that this felt very "judge a book by it's cover" – for clicking on posts all you had were the titles, authors and images to go off of. This felt a bit off for a site encouraging deep intellectual engagement, especially on a page celebrating posts we'd especially deeply-intellectually-engaged-with.

We shipped some updates such that, if below the "book covers", you'll see a nicer table-of-contents:

Sliding Scale of Voting Power

The past few years, everyone with 2+ year old accounts could vote in the Review, but the vote weight of people with 1000 karma was counted more strongly (3x).

I haven't been happy with the arbitrary cliff here. It gives more power than I really wanted to allocate to people with 1000 karma, and not enough weight to people who have been around much longer and have demonstrated good judgment. But, karma is still a pretty messy indicator, so I don't want to give too much power to high karma users either.

I'm still^[6] ironing out the details, but I'm planning a ~log scale that looks something like:

0x: 0 karma
1x: 500 karma
2x: 1000 karma
3x: 2000 karma
4x: 4000 karma
...etc

---

Nuts and Bolts: How does the review work?

Phase 1: Preliminary Voting

To nominate a post, cast a preliminary vote for it. Eligible voters will see this UI:

If you think a post was an important intellectual contribution, you can cast a vote indicating roughly how important it was. For some rough guidance:

• A vote of 1 means “it was good.”
• A vote of 4 means “it was quite important”.
• A vote of 9 means it was "a crucial piece of intellectual progress."

Votes cost quadratic points – a vote strength of "1" costs 1 point. A vote of strength 4 costs 10 points. A vote of strength 9 costs 45. If you spend more than 500 points, your votes will be scaled down proportionately.

Use the Nominate Posts page to find posts to vote on. 

Posts that get at least one positive vote go to the Voting Dashboard, where other users can vote on it. You’re encouraged to give at least a rough vote based on what you remember from last year. It's okay (encouraged!) to change your mind later. 

Posts with at least 2 positive votes will move on to the Discussion Phase. 

Writing a short review

If you feel a post was important, you’re also encouraged to write up at least a short review of it saying what stands out about the post and why it matters. (You’re welcome to write multiple reviews of a post, if you want to start by jotting down your quick impressions, and later review it in more detail)

Posts with at least one review get sorted to the top of the list of posts to vote on [? · GW], so if you'd like a post to get more attention it's helpful to review it.

Why preliminary voting? Why two voting phases?

Each year, more posts get written on LessWrong. The first Review of 2018 considered 1,500 posts. In 2021, there were 4,250. Processing that many posts is a lot of work. 

Preliminary voting is designed to help handle the increased number of posts. Instead of simply nominating posts, we start directly with a vote. Those preliminary votes will then be published, and only posts that at least two people voted on go to the next round.

In the review phase this allows individual site members to notice if something seems particularly inaccurate in its placing. If you think a post was inaccurately ranked low, you can write a positive review arguing it should be higher, which other people can take into account for the final vote. Posts which received lots of middling votes can get deprioritized in the review phase, allowing us to focus on the conversations that are most likely to matter for the final result.

Phase 2: Discussion

The second phase is a month long, and focuses entirely on writing reviews. Reviews are special comments that evaluate a post. Good questions to answer in a review include:

• What does this post add to the conversation?
• How did this post affect you, your thinking, and your actions?
• Does it make accurate claims? Does it carve reality at the joints? How do you know?
• Is there a subclaim of this post that you can test?
• What followup work would you like to see building on this post?

In the discussion phase, aim for reviews that somehow give a voter more information. It's not that useful to say "this post is great/overrated." It's more useful to say "I link people to this post a lot" or "this post seemed to cause a lot of misunderstandings." 

But it's even more useful to say "I've linked this to ~7 people and it helped them understand X", or "This post helped me understand Y, which changed my plans in Z fashion" or "this post seems to cause specific misunderstanding W."

Phase 3: Final Voting

Posts that receive at least one review move on the Final Voting Phase. 

The UI will require voters to at least briefly skim reviews before finalizing their vote for each post, so arguments about each post can be considered. 

As in previous years, we'll publish the voting results for users with 1000+ karma, as well as all users. The LessWrong moderation team will take the voting results as a strong indicator of which posts to include in the Best of 2023, although we reserve some right to make editorial judgments.

Go to the Nomination Page to get started!

1. ^{^}

   Oliver Habryka insisted that things are almost never greater than the sum of their parts Because Reductionism and that I should add this disclaimer.

2. ^{^}

   Maybe also literal but I haven't done the UI design yet.

3. ^{^}

   In previous years, we had a distinction between "nomination" comments and "review" comments. I streamlined them into a single type for the 2020 Review, although I'm not sure if that was the right call. Next year I may revert to distinguishing them more.

4. ^{^}

   These don't have to be long, but aim to either a) highlight pieces within the post you think a cursory voter would most benefit from being reminded of, b) note the specific ways it has helped you, c) share things you've learned since writing the post, or d) note your biggest disagreement with the post.

5. ^{^}

   Awkwardly, during Review Season we actually hide the spotlights to make room for the complex Review UI on LessWrong.com, but, you can see how it'd look on AlignmentForum.org. Note that not all spotlights had reviews with 10+ karma, and on AlignmentForum it shows the Ω Karma (which is less than the LessWrong karma)

6. ^{^}

   It'll be finalized by the end of the Nomination Voting phase.

8 comments

Comments sorted by top scores.