Voting Phase of 2018 LW Review

The list is now shuffled (as a tiebreak after sorting by your own vote). The shuffle is done once per user, so each user should see the posts in a random order, but it'll be the same order each time you revisit it. This change went live around the 13th.

[EDIT: When I say that posts earlier in the list got 25-50% more votes, I mean simply the number of non-neutral votes cast on those items, regardless of direction of magnitude. It would perhaps be more accurate to say these posts had 25-50% more people vote on them.]

The posts available for review are presented in (what I guess is) a consistent order that is (so far as I know) the same for everyone. I expect this to mean that posts presented earlier will get more votes.

Good call. I looked into this and found an effect of somewhere between 25-50% more votes for posts being displayed earlier in the list. The team rolled out a fix to randomize loading this morning.

Interestingly, the default sort order was by number of nominations in ascending, so the most heavily nominated (approx, the most popular) posts were being displayed last. These posts were getting as many votes as those at the beginning of the list (though possibly not as many as they might have otherwise), and it's the posts in the middle that were getting less.

This was an oversight which we're glad to have caught. We've around halfway through the voting, plus the second half will have the deadline rush, so hopefully this bias will get countered in the coming week. 

Unfortunately you just make mistakes the first time you're doing things. :/

You don't have to vote on all posts, you don't have to have read all the posts. I think it's fine for the vote to correlate with what posts people actually read. The default vote on all is ‘neutral’, which equals zero quadratic votes. I voted on about half of the posts, and I think I read way more than most people.

Edit: Rewritten.

I think it would make sense if you weakly vote on them, by spending relatively few points of your quadratic budget on them. Voting very strongly on them feels wrong to me. Basically, vote in strength proportional to your confidence times the goodness/badness of your assessment of the post, would be my guess.

I also note that I think there's signal in your decision to only skim a post, as opposed to reading it, but as noted in habryka's response, it's probably a weak signal.

Woop! I did the same yesterday.

woo!

We see information about how much individuals vote (i.e. total amount of points you’ve assigned and how many posts you’ve voted on), but not the details of which votes.

The current plan is something like: carve up vote totals from a couple different cross sections (i.e. check if there’s a difference between how the highest karma users voted vs the overall vote score, and probably checking what AF users thought about AF posts). I expect us to make those totals public.

I’m not sure about all the ways we will look at data but I expect the general principle will be “look for ways to to get insights about how people used it, while keeping everyone anonymized”.

I realize we didn't justify the Voting very hard. Here's my offhand attempt, which maybe we'll roll into the actual post after chatting about it more on Monday.

LessWrong runs, for good or for ill, off the same forces much of the rest of the internet runs on: people who are slightly bored at work. Naturally, posts get rewarded mostly by upvotes and comments, which disproportionately reward things for being exciting and for controversial (respectively). These are quite easy to goodhart on.

The Review (in general), and Voting (in particular) are an attempt to do a more nuanced thing – to take the accumulated taste of the LessWrong community, and use it to reflect hard on what was actually good, and then backpropagate that signal through people's more general sense of "what sort of posts are good to write and why?"

Without the Vote, the signal would basically be entirely "what the Mod Team Thinks Was Best", or, if we weren't doing this at all "what posts were memorable, and/or high karma". And this isn't ideal for a few reasons:

• The Mod Team doesn't have domain expertise in all the areas that posts explore
• Even though we're putting a lot of work into it, it's still a really daunting project to form opinions on all 75 posts. Having a mixture of people who've looked harder at different posts helps give more coverage of nuanced opinions.
• Something something wisdom of crowds – each person is biased in some way, or has different knowledge. Getting many people to participate helps counterbalance various knowledge and biases that individuals have.

I meanwhile expect the voting here to be better than usual karma-voting, because it's more comparative. You're not just voting on "this post seems good!" but "this post seems better than this other post". What I found useful for my own voting was being forced to stop and think and build a model of what-sorts-of-posts-are-good-and-why.

When it comes to the interface I think it would be great if the interface would show me my past karma votes on the post. It's useful to have the information of how I found the post after reading it the first time at hand when trying to evaluate 75 posts at once.

Yeah, I definitely agree with this. I think we've put about as much work into the UI as we're going to this year (I originally budgeted a couple weeks of time for the Review and ended up spending 1.5 months on it, but I think, assuming it stays in roughly the same form next year, this is an obvious thing to include)

Oh, I think a) that may get fixed (or at least replaced by newer and more interesting bugs) in another 30 minutes when our current deploy finishes, and b) in the meantime, try scrolling on alternate parts of the page? What I just found was that the scroll depended on whether my cursor was within a particular sub-window.

Alas, will look into it.

Should be fixed now.