The Great Karma Reckoning

post by habryka (habryka4) · 2021-01-15T05:19:32.447Z · LW · GW · 15 comments

Today the LessWrong team ran a script that recalculated the scores and karma of all posts, comments, tags and users based on the complete vote history. This has resulted in very substantial changes to all the karma scores on the site. Here are the biggest changes: 

We've been planning on doing this for a long time, mostly to make historical comparisons of the scores of different posts easier and also have more confidence that current karma balances are not the result of bugs we fixed many months ago. 

For the purpose of the 2019 review, you will be counted in the 1000+ vote bracket if you either had 1000+ karma before we ran todays migration, or if you just hit that target after today's migration. So nobody should suddenly be negatively surprised about their ability to participate in the vote. 

15 comments

Comments sorted by top scores.

comment by Zvi · 2021-01-15T19:47:26.813Z · LW(p) · GW(p)

Going to take John's comment and this reckoning as a good time to say that while 10x is a large multiplier on top-level front-page posts, 1x is not a large multiplier, and to the extent that karma matters comments are getting too large a share.

Replies from: Dagon, habryka4
↑ comment by Dagon · 2021-01-15T22:31:25.444Z · LW(p) · GW(p)

Front-page posts don't necessarily need a multiplier to be more valuable - they already get a lot more visibility,.  A larger number of voters will see and consider voting on a top-level or front-page post than any comment or short-form.

A lot comes down to "why do we track accumulations of karma from the past in the first place?" I'd love to see a reasoning for what levels various posters should have, and then backwards-calculate (fit) a scoring mechanism into that.

Replies from: Raemon
↑ comment by Raemon · 2021-01-15T23:32:34.554Z · LW(p) · GW(p)

Insofar as "total karma should try to approximate ideal 'Rationalist Social Status'", I think ideally it would incorporate things like "how often do they introduce novel information that turns out to actually be true." 

And this suggests a bunch of things like "tracking their predictions / success" and "noticing when their early ideas were relevant to things that later get well established as true." Which all seems like important things to figure out how to do. But I'm not sure whether it fits into the abstraction of what karma currently is. One key goal of karma is "dole out bits of reward that create a positive incentive gradient to follow", which is a pretty different goal than "track total idealized social status."

Currently we go out of our way to not make people's total karma super prominent (it doesn't appear when you mouseover their username, nor does your karma appear at the top of each page, like it did on old lesswrong).

Replies from: Dagon
↑ comment by Dagon · 2021-01-16T00:48:09.053Z · LW(p) · GW(p)

Insofar as "total karma should try to approximate ideal 'Rationalist Social Status'"

Cool, that's a good goal.  The ideal approach to implementation would be propose the social status of, say 2000 randomly-selected posters, and then to run a regression over lots of variables to figure out how to predict that from posts.  I don't think that starting with "a function of votes on posts and comments" is likely to get there - too many other dimensions play into it.

I suspect my karma total is ridiculously high in terms of social status.  It's huge just because I've been here a long time.  

↑ comment by habryka (habryka4) · 2021-01-15T21:21:04.121Z · LW(p) · GW(p)

Yeah, this seems reasonable. One of the nice things about this migration is that it's now very easy for us to adjust the rules and then just rerun the history again. So now is a pretty good time for suggestions for how the rules should change.

comment by ryan_b · 2021-01-15T15:46:48.479Z · LW(p) · GW(p)

Reviewing some older posts current scores, I am in that weird place where I have to un-do some updating about community preferences. It is mildly gratifying to have been pretty accurate in the first place, though.

Replies from: Raemon
↑ comment by Raemon · 2021-01-15T18:38:05.949Z · LW(p) · GW(p)

Curious for examples.

Replies from: ryan_b
↑ comment by ryan_b · 2021-01-15T19:54:55.696Z · LW(p) · GW(p)

The two I immediately thought to check were 1, [LW · GW] 2. [LW · GW] I fully expected these posts to be at least controversial, with a substantial chance of negative score, but they outperformed my expectations. Now that the karma has been normalized, the ratio of vote:score is much more consistent with the some people liked it and some people didn't expectation I had.

I note the breakdown is also consistent with a medium amount of small upvotes and very little dislike, but considering the content I was betting on at least a few strong downvotes.

comment by John_Maxwell (John_Maxwell_IV) · 2021-01-15T14:59:35.116Z · LW(p) · GW(p)

We removed the historical 10x multiplier for posts that were promoted to main on LW 1.0

Are comments currently accumulating karma in the same way that toplevel posts do?

Replies from: Vaniver
↑ comment by Vaniver · 2021-01-15T18:02:50.372Z · LW(p) · GW(p)

I think this is a little sad (in years past, I definitely put more effort into posts because of that sweet sweet 10x). I remember thinking that this doesn't do all that much to change the relative ranking of users, and so it's not clear it's worth the code complexity, but if it were free I personally would like some smoother gradation (like 2x for posts, another 2x for frontpage, another 2x for curated).

comment by jimrandomh · 2021-01-15T06:04:58.294Z · LW(p) · GW(p)

Note that even things where the scores aren't affected directly may still change score, because the vote-strength of users who voted on them have changed. The karma-change notifier (the star icon in the top-right corner of the page) won't notify you of these changes, as it works by looking at the recent votes themselves rather than at computed scores.

comment by habryka (habryka4) · 2021-01-15T05:21:51.643Z · LW(p) · GW(p)

(I am promoting this site-meta post to the frontpage, so that users who have personal blogposts filtered out aren't surprised that suddenly a lot of scores are different. But generally we try to keep most site-meta stuff that isn't crucial for people to know about off the frontpage.)

comment by Dagon · 2021-01-15T17:30:37.877Z · LW(p) · GW(p)

I was hoping, from the title, that this was an announcement that there was some reward for high or (mild) penalty for low accumulated karma, to be followed by wiping out all karma and starting everyone even.  Where's the reckoning I was promised?

More seriously, I'm glad you're willing to retroactively change the results of votes, and I support continuing to tweak things.

Replies from: Vaniver
↑ comment by Vaniver · 2021-01-15T17:59:15.500Z · LW(p) · GW(p)

From Wiktionary:

reckoning (plural reckonings)

  1. The action of calculating or estimating something.

Sometimes words are meant literally :P

comment by shanen · 2021-02-15T19:22:41.453Z · LW(p) · GW(p)

This topic of karma in general interests me, per my reaction to the karma project from 2019 [LW(p) · GW(p)]. However my question in response to this "site meta" item is: "Is there a karma explorer?" One side would be a way to see the basis of my own karma, but I would also like a way to understand the basis of the karma of other users. For example, I see that the author habryka has over 13,000 points of karma here and 242 points of karma somewhere else, but what does that actually mean? Does any of that karma represent reasons I should read comments from habryka with greater attention? (Right now it feels like there are a lot of magic numbers involved in karma calculations?)

More fuzzy reaction, but I feel like whatever forms the basis of karma, it should age over time. Recent contributions to karma should matter more than old ones.