Easy Predictor Tests

post by MrHen · 2010-01-21T18:40:21.079Z · LW · GW · Legacy · 66 comments

Contents

66 comments

A fun game you can play on LessWrong is to stop just as you are about to click "comment" and make a prediction for how much karma your comment will receive within the next week. This will provide some quick feedback about how well your karma predictors are working. This exercise will let you know if something is broken. A simpler version is to pick from these three distinct outcomes: Positive karma, 0 karma, negative karma.

What other predictors are this easy to test? Likely candidates match one or more of the following criteria:

A more difficult challenge is predicting karma on your top level posts. My predictors in this area tend to be way off the mark. For this post, my guess is between +4 and +20. Reasoning: I don't see how it could get over 20 unless it gets promoted and the concept surprises readers; 4 seems like a solid guess for "interesting, uncontroversial, but not groundbreaking."
Update: As of January 29, this post is at +10.

66 comments

Comments sorted by top scores.

comment by dclayh · 2010-01-21T18:53:43.888Z · LW(p) · GW(p)

A simpler version is to pick from these three distinct outcomes: Positive karma, 0 karma, negative karma.

Personally, if I expect a comment or post to get negative karma, I don't make it. Perhaps that's too conservative on my part, but it's the case.

Replies from: thomblake, Unknowns, ciphergoth, AngryParsley, komponisto
comment by thomblake · 2010-01-21T19:56:07.050Z · LW(p) · GW(p)

I make plenty of comments for which I expect negative karma. It's not too hard to shield unpopular views in language that will soften the karma hit, but sometimes it's better to state things plainly.

For a while, I had a rough handle on how various factors influenced the karma of a comment (like a link to a previous Lw post, or a relevant XKCD) but never crunched any numbers.

Replies from: GuySrinivasan, Douglas_Knight, whpearson, wedrifid
comment by GuySrinivasan · 2010-01-21T23:48:52.882Z · LW(p) · GW(p)

My intuition is that if you are about to make a comment for which you expect negative karma, nearly all of the time you should instead choose one of: a) send the comment in a private communication b) expand your comment with a more clear explanation of where you're coming from c) make your comment into a top-level post (or posts if the idea needs introduction) complete with your arguments and reasoning for all to learn from without the miscommunication perils of quick comments d) refrain from commenting

Sure, I'd bet there are exceptions, but making plenty of comments for which you expect negative karma? Strikes me as off. Pros: you can comment faster and more often since you don't need to thoroughly explain yourself, and since you don't need to prune as many threads. Cons: negative karma presumably means readers would rather have not read what you wrote, which is a bad sign unless maybe it's a topic that you feel will actually improve their lives even without sufficient explanation, even when after reading it they decide they wish they hadn't.

That last bit? Sure, maybe it happens, sometimes... maybe.

Replies from: wedrifid
comment by wedrifid · 2010-01-22T00:04:45.721Z · LW(p) · GW(p)

Cons: negative karma presumably means readers would rather have not read what you wrote, which is a bad sign unless maybe it's a topic that you feel will actually improve their lives even without sufficient explanation, even when after reading it they decide they wish they hadn't.

Not necessarily. You can, for example, make comments that you expect to improve the lives of a majority despite the displeasure of a majority.

comment by Douglas_Knight · 2010-01-22T02:24:54.102Z · LW(p) · GW(p)

I make plenty of comments for which I expect negative karma.

You are not well-calibrated.

comment by whpearson · 2010-01-21T21:42:51.776Z · LW(p) · GW(p)

I've been mulling over asking for a data set of posts to do some stats on. I've noticed that short pithy statements can get a lot of karma, and would like to see some stats of karma vs comment length.

Replies from: Bindbreaker
comment by Bindbreaker · 2010-01-21T23:57:11.534Z · LW(p) · GW(p)

I suspect that short, concise posts and long, thought-out ones both get higher karma than ones that fall in between.

comment by wedrifid · 2010-01-21T23:54:39.274Z · LW(p) · GW(p)

I make plenty of comments for which I expect negative karma. It's not too hard to shield unpopular views in language that will soften the karma hit, but sometimes it's better to state things plainly.

I tend to share your philosophy. It is also not too hard to offset karma costs by making a few extra posts.

Interestingly, I am often surprised when posts that I am expecting to be a karma sacrifice turn out not to be. As you suggest, extra time phrasing potentially unpopular views probably makes some difference. Even stating things plainly can be done elegantly.

comment by Unknowns · 2010-01-21T20:15:28.870Z · LW(p) · GW(p)

Voted down for reasons suggested by thomblake: it's often important to make a comment that you expect to get negative karma.

comment by Paul Crowley (ciphergoth) · 2010-01-21T23:23:44.752Z · LW(p) · GW(p)

I made a comment recently that got a lot of karma, and I must confess, I did expect it. I printed out the thread and read it all before making it to make sure it wasn't already made, and I was quite proud of it, so I was really pleased people liked it.

comment by AngryParsley · 2010-01-21T18:56:56.483Z · LW(p) · GW(p)

Voted down just to ruin your expectation of positive karma (not really).

SHA-1 of my karma prediction for this post: 4154a31be35736ebed12ba65cbcb89cabc08b375

Replies from: Liron, AngryParsley, dclayh
comment by Liron · 2010-01-22T22:36:29.118Z · LW(p) · GW(p)

How will I ever guess the 7-bit message behind such a secure 160-bit hash?

Replies from: AngryParsley, ciphergoth
comment by AngryParsley · 2010-01-22T23:35:01.953Z · LW(p) · GW(p)

Nice try, but I phrased my prediction as a sentence. Granted, the search space of all sentences describing a karma prediction is smaller than the hash search space, but it's probably big enough that you won't be able to guess my prediction and spite me by voting in the opposite direction.

Replies from: ciphergoth
comment by Paul Crowley (ciphergoth) · 2010-01-23T09:24:28.712Z · LW(p) · GW(p)

Damn, should have read this before I checked!

comment by Paul Crowley (ciphergoth) · 2010-01-22T23:24:50.723Z · LW(p) · GW(p)

I confess I can't quite be bothered to check, but: it's not salted? oops!

comment by AngryParsley · 2010-01-28T23:31:23.586Z · LW(p) · GW(p)

ggreer@carbon:~$ echo Positive but less than 10 | shasum 4154a31be35736ebed12ba65cbcb89cabc08b375 -

0 isn't positive so I was wrong.

comment by dclayh · 2010-01-21T19:40:35.188Z · LW(p) · GW(p)

Voted down just to ruin your expectation of positive karma (not really).

Yep, I was expecting that reaction. :)

Replies from: wedrifid
comment by wedrifid · 2010-01-21T23:44:32.255Z · LW(p) · GW(p)

Yep, I was expecting that reaction. :)

You were obviously expecting the 'not really' part too!

Unfortunately your post is at +2 at the moment so it isn't worth me bothering with a downvote.

comment by komponisto · 2010-01-21T23:24:52.512Z · LW(p) · GW(p)

Personally, if I expect a comment or post to get negative karma, I don't make it. Perhaps that's too conservative on my part, but it's the case.

Me too. I remain as absolutely terrified of negative karma as I was when LW started. (In fact, it's possibly even worse now, since I have a reputation to uphold!)

Right now, I have a recent comment at -1, which for contextual reasons I can't delete, and it's driving me crazy.

(Yes, I know this is bad, and I shouldn't be like this. Unfortunately, I don't know how to change myself in this way.)

Replies from: Blueberry
comment by Blueberry · 2010-01-21T23:37:04.762Z · LW(p) · GW(p)

I remain as absolutely terrified of negative karma as I was when LW started.

I'm curious why you care.

Replies from: komponisto
comment by komponisto · 2010-01-21T23:53:53.819Z · LW(p) · GW(p)

I'm curious why you care.

My guess: humans are naturally wired to care about what others think; that part of my brain is hugely oversized or overactive.

It's not without its advantages, but there's a tradeoff. I get a huge motivational high when something is well received, but the price I pay is near-depression when something isn't.

Unfortunately the "exchange rate" for me is bad: downvotes are worth about five to ten times what upvotes are.

Replies from: wedrifid
comment by wedrifid · 2010-01-22T00:16:05.184Z · LW(p) · GW(p)

Unfortunately the "exchange rate" for me is bad: downvotes are worth about five to ten times what upvotes are.

This approximately matches the impact that positive and negative social exchanges have on people in general in person. (From my recollection of relevant studies read years ago.)

Replies from: pdf23ds, komponisto
comment by pdf23ds · 2010-01-22T07:40:24.238Z · LW(p) · GW(p)

Plus, downvotes actually do mean more, because people don't make them nearly as frequently.

comment by komponisto · 2010-01-22T00:42:57.628Z · LW(p) · GW(p)

Wow, that's interesting. I might like to read about that at some point.

Replies from: wedrifid, magfrump, bgrah449
comment by wedrifid · 2010-01-22T00:50:45.567Z · LW(p) · GW(p)

It isn't where I first encountered it but The No Asshole Rule: Building a Civilized Workplace and Surviving One That Isn't included a reference.

comment by magfrump · 2010-01-22T19:37:01.979Z · LW(p) · GW(p)

Also discussed in How Full is Your Bucket?

comment by nawitus · 2010-01-21T23:18:14.893Z · LW(p) · GW(p)

Somewhat offtopic, but I'd like to see someone writing a GreaseMonkey script which hides the name of the commenter and the current score level on all comments, so you're not being influenced by the status of the commenter and/or the current score level on that comment. The commenter name could be seen with a mouseover so you can reply to it though.

Replies from: JGWeissman
comment by JGWeissman · 2010-01-21T23:27:35.005Z · LW(p) · GW(p)

You seem to be describing Marcello's anti-kibitzer.

Replies from: Morendil
comment by Morendil · 2010-01-21T23:37:10.114Z · LW(p) · GW(p)

I've been using that for a while. It's, not so much surprising as interesting, how much it changes one's outlook.

It doesn't deal very well with the "Recent Comments" sidebar, I added a .5s delay to wait while the AJAX loads. That's not quite satisfactory, but I don't know how to tell a GreaseMonkey script to wait for a specific AJAX query to load.

EDIT: OK, now I know. It would be helpful if LessWrong could be upgraded to a more recent version of the Prototype framework.

comment by rolf_nelson · 2010-01-23T22:13:57.164Z · LW(p) · GW(p)

Cool idea. If there were an easy way to hide the scores on comments, you could wait a week to make the predictions and then immediately see if your predictions are correct. This would reduce the interval between making the prediction and seeing if it's correct, which presumably would provide better training and feedback.

comment by komponisto · 2010-01-21T23:39:44.951Z · LW(p) · GW(p)

A more difficult challenge is predicting karma on your top level posts.

That is difficult, and one of the reasons I'm very hesitant to post. (Luckily, my first three posts were "important" enough to me that I almost didn't worry about their scores before posting; as for the fourth, I somehow managed to predict the score almost exactly correctly.)

My predictors in this area tend to be way off the mark. For this post, my guess is between +4 and +20. Reasoning: I don't see how it could get over 20 unless it gets promoted and the concept surprises readers; 4 seems like a solid guess for "interesting, uncontroversial, but not groundbreaking."

I've been thinking it might be useful to establish a consensus on the "meaning" of various levels of post karma. Here's my top-of-the-head-suggestion:

0-10: Acceptable

10-20: Solid

20-30: Good

30 - 40: Quite Good

40+: Outstanding

100+: Unheard-of Stupendous Brilliance

I'd be curious to hear how this compares to others' views of post scores.

Replies from: orthonormal, MichaelGR, RobinZ
comment by orthonormal · 2010-01-22T01:15:27.735Z · LW(p) · GW(p)

Incidentally, this reminds me: there ought to be a specific term for the glee experienced when one's post or comment is voted into the karma stratosphere.

I suggest "Yvainity."

(Sorry, I couldn't resist. No offense to Yvain, whose posts deserve their high ratings!)

comment by MichaelGR · 2010-01-22T02:38:26.904Z · LW(p) · GW(p)

Don't forget karma inflation.

As more people join Less Wrong, more currency is also created, devaluing each unit of karma. This means that a post made a year ago that got 30 votes is getting more support than a post getting 30 votes today.

But at least we don't have a central bank yet...

Replies from: komponisto
comment by komponisto · 2010-01-22T03:22:40.758Z · LW(p) · GW(p)

But surely this is offset somewhat by the possibility of new members voting on old posts.

Replies from: MichaelGR
comment by MichaelGR · 2010-01-22T03:28:35.499Z · LW(p) · GW(p)

Indeed, but in that regard not all old posts are created equal.

Some are highly linked from more recent posts, from the about page, from the wiki and sequences page. Other are orphans that will only be found by a very small fraction of recent members.

I bet someone could do a thesis on the economics of karma in online communities (maybe it has already been done, in fact -- anyone has seen one?).

comment by RobinZ · 2010-01-22T01:19:26.189Z · LW(p) · GW(p)

I think of 0-10 as Weak and 10-20 as Acceptable, but otherwise similarly to you.

comment by Thomas · 2010-01-21T20:41:28.547Z · LW(p) · GW(p)

It will not remain 0.

Replies from: magfrump
comment by magfrump · 2010-01-22T19:41:16.302Z · LW(p) · GW(p)

Daaaaaaaaaang.

(note to future observers: parent was at 0 when I posted this)

Would have been downvoted had I not included this prediction.

comment by RobinZ · 2010-01-21T23:11:08.185Z · LW(p) · GW(p)

I will not predict that my prediction will be wrong. That would be silly.

I have a file in my home directory which includes my prediction, along with some extraneous text. md5 hash of this file: cc58112f13e9e92495782bac4a9443bc

Edit: wedrifid and AngryParsley have correctly informed me that md5 is broken. sha1 hash: 0cc4e8bd90a897c2f0d0c561780f69561b7af072

Replies from: RobinZ, wedrifid
comment by RobinZ · 2010-01-29T00:50:00.362Z · LW(p) · GW(p)

File.

Predicted 1, got 0.

Replies from: wedrifid
comment by wedrifid · 2010-01-29T02:23:48.820Z · LW(p) · GW(p)

Now that could have been an interesting one to bet on. All kinds of second guessing. Including calibrating the reward size compared to the overheads (ethical and practical) for cheating.

Replies from: RobinZ
comment by RobinZ · 2010-01-29T02:48:59.102Z · LW(p) · GW(p)

Were it practicable to prevent sabotage...

Replies from: wedrifid
comment by wedrifid · 2010-01-29T02:50:54.836Z · LW(p) · GW(p)

Just makes the prediction more difficult.

Replies from: RobinZ
comment by RobinZ · 2010-01-29T02:51:47.255Z · LW(p) · GW(p)

Was thinking on the other end, actually.

comment by wedrifid · 2010-01-21T23:49:55.486Z · LW(p) · GW(p)

MD5? Colluder!

Replies from: RobinZ
comment by RobinZ · 2010-01-22T01:08:31.240Z · LW(p) · GW(p)

I get ... a reference? All I know about MD5 is that (1) it shows up in the same UNIX man page as sha1, (2) it's a command in Macintosh UNIX, and (3) it's a cryptographic hash.

I'm really just ripping off AngryParsley.

Replies from: AngryParsley
comment by AngryParsley · 2010-01-22T01:17:27.198Z · LW(p) · GW(p)

MD5 isn't very useful as a cryptographic hash these days. It's not hard to find collisions for a given hash or create two plaintexts with the same hash. In fact, this has been used to create a rogue certificate authority. SHA-1 is looking pretty weak, but finding or constructing collisions with it is still infeasible.

Replies from: RobinZ
comment by RobinZ · 2010-01-22T01:26:48.166Z · LW(p) · GW(p)

To quote a certain BBC television presenter: oh, cock. Is there a better cryptographic hash than those two commonly available?

Edit: Let me clarify - of course they're out there, but I was hoping to pick something which is very easy to find, install, and use among those interested. (Anything with a reputable website I could link to for Windows & UNIX downloads would be fine, I'm sure.)

Replies from: AngryParsley
comment by AngryParsley · 2010-01-22T01:34:25.420Z · LW(p) · GW(p)

Yup. There's SHA-2 and some other algorithms. Right now the NIST is holding a contest for SHA-3. It's narrowed down to 14 candidates. The winner will be announced in 2012.

ETA: Really though, unless you're some super-spy, SHA-1 should be good enough until stronger hashes become common.

Replies from: RobinZ
comment by RobinZ · 2010-01-22T02:01:37.475Z · LW(p) · GW(p)

I don't even have SHA-2 on my computer - SHA-1 hash added to original comment.

Replies from: AngryParsley
comment by AngryParsley · 2010-01-22T02:21:05.501Z · LW(p) · GW(p)

shasum should support all the algorithms:

ggreer@carbon:~$ echo "The quick brown fox jumps over the lazy dog" | shasum -a 224 62e514e536e4ed4633eeec99d60f97b4d95889227975d975b2ad0de3 -

ggreer@carbon:~$ echo "The quick brown fox jumps over the lazy dog" | shasum -a 512 a12ac6bdd854ac30c5cc5b576e1ee2c060c0d8c2bec8797423d7119aa2b962f7f30ce2e39879cbff0109c8f0a3fd9389a369daae45df7d7b286d7d98272dc5b1 -

Take a look at the shasum man pages for more parameters.

Replies from: RobinZ
comment by RobinZ · 2010-01-22T03:35:31.130Z · LW(p) · GW(p)

Unfortunately, I don't have shasum on my MacBook - I had to use openssl.

Replies from: AngryParsley
comment by AngryParsley · 2010-01-22T03:38:55.142Z · LW(p) · GW(p)

Are you using an earlier version of OS X? I'm on 10.6 and it looks like shasum comes with 10.6.

Replies from: RobinZ
comment by RobinZ · 2010-01-22T03:47:57.801Z · LW(p) · GW(p)

Yes - 10.5.8. With drop damage, so it's not practical or economical to invest in an upgrade. (Am considering the purchase of a netbook as a replacement sometime in the unspecified probably-more-than-a-year-from-now future.)

comment by zero_call · 2010-01-21T23:44:56.457Z · LW(p) · GW(p)

I already have a simple, accurate method for predicting the karma values of my comments. Agree with EY and get positive karma, or disagree with EY and get negative karma. More generally, support status quo and get positive karma, or reject status quo and get negative karma.

Replies from: ciphergoth, Bindbreaker, thomblake, GuySrinivasan, wedrifid, Kaj_Sotala
comment by Paul Crowley (ciphergoth) · 2010-01-22T17:57:20.435Z · LW(p) · GW(p)

Yes, this community is really about nothing but slavish obedience; it's not worth your time to contribute here.

Replies from: bgrah449
comment by bgrah449 · 2010-01-22T18:15:56.226Z · LW(p) · GW(p)

He didn't say anything about the community, just a method of predicting karma values for his comments. He didn't make a value judgment or offer an explanation as to why that was. I'm pretty sure agreement with EY does correlate well with positive karma.

Replies from: ciphergoth
comment by Paul Crowley (ciphergoth) · 2010-01-22T23:28:20.463Z · LW(p) · GW(p)

It's an ADBOC thing. I've just learned there's a technical term for what I want to say here: Implicature - there is a reason for mentioning this particular correlation besides its truth.

Replies from: bgrah449
comment by bgrah449 · 2010-01-22T23:50:14.834Z · LW(p) · GW(p)

Connotations travel through value-judgments and through false dichotomies - for example, "These people do this, while good people have bad things happen to them." You are juxtaposing two groups, and you are relying on the audience to figure out what dimension you're showing two sides of. The latter half of that sentence says "Good people" and "Bad things," so the first half must be "bad people" and "good things."

The only value judgment I see in zero_call's statement is the word "simple."

comment by Bindbreaker · 2010-01-21T23:54:23.525Z · LW(p) · GW(p)

I don't find that that's necessarily correct. For example, this post of mine expressing skepticism about cryonics or this one questioning a highly rated post were both fairly highly rated. I think needless contrarianism gets downvoted, but reasonable arguments generally don't, even if they advance unpopular cases.

comment by thomblake · 2010-01-22T19:30:17.936Z · LW(p) · GW(p)

You'd be surprised at the amount of positive karma you can get from a well-phrased criticism of Eliezer, even ones that are relatively content-free. "I'll believe you know what you're talking about when you actually build an AI" is generally well-received.

comment by GuySrinivasan · 2010-01-21T23:53:40.834Z · LW(p) · GW(p)

A fine first-approximation, low-computation predictor! You can do better, though. For example, I'm pretty sure EY would agree that your method is simple and that it is fairly accurate (if nothing else, hopefully agreeing with EY or supporting the status quo has positive correlation with being right!). Which means your approximation predicts that your comment will get positive karma.

comment by wedrifid · 2010-01-22T00:10:32.541Z · LW(p) · GW(p)

As well as the implied disapproval and disagreement in my downvote I will add that the reception your comment has received does not constitute any sort of meta-level proof of your point. You are being downvoted here for being wrong and being condescending. Posters can gain karma for disagreeing with EY, often even when their reasoning is poor.

comment by Kaj_Sotala · 2010-01-22T17:30:51.293Z · LW(p) · GW(p)

Also - make a comment implying that people agree with EY because of groupthink, get downvoted. ;)

comment by Aurini · 2010-01-21T21:06:22.239Z · LW(p) · GW(p)

This post will get negative karma. Prove me wrong /LW/

In b4 "Bayesian proof or GTFO."