1001 PredictionBook Nights

Another problem:

After fall of Tripoli and 90% of Libya falling under control of NTC, intrade contract that "Muammar al-Gaddafi to no longer be leader of Libya before midnight ET 31 Aug 2011" was getting lower each day intrade did not close it as done, since it became the contract that "intrade will not fuck up judging".

There was zero disagreement about facts and predictions of facts on the ground, everybody agreed that Gaddafi's government fell, so it turned into very heavy trading about intrade's sanity, and people estimated chance of sane judgment as ~67% event.

Intrade judged sanely that time, but with so much distrust in judges, prediction markets on anything more clear-cut than election results simply cannot work.

In one of previous threads about it, SilasBarta had an example where intrade fucked up, so I doubt this is an off chance.

This detachment itself seems to help accuracy; I was struck by a psychology study demonstrating that not only are people better at falsifying theories put forth by other people, they are better at falsifying when pretending it is held by an imaginary friend!

I think we've just derived a new heuristic. Pretend that your beliefs are held by your imaginary friend.

As we evaluate predictions for accuracy, one thing we should all be hyper-aware of is that predictions can affect behavior. It is always at least a little bit suspect to evaluate a prediction simply for accuracy, when its goal might very well have been more than accuracy.

If I bet you $100 on even odds that I'm going to lose 10 lbs this month, it doesn't necessarily indicate that I think the probability of it happening is > 50%. Perhaps this bet increases the probability of it happening from 10% to 40%, and maybe that 30% increase in probability is worth the expected $20 cost. More generally, even with no money on the table, such a bet might motivate me, and so it cannot be inferred from the fact that I made the bet that I actually believe the numbers in the bet.

Or let's talk about the planning fallacy. Instrumentally, it might make sense to hold the belief that you'll get your project done before the deadline, if that sort of thinking motivates you to finish it earlier than you would if you were completely resigned to finishing it late from the get-go. It might even be detrimental to your project-finishing goals to become enlightened about the reality.

And of course, predictions can affect other people than the predicter. It is ludicrous to look at the public predictions of the Fed chairman and call him a fool when he turns out to be wrong.

Sometimes a prediction is a prediction. But this is definitely something to keep in mind. And gwern, given that you have all this data now, you might find it interesting to see if there are any systematic differences between predictions that do and don't have any significant effect on behavior.

Okay, I'm sold. See you on PredictionBook...

I am considerably more skeptical of op-eds and other punditry, after tracking the rare clear predictions they made

In the case of a few well-studied pundits you should examine the evidence gathered by other prediction trackers. Some pundits are well outside the dumb luck range on a ten-point scale:

The best? Paul Krugman with a PVS of 8.2 (You can see a screenshot of his score sheet to the right. Note: Score sheets for each of the pundits are in the full text document).

The worst? Cal Thomas, with a PVS of -8.7 (You read that right. Negative eight point seven...).

Kinda surprising to me that you can beat dumb luck in inaccuracy. I hope they do a followup.

This post inspired me to actually create a prediction book account. So far I only have one prediction but its a start.

I was thinking about what's a good way to measure how well-calibrated you are. The most obvious way is to say your well calibrated if, e.g., 70% of your predictions at 70% confidence level are correct; however, that implicitly assumes your predictions are independent. You can try getting around this by making lots of predictions in different areas; however, this leaves open the possibility that you might be differently calibrated in different areas.

gwern, what type of predictions did you find most helpful? (E.g. political ones, personal ones, ones you were completely unsure about, ones you had to research, etc...?)

There is a lot of good stuff to think about in this article. I especially like the poker post on "this is what 5% feels like" as it is in line with a number of things which I have enjoyed doing to calibrate my own intuition. Some of those items posted on prediction book confuse me, however.

Example: "‘I predict…a major earthquake, magnitude seven or more will happen within twenty kilometers of a line from nagano city, nagano to matsudo, chiba’ posted by gwern." And then you predict 100% against. (Ignoring the conventional wisdom that 1.0 is not a probability) I wonder what you are doing with such an item which seems a waste of keystrokes and brain thought cycles. Can you go into (a little more) what you are doing there?

A related issue I seem to be having on Prediction Book. Judging from my past predictions I'm generally underconfident by quite a bit. According to the outcome graph no matter what percentage I estimate for something to happen, as long as I am giving it more than 50%, it seems to happen around 88% of the time. Slightly more for the >90% section, and similarly for the rounding to 100%, so my probability are at least weakly correlated in the right direction. Any thoughts on what I can do to better calibrate myself?

Footnotes beyond 10 have gone missing.

Very interesting stuff - thank you!

The link under fee changes does not work.

Otherwise this seems like an interesting summary. I've been thinking of making a similar post with my experience on PredictionBook, but you've been a lot more systematic about your predictions.

1514 public predictions? Gwern, you don't just have more courage than I have. You have orders of magnitude more courage than I have.

Footnotes like that are a horrible idea from usability point of view. This is a wrong medium. Don't we have some kind of spoiler tags?