Prediction Calibration - Doing It Right

What's wrong with traditional error bars or some equivalent thereof?

If you are forecasting boolean events, you can either calculate the expected dispersion analytically (if you're lazy) or just simulate it (if you're lazy in a different way). Plot it to taste (error bars, boxplots, watercolour plots, etc.) and then overlay actual results.

There is no reason to restrict ourselves to a simple set of lines or bars, is there?

I'm not a fan of the traditional method -- I am particularly unenthusiastic about the way it depends on allowing only a limited number of specific probability estimates -- but I could do with a little more information and/or persuasion before being convinced that this proposal is Doing It Right.

If I have one of your graphs, how do I (1) quantify (if I want to) how well/badly I'm doing and (2) figure out what I need to change by how much?

Consider the graph you took from Slate Star Codex (incidentally, you have a typo -- it says "Start" in your post at present). If I'm Scott looking at that graph, I infer that maybe I should trust myself a little more when I feel 70% confident of something, and that maybe I'm not distinguishing clearly between 70% and 80%; and that when I feel like I have just barely enough evidence for something to mention it as a 50% "prediction", I probably actually have a little bit more. And, overall, I see that across the board I'm getting my probabilities reasonably close, and should probably feel fairly good about that.

(Note just in case there should be the slightest doubt: I am not in fact Scott.)

On the other hand, if I'm Scott looking at this

which is, if I've done it right, the result of applying your approach to his calibration data ... well, I'm not sure what to make of it. By eye and without thinking much, it looks as if it gets steadily worse at higher probabilities (which I really don't think is a good description of the facts); since it's a cumulative plot, perhaps I should be looking at changes in the gap sizes, in which case it correctly suggests that 0.5 is bad, 0.6 isn't, 0.7 is, 0.8 isn't ... but it gives the impression that what happens at 0.9-0.99 is much worse than what happens at lower probabilities, and I really don't buy that. And (to me) it doesn't give much indication of how good or bad things are overall.

Do you have some advice on how to read one of your graphs? What should Scott make of the graph shown above? Do you think the available evidence really does indicate a serious problem around 0.9-0.99?

I also wonder if there's mileage in trying to include some sort of error bars, though I'm not sure how principled a way there is to do that. For instance, we might say "well, for all we know the next question of each type might have gone either way" and plot corresponding curves with 1 added to all the counts:

The way the ranges overlap at the right-hand side seems to me to back up what I was saying above about the data not really indicating a serious problem for probabilities near 1.

If this is a valid method of scoring predictions, some mathematician or statistician should have already published it. We should try to find this publication rather than inventing methods from scratch.

A good calibration measure might be the ratio between your score (using the logarithmic scoring rule, say) and the score of your predictions transformed by whichever monotonic function makes your score greatest (I suppose this function should also be symmetric about 50%). To put that in words: "What proportion of your score cannot possibly be explained by miscalibration rather than ignorance?"

For example Scott's predictions were

Of 50% predictions, I got 8 right and 5 wrong, for a score of 62%

Of 60% predictions, I got 12 right and 9 wrong, for a score of 57%

Of 70% predictions, I got 13 right and 3 wrong, for a score of 81%

Of 80% predictions, I got 13 right and 3 wrong, for a score of 81%

Of 90% predictions, I got 16 right and 1 wrong, for a score of 94%

For 95% predictions, I got 9 right and 0 wrong, for a score of 100%

For 99% predictions, I got 3 right and 0 wrong, for a score of 100%

So he wishes that instead of 60% he had said 57%, instead of 70% and 80% he had said 81%, instead of 90% he had said 94% and instead of 95% and 99% he had said 100%. This would have improved his log score from -0.462 per question to -0.448 per question. So his calibration is -0.448/-0.462 = 97.2%, which is presumably very good.

EDIT: I'd say now that it might be better to take the difference rather than the ratio. Otherwise you'll look better calibrated on difficult problems just because your score will be worse overall.

This is a distinct problem from my (1) since every field has things that are nearly 100% certain to happen.

I think that's because your #1 is actually two distinct problems :-). You mentioned two classes of prediction whose result would be uninformative about calibration elsewhere: easy near-certain predictions, and straightforward unambiguous probability questions. (Though ... I bet the d10 results would be a little bit informative, in cases where e.g. the die has rolled a few 1s recently. But only via illuminating basic statistical cluefulness.) I agree that there are some of the former in every field, but in practice people interested in their calibration don't tend to bother much with them. (They may make some 99% predictions, but those quite often turn out to be more like 80% predictions in reality.)

this approach gets the wrong sign [...] would count as two failed predictions when it is effectively just one.

It gets the right sign when the question is "how do we use these calibration numbers to predict future reliability?" (if someone demonstrates ability in guessing the next president, that confers some advantages in guessing the next supreme court judge) and the wrong sign when it's "how do we use these prediction results to estimate calibration?". I agree that it would be useful to have some way to identify when multiple predictions are genuinely uninformative because they're probing the exact same underlying event(s) rather than merely similar ones. In practice I suspect the best we can do is try to notice and adjust ad hoc, which of course brings problems of its own.

(There was a study a little while back that looked at a bunch of pundits and assessed their accuracy, and lo! it turned out that the lefties were more reliable than the righties. That conclusion suited my own politics just fine but I'm pretty sure it was at least half bullshit because a lot of the predictions were about election results and the like, the time of the study was a time when the left, or what passes for the left in the US context, was doing rather well, and pundits tend to be overoptimistic about political people on their side. If we did a similar study right now I bet it would lean the other way. Perhaps averaging the two might actually tell us something useful.)

I think #1 is better understood as follows: you can be differently calibrated (at a given confidence level) for different kinds of question. Most of us are very well calibrated for questions about rolling dice. We may be markedly better or worse -- and differently -- calibrated for questions about future scientific progress, about economic consequences of political policies, about other people's aesthetic preferences, etc.

That suggests a slightly different remedy for #1 from yours: group predictions according to subject matter. (More generally, if you're interested in knowing how well someone calibrated for predictions of a certain kind, we could weight all their predictions according to how closely related we think they are to that. Grouping corresponds to making all the weights 0 or 1.) This will also help with #2: if someone was 99% confident that Clinton would win the election then that mistake will weigh heavily in evaluating their calibration for political and polling questions. [EDITED to add: And much less heavily in evaluating their calibration for dice-rolling or progress in theoretical physics.]

Talk of the Clinton/Trump election brings up another issue not covered by typical calibration measures. Someone who was 99% confident Clinton would win clearly made a big mistake. But so, probably, did someone who was 99% confident that Trump would win. (Maybe not; perhaps there was some reason why actually he was almost certain to win. For instance, if the conspiracy theories about Russian hacking are right and someone was 99% confident because they had inside knowledge of it. If something like that turns out to be the case then we should imagine this replaced with a better example.)

Sometimes, but not always, when an outcome becomes known we also get a good idea what the "real" probability was, in some sense. For instance, the Clinton/Trump election was extremely close; unless there was some sort of foul play that probably indicates that something around 50% would have been a good prediction. When assessing calibration, we should probably treat it like (in this case) approximately 50% of a right answer and 50% of a wrong answer.

(Because it's always possible that something happened for a poorly-understood reason -- Russian hacking, divine intervention, whatever -- perhaps we should always fudge these estimates by pushing them towards the actual outcome. So, e.g., even if the detailed election results suggest that 50% was a good estimate for Clinton/Trump, maybe we should use 0.7/0.3 or something instead of 0.5/0.5.)