A Proper Scoring Rule for Confidence Intervals

EDIT: I originally said you can do this for multiple choice questions, which is wrong. It only works for questions with two answers.

(In a comment, to keep top level post short.)

One cute way to do calibration for probabilities, is to construst a spinner. If you have a true/false question, you can construct a spinner which is divided up according to your probability that each answer is the correct answer.

If you were to then spin the spinner once, and win if it comes up on the correct answer, this would not incentize constructing the spinner to represent your true beliefs. The best strategy is to put all the mass on the most likely answer.

However, if you spin the spinner twice, and win if either spin lands on the correct answer, you are actually incentivized to make the spinner match your true probabilities!

One reason this game is nice is that it does not require having a correctly specified utility function that you are trying to maximize in expectation. There are only two states, win and lose, and as long as winning is prefered to losing, you should construct your spinner with your true probabilities.

Unfortunately this doesnt work for the confidence intervals, since they seem to require a score that is not bounded below.

This is an underappreciated fact! I like how simple the rule is when framed in terms of size and distance.

You mention both the linear and log rules. The log rule has the benefit of being scale-invariant, so your score isn't affect by the units the answer is measured in, but it can't deal with negatives and gets overly sensitive around zero. The linear rule doesn't blow up around zero, is shift-invariant, and can handle negative values fine. The best generic scoring rule would have all these properties.

Turns out (based on Lambert and Shoham, "Eliciting truthful answers to multiple choice questions") that all scoring rules for symmetric confidence intervals $(a,b)$ with coverage probability $1-\alpha$ can be represented (up to affine transformation) as

where $x$ is the true value, $I$ is the indicator function, and $g(\cdot )$ is any increasing function. Unsurprisingly, the linear rule uses $g\left(x\right)=x$ and the log rule uses $g\left(x\right)=log\left(x\right)$. If we want scale-invariance on the whole real line, first thing I'd be tempted to do is use $log\left(x\right)$ for positive $x$ and $-\log \left(|x|\right)$ for negative $x$ except for that pesky bit about going off to $\pm \infty$ around zero. Let's paste in a linear portion around zero so the function is increasing everywhere: $g\left(x\right)=I(|x|\le 10)\cdot \left(x/10\right)+I(|x|>10)\cdot \mathrm{sign}\left(x\right)\cdot {\log }_{10}\left(|x|\right)$

Using this $g(\cdot )$, the score is sensitive to absolute values around zero and sensitive to relative values on both sides of it. Since the rule expects more accuracy around zero, the origin should vary depending on question domain. Like if the question is about dates, accuracy should be the highest around the present year and get less accurate going into the past or future. That suggests we should set the origin at the present year. For temperatures, the origin should probably be room temperature. Are there any other standard domains that should have a non-zero origin? An alternate origin $t$ can be added as a shift everywhere:

Not something you'd want to calculate by hand, but if someone implements a calibration app, this has more consistent scores. Going one step further, the scores could be made more intepretable by comparison to a perfectly calibrated reference score: $100+k\cdot \left(S_{\alpha }\right(a,b,x)-S_{\alpha }^{\ast })$ where $S_{\alpha }^{\ast }$ is the expected score for perfectly calibrated intervals if, say, $x\sim N(0,10)$ and $k$ is a fixed value chosen to keep plausible scores mostly positive.

This scoring rules has some downsides from a usability standpoint. See Greenberg 2018, a whitepaper prepared as background material for a (forthcoming) calibration training app.

I need help figuring out how to use this scoring rule. Please consider the following application.

How much does it cost to mail a letter under 30g in Canada?${}^1$

I remember when I was a child buying 45c stamps, so it's likely to be larger than that. It's been over a decade or so, and assuming a 2% rise in cost per year, then we should be around $45\ast (1.02{)}^{10}\sim 60$c per stamp. However, we also had big budget cuts to our postal service that even I learned about despite not reading the news. Let's say that Canada Post increased their prices by 25% to accomodate some shortfall. My estimate is that stamps cost 75c.

What should be my confidence interval? Would I be surprised if a stamp cost a dollar? Not really, but it feels like an upper bound. Would I be surprised if a stamp cost less than 50c? Yes. 60c? Yes. 70c? Hmmm.... Assume that I'm well calibrated, so I'm reporting 90% confidence for an interval of stamps costing 70c to 100c.

Answer: Stamps in booklets cost 85c each, individual stamps are 100c each. Because I would always buy stamps in booklets, I will use the 85c figure.

S is the size of my confidence interval, $S=100-70=30$ . D is the distance between the true value and the interval, but is 0 in this case because the true value is in the interval.

$\text{Score}=-S-20\cdot D=-30$

I'm not really sure what to do with this number, so let's move to the next paragraph of the post.

The true value is $T=85$ and the interval is $(L,U)=(70,100)$. Because the true value is contained in the interval, $D=0$.

$S=log\left(\frac{U}{L}\right)=log\left(\frac{100}{70}\right)=0.15$

$\text{Score}=-S-20\cdot D=-0.15$

How does this incentivise honest reporting of confidence intervals?

Let's say that, when I intuited my confidence interval above that I was perturbed that it wasn't symmetric about my estimate of 75c, so I set it to $(L,U)=(50,100)$ for aesthetic reasons. In this case, my score would be $\text{Score}=-0.30$ Which is worse than my previous score by a factor of 2.

Let's say that, when I remembered the price of stamps in my childhood, I was way off and remembered 14c stamps. Then I would believe that stamps should cost around 22c now. (Here I have the feeling of "nothing costs less than a quarter!", so I would probably reject this estimate.)That would likely anchor me, so that I would set a high confidence on the price being within $(L,U)=(20,24)$

$S=0.08$, $D=log\left(\frac{L}{T}\right)=log\left(\frac{20}{85}\right)=-0.63$

$\text{Score}=-S-20\cdot D=12.52$

Am I trying to maximize this score?

${}^1$I looked up the answer, and the lowest cost standard delivery is for letters under 30g.

This is one of these simple insights that I think are actually important for a large variety of forecasting related tasks, and that turn out to be a large pain to rederive for yourself.

Is there a way to incentivize reporting true probability distribution? Say I Bob wants Alice to provide her probability distribution of IQ she'll get on the test. He is willing to give her a real number as a reward, he wants to hear her probability distribution of her result. What should he do?

Would be nice if it worked for both discrete and non-discrete probability spaces.

Incentivising accurate probabilistic predictions is central to any art of rationality, this post gives a signficant part of this that's super readable, so I've curated it.

(Also, nice move adding extra points in the comments.)

Do you have some argument that your proposed formula are optimal?

Is there a way to adjust this to support better scores for tighter confidence intervals?

For instance, using natural log, with a range of 8-10 and a true value of 10, I get -0.2231 whether I pick a 90% confidence interval, or a 95% confidence interval (coefficient of 40). It'd be nice if the latter scored better.

[Edit: I'm retracting this comment, as I made some incorrect assumptions about Scott's claim.] This is wrong. It is well known that the only strictly proper scoring rule that depends only on the probability at the actually occurring value is the logarithmic scoring rule (if there are more than two alternatives), or translations and/or positive scaling of the same. In this case, that would be log(Normal(x | mu, sigma)), where x is the value that occurs, and mu and sigma^2 are the mean and variance of the normal distribution that fits the interval you defined at the given confidence level. This may be simplified to

-log(sigma^2) - (x - mu)^2 / sigma^2.

Your scoring rule is not a translation and/or positive scaling of the logarithmic scoring rule.