Thoughts on the SPIES Forecasting Method?

post by Fer32dwt34r3dfsz (rodeo_flagellum) · 2022-03-19T15:22:43.277Z · LW · GW · 3 comments

This is a question post.

Contents

  Answers
    4 SimonM
None
3 comments

Here, I discuss the SPIES forecasting method, and ask for the community's thoughts on it. 

Not too long ago, I came across the SPIES (Subjective Probability Interval Estimates) method for judgmental forecasting. The method was developed by Uriel Haran, and seems first to have been published as part of his 2011 dissertation Subjective Probability Interval Estimates: A Simple and Effective Way to Reduce Overprecision in Judgment. Haran writes

Overprecision in judgment is the most robust type of overconfidence, and the one least
susceptible to debiasing. It refers to people’s excessive certainty in the accuracy of their
estimates, predictions or beliefs. Research on overprecision finds that confidence intervals, estimated ranges that judges are confident will include the correct answer, tend to include the correct answer significantly less often than what their assigned confidence level would suggest. For example, 90% confidence intervals typically include the correct answer about 50% of the time (Klayman, Soll, González-Vallejo, & Barlas, 1999).

He eventually makes the claim that SPIES reduces the overprecision of confidence interval forecasts, and evidences this claim with the results of several forecasting experiments he conducted. Participants made forecasts using the following methods: provide a confidence interval that contains the target value 90% of the time; provide a 5% lower bound and 95% upper bound that they believe the target value would not be below and above, respectively; use the SPIES method, which consists of decomposing a numerical range into several intervals, then having participants assign a likelihood of 0-100 for each interval, then normalizing these likelihoods into a probability uniformly distributed over the values in the interval, and then finding the shortest subinterval of the numerical range that constitutes 90% of the cumulative probability. 

For example, if we want to forecast the monthly rainfall for NYC in March, we can begin by looking at the following intervals 40-65mm, 66-90mm, 91-115mm, 116-140mm, and >140mm (40mm to 140+mm could have been partitioned into 5, 10, etc... intervals, I just chose 5 for this example). I do not know much about rainfall in NYC, but might assign these intervals the following likelihoods: 35/100, 55/100, 85/100, 25/100, and 5/100, respectively. My probabilities for these intervals would then be

• 35 / (35+55+85+25+5) = 0.1707 for 40-65mm
• 55 / (35+55+85+25+5) = 0.2683 for 66-90mm
• 85 / (35+55+85+25+5) = 0.4146 for 91-115mm
• 25 / (35+55+85+25+5) = 0.122 for 116-140mm
• 5 / (35+55+85+25+5) = 0.0244 for >140mm

The smallest subinterval of [40, >140] subsuming 90% of this probability produces the following estimate: With 90% confidence, I believe NYC's rainfall in March will be between 40mm and 125mm. Note that, to get this estimate, I had to use programming. Also, the 90% confidence interval was somewhat arbitrary; I'm also 75% confident that NYC's rainfall in March will be between 55mm and 115mm. 

I haven't come across SPIES anywhere on LW, and first found out about it in this Harvard Business Review article A Simple Tool for Making Better Forecasts, which contains an interactive example (temperatures in June) of SPIES.  

So, what do you think? Does this method seem at all promising? I'm debating with myself whether I should begin using SPIES on Metaculus or elsewhere. Would anyone be interested in performing some experiments with me on using SPIES in a greater variety of forecasting situations, or perhaps in improving SPIES or in building better methods to control for overconfident forecasts?

Answers

3 comments

Comments sorted by top scores.