Calibrate words, not just probabilities

post by MikkW (mikkel-wilson) · 2020-07-18T05:56:11.120Z · LW · GW · 3 comments

Contents

3 comments

In Phil Tetlock's book Superforecasters, much emphasis is put on making sure that a forecaster's predictions are well calibrated- if a forecaster gives a 70% chance to 100 different events, we should expect that 70 of those events will have happened, and 30 didn't happen. If only 50 events actually happened when the forecaster said 70%, then the forecaster may want to improve their calibration (especially if they are the betting sort), and anybody listening to them would be well advised to take their poor calibration into account when hearing a prediction.

Tetlock laments that television pundits almost never give probabilities for their predictions, so they basically can't be calibrated, furthermore, the events they describe are so vague that it can't even be readily agreed whether or not their predictions were correct even in retrospect. All of these things give a headache to someone who actually wants to hold pundits accountable, and to have a meaningful conversation with people who are influenced by such pundits.

Now consider the phrase "I predict event X will happen by November 31st with 80% probability"- the function of such an utterance is that upon hearing these sounds, an idea will be formed inside my mind, that if we saved the state of the simulator we live in, and ran the simulator from that point 100 times, I should expect to see event X happen in 80 of these histories by November 31st". When a pundit utters the phrase "Zombies will certainly roam the Earth if we implement policy Y", a similar idea is formed in my mind. But where the first statement allowed me to form an idea with a clear timeframe, and a precise level of certainty, listening to the pundit, I have to infer these for myself.

To help me get the best possible understanding from the pundit's words, I can calibrate his words just the same as I can calibrate probabilities. After all, if Freddy Forecaster says "70% probability" for events that happen only 60% of the time, I know to correct, in mind, Freddy's forecast- when he says 70%, I know to anticipate that it will actually happen only 60% of the time, and would bet accordingly. So if Peter Pundit says something "certainly" will happen 100 times, and we see 55 of these events actually happen, the next time he says something "certainly" will happen, I would be willing to bet based on his words suggesting a 55% probability. I can likewise calibrate when he says he's "extremely confident", or that "X will happen" (without any indicators), or "it's probable", and understand that such words are correlated with certain underlying probabilities.

Likewise, even if a pundit fails to give a meaningful timeline for his predictions, we can still calibrate timelines based on previous predictions. If he said previously that "This will cause the price of X to go up", then 21 months later, the price does indeed go up, we don't have to concern ourselves with whether the price going up has to do with the original event the predictor linked it to. We can simply observe that typically, if Peter Pundit says something will happen, it will happen 1.25 times sooner than the base rate would suggest, and use this to translate a vague, non-time-bound prediction into something attached to a verifiable timeframe, which can then help our calibration for other types of vague wording.

Throughout this post, I've been talking about forecasters and pundits, but these principles can be used to analyze everyday discourse and conversation as well. Oftentimes we hear people say things, without actually giving much thought to what falsifiable model of the world we should actually infer from and be willing to trust based upon their words. It could be valuable to calibrate the statements of different people in a given social environment, and use that calibration to inform our communication, decisions, and thinking based on other people's proclamations.

3 comments

Comments sorted by top scores.