Posts

An examination of Metaculus' resolved AI predictions and their implications for AI timelines 2021-07-20T09:08:43.921Z
Data on forecasting accuracy across different time horizons and levels of forecaster experience 2021-05-27T18:53:07.884Z

Comments

Comment by CharlesD on Range and Forecasting Accuracy · 2021-05-04T14:46:35.127Z · LW · GW

Thanks for a great post! I have a concern about your sample sizes however.

I am looking into similar questions myself, and while reading your post I was surprised to see your Metaculus sample claimed as 45k predictions. These are not actually individual predictions, but rather the time series of community predictions, which are much less information dense, as this is just the median of the recent community predictions at that time and typically a new prediction will have a small effect on this value. I think claiming the sample size is 45k is therefore a bit misleading. 

It also has the effect of linearly weighting Metaculus questions by community interest, which is not obviously a desirable method (this is mitigated by the cap on the time series length to 101, which means the effect will ultimately be small as you have 557 Metaculus questions, implying your average question must have more than 80 predictions)