Introducing Pastcasting: A tool for forecasting practice

post by Sage Future (aaron-ho-1) · 2022-08-11T17:38:06.474Z · LW · GW · 10 comments

Contents

  Motivation
  Pastcasting
  How it works
    Question sources
    Scoring
    Vantage Search
    Prior Knowledge
  Give it a try!
None
10 comments

TL;DR: Visit pastcasting.com to forecast on already resolved questions you don’t have prior knowledge about to get quick feedback on how you’re doing.

We may try to coordinate multiplayer sessions in our Discord

Motivation

We want to make it easy for people to get better at forecasting. Existing tools are not well optimized for practicing relevant skills.

• Forecasting platforms and betting markets^[1] have slow feedback loops between predictions and question resolution. The questions with shorter time horizons often feel less important and may be systematically different than those with longer ones. Their scoring systems also incentivize constantly keeping predictions up to date and often heavily reward being the first to react to news
• Calibration training^[2] isolates the skill of intuiting probabilities and confidence intervals but doesn’t help with other aspects of forecasting (choosing reference classes & base rates, trend extrapolation, coming up with considerations, investigating different views, and determining the trustworthiness of news sources). Additionally, there is usually no reference point to compare your accuracy against.

Pastcasting

With pastcasting, you can:

• Forecast on already resolved questions from a vantage point further in the past
• Use our filtered search engine (“Vantage Search”) to look up relevant information without accidentally revealing the answer
• Receive immediate feedback on your forecasts and get scored against the crowd
• Host friendly multiplayer competitions where you and your friends can simultaneously pastcast on the same questions and see who does best

How it works

Question sources

Our questions are currently pulled from resolved Metaculus and GJOpen questions.

Scoring

We use relative log scoring against the original crowd forecast at that time. This means that you will receive zero points if you submit the same value as the crowd. The scoring rule is also strictly proper, meaning that your expected score is maximized if you report your true beliefs.

Vantage Search

Our preliminary results (in yellow) come from a search api with a restricted date range up to the vantage date. To further reduce information leakage (from the website changing its contents), we then pass the results through the internet archive api (in blue). This tends to be much slower, so some pastcasters opt to use the preliminary results directly.  

Prior Knowledge

In many cases, users will have specific knowledge of how a particular question was resolved (from reading about it in the news, participating in forecasting that question, etc). We provide a button to skip these questions, and over time we will show questions that many users know the answer to less often.

Give it a try!

If this sounds useful, give it a try and give us feedback on what features would be the most useful and any aspects of the site you find confusing.
 

1. ^{^}

   Such as Metaculus, Manifold Markets, etc…

2. ^{^}

   Such as https://www.openphilanthropy.org/calibration, https://acritch.com/credence-game/, Wits & Wagers.

10 comments

Comments sorted by top scores.