Forecasting Newsletter: May 2020.

post by NunoSempere (Radamantis) · 2020-05-31T12:35:58.063Z · LW · GW · 1 comments

Contents

  Index
  Prediction Markets & Forecasting platforms.
    Augur: augur.net
    Coronavirus Information Markets: coronainformationmarkets.com
    CSET: Foretell
    Epidemic Forecasting: epidemicforecasting.org (c.o.i)
    Foretold: foretold.io (c.o.i)
    /(Good Judgement?[^]*)|(Superforecast(ing|er))/gi
    Metaculus: metaculus.com
    Predict It & Election Betting Odds: predictIt.org & electionBettingOdds.com
    Replication Markets: replicationmarkets.com
  In The News.
  Grab Bag
  Negative examples.
  Long content
None
1 comment

A forecasting digest with a focus on experimental forecasting. The newsletter itself is experimental, but there will be at least four more iterations. Feel free to use this post as a forecasting open thread; feedback is welcome.

Index

Prediction Markets & Forecasting platforms.

Augur: augur.net

Augur is a decentralized prediction market. Here is a fine piece of reporting outlining how it operates and the road ahead.

Coronavirus Information Markets: coronainformationmarkets.com

For those who want to put their money where their mouth is, this is a prediction market for coronavirus related information.

Making forecasts is tricky, so would-be-bettors might be better off pooling their forecasts together with a technical friend. As of the end of this month, the total trading volume of active markets sits at $26k+ (upwards from $8k last month), and some questions have been resolved already.

Further, according to their FAQ, participation from the US is illegal: "Due to the US position on information markets, US citizens and residents, wherever located, and anyone physically present in the USA may not participate in accordance with our Terms." Nonetheless, one might take the position that the US legal framework on information markets is so dumb as to be illegitimate.

CSET: Foretell

The Center for Security and Emerging Technology is looking for (unpaid, volunteer) forecasters to predict the future to better inform policy decisions. The idea would be that as emerging technologies pose diverse challenges, forecasters and forecasting methodologies with a good track record might be a valuable source of insight and advice to policymakers.

One can sign-up on their webpage. CSET was previously funded by the Open Philanthropy Project; the grant writeup contains some more information.

Epidemic Forecasting: epidemicforecasting.org (c.o.i)

As part of their efforts, the Epidemic Forecasting group had a judgemental forecasting team that worked on a variety of projects; it was made up of forecasters who have done well on various platforms, including a few who were official Superforecasters.

They provided analysis and forecasts to countries and regions that needed it, and advised a vaccine company on where to locate trials with as many as 100,000 participants. I worked a fair bit on this; hopefully more will be written publicly later on about these processes.

They've also been working on a mitigation calculator, and on a dataset of COVID-19 containment and mitigation measures.

Now they’re looking for a project manager to take over: see here [LW · GW] for the pitch and for some more information.

Foretold: foretold.io (c.o.i)

I personally added a distribution drawer to the Highly Speculative Estimates utility, for use within the Epidemic Forecasting forecasting efforts; the tool can be used to draw distributions and send them off to be used in Foretold. Much of the code for this was taken from Evan Ward’s open-sourced probability.dev tool.

/(Good Judgement?[^]*)|(Superforecast(ing|er))/gi

(The title of this section is a regular expression, so as to accept only one meaning, be maximally unambiguous, yet deal with the complicated corporate structure of Good Judgement.)

Good Judgement Inc. is the organization which grew out of Tetlock's research on forecasting, and out of the Good Judgement Project, which won the IARPA ACE forecasting competition, and resulted in the research covered in the Superforecasting book.

Good Judgement Inc. also organizes the Good Judgement Open gjopen.com, a forecasting platform open to all, with a focus on serious geopolitical questions. They structure their questions in challenges. Of the currently active questions, here is a selection of those I found interesting (probabilities below):

Probabilities: 25%, 75%, 40%, 62%, 20%

On the Good Judgement Inc. side, here is a dashboard presenting forecasts related to covid. The ones I found most worthy are:

Otherwise, for a recent interview with Tetlock, see this podcast, by Tyler Cowen.

Metaculus: metaculus.com

Metaculus is a forecasting platform with an active community and lots of interesting questions. In their May pandemic newsletter, they emphasized having "all the benefits of a betting market but without the actual betting", which I found pretty funny.

This month they've organized a flurry of activities, most notably:

Predict It & Election Betting Odds: predictIt.org & electionBettingOdds.com

PredictIt is a prediction platform restricted to US citizens, but also accessible with a VPN. This month, they present a map about the electoral college result in the USA. States are colored according to the market prices:

Some of the predictions I found most interesting follow. The market probabilities can be found below; the engaged reader might want to write down their own probabilities and then compare.

Some of the most questionable markets are:

Market probabilities are: 76%, 9%, 75%, 82%, 8%, 2%, 6%, 11%.

Election Betting Odds aggregates PredictIt with other such services for the US presidential elections, and also shows an election map. The creators of the webpage used its visibility to promote ftx.com, another platform in the area, whose webpage links to effective altruism and mentions:

FTX was founded with the goal of donating to the world’s most effective charities. FTX, its affiliates, and its employees have donated over $10m to help save lives, prevent suffering, and ensure a brighter future.

Replication Markets: replicationmarkets.com

On Replication Markets, volunteer forecasters try to predict whether a given study's results will be replicated with high power. Rewards are monetary, but only given out to the top few forecasters, and markets suffer from sometimes being dull.

The first week of each round is a survey round, which has some aspects of a Keynesian beauty contest, because it's the results of the second round, not the ground truth, what is being forecasted. This second round then tries to predict what would happen if the studies were in fact subject to a replication, which a select number of studies then undergo.

There is a part of me which dislikes this setup: here was I, during the first round, forecasting to the best of my ability, when I realize that in some cases, I'm going to improve the aggregate and be punished for this, particularly when I have information which I expect other market participants to not have.

At first I thought that, cunningly, the results of the first round would be used as priors for the second round, but a programming mistake by the organizers revealed that they use a simple algorithm: claims with p < .001 start with a prior of 80%, p < .01 starts at 40%, and p < .05 starts at 30%.

In The News.

Articles and announcements in more or less traditional news media.

Grab Bag

Podcasts, blogposts, papers, tweets and other recent nontraditional media.

Negative examples.

I have found negative examples to be useful as a mirror with which to reflect on my own mistakes; highlighting them may also be useful for shaping social norms. Andrew Gelman continues to fast-pacedly produce blogposts on this topic. Meanwhile, amongst mortals:

Long content

This section contains items which have recently come to my attention, but which I think might still be relevant not just this month, but throughout the years. Content in this section may not have been published in the last month.

Vale.

Conflicts of interest: Marked as (c.o.i) throughout the text.
Note to the future: All links are automatically added to the Internet Archive. In case of link rot, go there.

1 comments

Comments sorted by top scores.

comment by Charles R. Twardy (charles-r-twardy) · 2020-07-31T21:51:57.189Z · LW(p) · GW(p)

Thank you for including Replication Markets! A couple of notes:

  • Yes, the survey round is potentially a Keynesian beauty contest, though it takes some doing. You're not forecasting the market round. You're forecasting the best estimate we can make using peer prediction on the independent surveys. Harvard's peer prediction algorithm has done well in previous tests, and in theory takes a lot of coordination to defeat.* The truth is probably a good bet. We'll know more in December.
  • We had originally planned to use the output of surveys to set the market starting prices - but it made for a very messy experimental design. So we used a 70% accurate decision tree. It wasn't intended to be secret - I thought we had stated it fully but searching the site I find only oblique references.

*We got to test that a bit in Round 8 when we discovered a coordinated "attack" that accounted for ~1/3 of our surveys. Some forecasts would have changed, prizes would have been won, but neither so much as we feared.