Forecasting for Policy (FORPOL) - Main takeaways, practical learnings & report

post by janklenha (jan-klenha) · 2023-09-18T17:44:16.305Z · LW · GW · None comments

Contents

  General takeaways
  Practical lessons learned - Policy
  Practical lessons learned - Forecasting tournaments
  Impacts
None
No comments

Tl;dr: In Czech Priorities, a think tank based in Prague, together with Metaculus, we recently ran a complex forecasting tournament funded by EAIF to support Czech policymaking. Today, we are publishing a write-up of our findings from the tournament, as well as supplementary materials to help any groups of forecasters looking to do the same.

The article was also cross-posted on EA Forum [EA · GW].
______

From October 2022 to March 2023, we ran a forecasting tournament with a total of 54 questions. In March, we have discussed some of our preliminary findings in this post [EA · GW].

Almost all of our forecasting questions were developed in cooperation with 16 different public institutions and ministerial departments. Each institution or department defined its most useful forecasting topics, participated in a workshop to define specific questions with us, and was later provided with the results. This was intended as a proof of concept of one possible approach to incorporating forecasting in public decision-making.

Once defined, our forecasting questions were then posted on a private Metaculus sub-domain (in Czech language), where an average of 72 forecasters had the opportunity to address them as they would any other question on Metaculus (median of 18 predictions per user). Throughout the tournament, we produced 16 reports detailing the rationales and forecasts, to be used by the cooperating institutions. The institutions and their topics were listed in our previous post [EA · GW].

This approach, in combination with multiple media appearances, also allowed us to strengthen our position as one of the leading Czech institutions with expertise in foresight methods.

We've created a write-up detailing our steps in both talking to institutions and managing the tournament with its specifics. Here are our five overarching takeaways for groups of forecasters who want to make an impact through policymaking, and some specific lessons learned:

General takeaways

  1. Develop partnerships with policymakers across various policy areas. Finding policymakers eager to try analytical innovations is usually not the main bottleneck.
  2. Diversify your portfolio of foresight methods. Particularly in the EU, policymakers are becoming increasingly aware of the importance of foresight. Their familiarity with methods such as scenario planning or horizon scanning can grant more inroads into the policy process for forecasting if leveraged well.
  3. Expect to deal with complex, long-term forecasting questions. They are requested often and can have larger impacts if used to adjust long-term plans and strategies.
  4. Even among promising leads, only approx. 25% will see the process to the end. Forecasting is not yet a priority for many policymakers and there are many steps where it may fail on the way to delivering impact.
  5. Don't expect clearly measurable large impacts. Large impacts are usually difficult to trace back to individual data points (as they require more data and negotiation), while measurable uses are usually based on individual decisions, producing smaller impacts.

Practical lessons learned - Policy

Practical lessons learned - Forecasting tournaments

Impacts

In our case, a handful of our partners already acted on the information/judgment presented in our reports. This has concerned, for example, the national foreclosure issue (some 6% of the total population have debts in arrears) where the debt relief process is being redesigned midst of strong lobbying and insufficient personal capacities; or the probabilities of outlier scenarios for European macroeconomic development, which was requested by the Slovakian Ministry of Finance to help calibrate their existing judgments.

Other partners claim to have incorporated this knowledge into the larger cycle of their policymaking process, but we haven't yet seen any actual evidence of it. Our experience and candid discussions with policymakers and forecasters alike, however, also gave us some pointers on what pitfalls are intensified when tournaments are focused on policy questions (such as the incentives and motivations of forecasters).

In general, it seems useful to explore various approaches to grow the number of policymakers with personal experience & skills in forecasting. In our case, we found curiosity and willingness to try forecasting even in unexpected institutional locations (i.e. the Czech R&I funding body). This makes us more confident that the “external forecasts” approach (as compared to building internal prediction tournaments or focusing on advancing the forecasting skills of public servants) is worth investigating further precisely because it allows us to detect and draw on this interest irrespective of institutional and seniority distinctions and resource constraints. 

While we hope that any readers with an interest in forecasting may find our experience useful, we expect that both this and any future projects of ours make it easier for other teams to work towards similar goals. To that effect, the write-up also contains an Annex of “Methodological Guidelines,” where we outline in more explicit terms the questions and decisions that we found were important to tackle when running the project, and what they may entail.

Access our full report HERE.

None comments

Comments sorted by top scores.