[Job ad] Lead an ambitious COVID-19 forecasting project [Deadline extended: June 10th]

post by jacobjacob · 2020-05-27T16:38:04.084Z · LW · GW · 1 comments

Contents

  What is the state of this project?
  Why would this project be valuable?
  Where can this project go?
  What's involved? What skills do I need?
  What exactly am I signing up for?
  What should I do next?
None
1 comment

In brief: I’m hiring for a project manager for Epidemic Forecasting, an independent project spun off from the Future of Humanity Institute that provides forecasting information to governments and other decision-makers during the covid pandemic. [Application deadline extended to June 10th]. You need to have good judgement, the ability to work fast and hard leading a team, and references from people I trust. You need to be able to commit to 3 months of full-time paid work, and there’s the potential to scale up afterwards to a full-time organisation if you find traction.

Private message [LW · GW] me to talk more. For the rest of the post, I’ll give more info on the project, what’s involved and why you might want to work on this.

What is the state of this project?

Since the beginning of March, I've led the Epidemic Forecasting project which spun off from the Future of Humanity Institute at Oxford.

So far we've:

• Provided policy analyses for senior health officials of regions with tens of millions inhabitants
• Advised a major vaccine candidate on where to locate trials with as many as 100,000 participants
• Built the interactive tools at EpidemicForecasting.org, which served 10k users/day at its peak
• Worked with an in-house team of expert forecasters to produce >3000 forecasts on questions of improvements to testing capacity and contact tracing, herd immunity, further lockdown, and more
• Built software infrastructure to integrate those forecasts into epidemiological modelling software like GleamViz and run simulations at scale
• Built one of the most thoroughly validated Bayesian models of the effects of countermeasures [LW · GW] we are aware of

Over the last 3 months we've found that this project has real traction and demand, and I think it's time for a focused team to commit to at least another 3-12 months on this, and either wrap up once opportunities go away, or build it into a non-profit/for-profit that serves an important long-term function for mitigating pandemics.

This post is an invitation for a new project manager to step up. I’m going to pursue other projects that I think have more long-term potential for me personally. If I find someone who can do the management work, can deliver output fast, and has the sound judgment needed to take over, then I will help you get started with the team, introduce you to major relationships (governments, vaccine companies, and funders) and help advise/mentor you over the coming month. If I don’t I will start winding down my responsibilities and involvement.

Why would this project be valuable?

Peter Thiel says the most successful projects are based on secrets that nobody else has realised. This seems true to me. Here is my sense of the secrets this project is based around.

• There is a severe shortage of epidemiological modelling ability globally. There is not enough supply to meet the sudden, massive increase in demand, from every country in the world and thousands of large organisations, following covid. For example, we’ve spoken to decision-makers governing tens of millions of people, who use no epidemiological modelling to support their decision-making. At the same time, most of the actual resources are massively skewed towards forecasting developments in high-income countries.
• This shortage can partially be filled by good generalist forecasters, who are available in greater supply than trained epidemiologists. (See McAndrew (2020) for evidence of forecasters outperforming domain experts in predicting covid. Farrow et al. (2017) gives more details on using human judgement for epidemiological forecasting, and Mellers et al. (2015) is a good review of Superforecasting.)
• There are major problems with epidemiological modelling as usually practiced, that are well placed to address by researchers who have thought sensibly about technology and reasoning under uncertainty in general. For example, we’ve spoken to decision-makers who simply extrapolate SIR models to 2021 and use that as a key input into their decisions, without regard for the fact that on such timelines the majority of variance in outcome is driven by how governments will respond, how people will respond, and how technology will change (e.g. with improved contact tracing or testing capacity).
• It is possible to extrapolate from limited data based on good judgement (in the superforecaster-sense), but this is barely utilised in much of policymaking. Many decision-makers are making policy based on confirmed cases numbers (even in low-and-middle-income countries with very limited testing). Even a rough application of basic forecasting from someone with a track-record on Metaculus or Good Judgement would provide a much better baseline. As another example, we’ve encountered senior researchers handing decision-makers complex models, but being very reluctant to set the input parameters due to only having limited data, instead expecting the decision-makers to do that(!) The methodology of using forecaster judgement in the face of scarce data has not yet been implemented across all the relevant fields (even though some groups are trying).

My sense is that, in line with what Scott Alexander says about failures during the covid pandemic, you can substantially improve the decision-making of many of humanity's best institutions by using the basic skill of reasoning under uncertainty and being willing to move fast and act quickly. You do need to be able to quickly make serious quality checks on a forecast (e.g. "What does this number really imply about the next 2 months of growth of the disease in country X?"), but you do not need years of expertise in this area to pick it up, especially when you have people around you doing a lot of the heavy lifting who aren't doing the management (and aren’t responsible for the final decisions).

Where can this project go?

Like many startups, the world should determine the product you make, not you. While I don't know where this will end up, here are some ambitious win conditions.

• Cause the date by which we have a vaccine to move several weeks forward by preventing failed efficacy trials.
• Spinning off our modelling infrastructure into open-source tooling for other researchers.
• Serve as coordination point connecting policy-makers to the right epidemiological modellers and forecasters, and rebalancing the current massive misallocation between high and low-income countries. (I think there are 100+ modellers who could have vastly more impact if they targeted countries more wisely, rather than making yet another marginal contribution to US/Europe modelling, and this project could play a substantial role in making that happen.)
• Giving state-of-the-art epidemic modelling tech to many orders of magnitude more people than currently by maintaining public dashboards and online tools (e.g. look at current epidemicforecasting.org and make it 10x as useful and intuitive). The win here is both informing decision-makers and the public.
• Build long-term relations with decision-makers to offer policy advice for future pandemics.
• ...or some as-of-yet unknown pivot that will be clear to the team in a few months

Thus far there have been many opportunities to pivot. We've moved from building public dashboards, to policy consulting, to vaccine trial portfolio optimisation.

There are a wealth of important problems that need solving, and we have some key skills allowing us to do it. There will be many opportunities for the next project manager to figure out how to navigate this space.

What's involved? What skills do I need?

Here are some of the kinds of tasks I've worked on, many of which you would do too:

• Writing policy reports (making complex estimates simple and clear)
• Giving input on UI design for websites
• Writing questions for Superforecasters to answer
• Running meetings and understanding the needs of clients

More generally, you need to be able to:

• Be the glue that connects different teams
• Learn how to manage modellers running epidemiological models using software like GleamViz
• Actually ship products
• Deal with wholly remote work
• Step up to working 70+ hour weeks in periods when it's called for, and know how to manage your sanity during these times (i.e. know how to take real rest [LW · GW]).

You need to be someone that someone I trust trusts.

What exactly am I signing up for?

Committing to run this project full-time, salaried, for at least 3 months.

The world is struggling to meet the overwhelming demand for covid forecasting and analysis. You're in a position to improve the decisions they make about covid, and save thousands of lives, if you want to put in the work. We've set up a team with some good skills and relationships. The highest variance part has already happened, where we didn't even know if there was demand. There is demand.

If you're someone who can execute and have shown good judgment to me or someone I trust (e.g. Ben Pace, Oliver Habryka) then I'm interested in handing the reins to you and you helping it take off.

What should I do next?

Message me here on LessWrong [LW · GW] (or by other channels if you know me), including the material you think is needed for me to evaluate your fit, by June 10th. Don’t send me more than 2 paragraphs of text, although links to google docs for me to skim are fine. Please include:

• A brief summary of past relevant experience of high-paced work
• Something showing your ability to reason sensibly about covid, or otherwise reason under uncertainty about real-life situations. I can come to trust your judgment here by seeing a history of excellent LessWrong posts.
• References (but only in case they are people I trust, or someone I trust trusts).

If the main reason you don't expect to do this is because you're already quite busy, especially if it's at a non-profit, I might still be quite interested in talking through this opportunity with you for half an hour so you have a clearer vision of what opportunity you're not taking. Your choice might be right but it's often good to properly consider alternative hypotheses for an hour or two.

[Note: due to me being very busy, Ben Pace wrote the majority of this post based on our conversations. I endorse everything said, but might have formulated some things differently had I written it myself.]

1 comments

Comments sorted by top scores.