Seemingly Popular Covid-19 Model is Obvious Nonsense

The model seems not far off estimating peak hospitalization date, at least for states that are currently peaking like CA and NY. The peaks in places that are close to peaking can be pretty accurately estimated just with curve fitting though, I assume that being fit to past data is why the model works OK for this.

It's clearly overly optimistic about the rate of drop-off after the peak in deaths, at least in some cases. Look at Spain and Italy. Right now here's how they look:

Italy: graph shows 610 deaths on April 9. Predicts 335 on April 10, 281 on April 11. Actual is 570 on April 10, 619 on April 11.

Spain: graph shows 683 on April 8, Predicts 372, 304, 262 on next three days. Actual 655, 634, 525.

The model for New York says deaths will be down to 48, 6% of the peak, in 15 days. Italy is 15 days from it's peak of 919 and is only down to 619, 67% of the peak.

The model for the US as a whole is a little less obviously over-optimistic, assuming the peak really was April 10. it's only predicting 40% decline in the next 15 days. California model predicts an even slower decline. It seems to think fast growth in cases in the outbreak phase leads to fast recovery, which has not been borne out thus far in Italy and Spain.

April 17th Stat News story: Influential Covid-19 model uses flawed methods and shouldn’t guide U.S. policies, critics say:

“It’s not a model that most of us in the infectious disease epidemiology field think is well suited” to projecting Covid-19 deaths, epidemiologist Marc Lipsitch of the Harvard T.H. Chan School of Public Health told reporters this week, referring to projections by the Institute for Health Metrics and Evaluation at the University of Washington.

Others experts, including some colleagues of the model-makers, are even harsher. “That the IHME model keeps changing is evidence of its lack of reliability as a predictive tool,” said epidemiologist Ruth Etzioni of the Fred Hutchinson Cancer Center, home to several of the researchers who created the model, and who has served on a search committee for IHME. “That it is being used for policy decisions and its results interpreted wrongly is a travesty unfolding before our eyes.”

“ Deaths lag positive tests by weeks.”

False. Deaths lag new infections by 3-4 weeks.

Positive tests are an extremely misleading stat and definitely do NOT represent actual infection rates except in the few places where testing is widespread (ie a few small countries that have highly prioritized testing like Iceland, Estonia, and Bahrain).

Epistemic Status: Something Is Wrong On The Internet.

If you think this applies, it would seem that "The Internet" is being construed so broadly that it includes the mainstream media, policymaking, and a substantial fraction of people, such that the "Something Is Wrong On The Internet" heuristic points against correction of public disinformation in general.

This is a post that is especially informative, aligned with justice, and likely to save lives, and so it would be a shame if this heuristic were to dissuade you from writing it.

"If I am incorrect, and that is how any of this works I have some very, very large bets I would like to place."

Maybe you can state what bets you'd like to make? Are you predicting that the number of cases or deaths in, say, NYC will look very different from consensus estimates?

You can be pretty sure that whatever forecast is touted by authorities is one designed to increase support+compliance with whatever measures they decided to take this time. Just like the previous was badly overestimating severity with social distancing (and probably without too), I'm willing to believe this one is optimistic about a gradual reopening of physical commerce in select areas.

This post was important to my own thinking because it solidified the concept that there exists the thing Obvious Nonsense, that Very Serious People would be saying such Obvious Nonsense, that the government and mainstream media would take it seriously and plan and talk on such a basis, and that someone like me could usefully point out that this was happening, because when we say Obvious Nonsense oh boy are they putting the Obvious in Nonsense. It's strange to look back and think about how nervous I was then about making this kind of call, even when it was this, well, obvious. Making that first correct call makes a difference.

But in terms of being part of an overall 'best of' or 'most important' collection for a community as a whole, it would only count if you think it had the same effect on you/others, and made it clear how nonsensical all the Very Serious People could be, and that you had to think for yourself. If all it did for others was point out that the Obvious Nonsense was obvious nonsense in this particular case, there's not much point.

Note that the model assumes that those level 4 measures remain in place forever.

What do you predict will be, assuming level 4 restrictions remain in place, the last day in which fewer than 100 people are infected who go on to become symptomatic in King County, and in CA?

Again assuming that there is a continuous level 4 quarantine, when do you predict the first non-weekend day without a C19-attributed death will be in those areas?

The money quote is misleading, because they don't actually have a mechanistic model. They're just fitting a parameterized logistic curve to all the death data in the world. They incorporate some black-box factor that causes more deaths without social distancing, and arbitrarily declare that factor's effect is 66%/33%/0 with 1/2/3+ social distancing measures. The goal isn't to claim that nobody's ever infected in the 0 case, just that the not-social-distancing factor is gone, so our course should follow the empirical progression of countries that do social distancing.

Your rule of models is flawed. Newton's mechanical model of the universe is good enough for all practical purposes except where it isn't. Then you have to go relativistic or quantum. Probabilities only apply to future events. Once events have been observed, the probability changes.

They are not very explicit about it (which is a huge problem by itself), but they seem to be saying that they are only predicting the "first wave" - so they are not predicting 0 deaths after July - they just defining them to not be a part of the "first wave" anymore. So the way they present the model predictions is even more unbelievably wrong than the model itself!

StatNews piece on the IHME model: https://www.statnews.com/2020/04/17/influential-covid-19-model-uses-flawed-methods-shouldnt-guide-policies-critics-say/

Interesting that the model hasn't been updated since April 13, which was the point when daily deaths started to rise above the model's predictions.

The problem the modelers have is how to account for reduced transmission in a continuous model. If you don't set it to zero, you can end up with 1/10,000th of a person still sick, and then the virus comes back full force a couple months later, despite having literally eradicated it. So yes, setting it to zero is wrong, but not doing so is also wrong. Because all models are wrong.

Perhaps you think they should be using an entirely different and more sophisticated model, and maybe they should, but it turns out that those have other drawbacks, like needing far more data than we have to calibrate and build, or needing you to make up inputs.

(1)

A wrong model could be useful if the action (based on the module) can compensate the models' error effectively. Usually, you need to know some properties of the model's error.

(2)

Even a wrong model could be very useful. For example, the earth is flat. That wrong model setup the question correctly and so that people could start thinking the shape of the earth.