The Minnesota Modeling Update, Part 1

It is going to take me a couple of posts, at least, to thoroughly address these issues.  I apologize in advance for what will be seemingly rambling and lengthy posts, but this is a complex matter and there are so many issues with the Minnesota model that it takes a long time to unpack them all.  In addition, the materials relating to the model are somewhat scattered across three documents and the briefing and are not well organized.

There are several basic issues with the model:  1) it isn’t consistent with clinical reality; 2) it uses outdated or inapt data to inform assumptions; 3) it ignores the complete failure of earlier versions or this version to comport with reality in Minnesota, even though this version supposedly was parameterized to the actual experience in the state.

The new and improved Minnesota coronavirus epidemic model, version 3, was unveiled this week.  The materials are here and the video from the briefing is also available on YouTube and other platforms.  (Mn. Modeling Materials)   (As a side note, the Governor must not have been too impressed with the ongoing high level of deaths projected, because he promptly on the same day lightened up on stay-at-home and business shutdowns.  And the modelers must have been a little disheartened because they kept projected deaths up, and then the Governor cuts the legs out from under their mitigation scenarios.)  The materials include a powerpoint slide, a technical paper and some FAQs.  They said they were going to provide some updates in the next couple of days, including full uncertainty estimates, but I haven’t seen them yet.  A couple of preliminary notes.  Minnesota has been exceptionally transparent about model development and operation compared to most people.  As I said when I wrote the earlier series about the model, this is a very difficult challenge and I have no reason to doubt that the modelers are doing their best to come up with a good model.  So while I definitely think it could be better, I don’t in any manner question the modeling teams’ motives, but they clearly are not experienced and frankly, at times don’t seem to be checking their work or keeping up with the latest data or research.

Once again, I think it makes sense to look at the schematic for the model first.  This drives formula development.  There are some significant changes from version 2, which are generally an improvement.  The teams’ apparent lack of actual clinical or health care experience, however, may still be hurting them.  The new schematic moves closer to what I would view as a realistic understanding of the epidemic.  Again, this is based on a pretty standard model that says you have a population, which is “Susceptible” to the pathogen.  Some people become “Exposed”.  Some of those “Exposed” become “Infected”.  Once Infected, the disease takes its course and the person either goes to the “Recovered” bucket or experiences Death.  Along the way, intensive health care may be needed by some patients, including Hospitalization and ICU use.

According to the literal drawing in the technical paper, people go from Susceptible to Exposed, then some percent go to an Asymptomatic bucket, and from that bucket all go to Recovered.  Some percent go from Exposed to Infected and from Infected some apparently go relatively directly to Recovered, some go to a Hospital bed but don’t need ICU care and some go to a hospital and need ICU care.  Some of the people in each of the ICU and Hospital without ICU buckets go to Recovered and some to Death.  Some of the Infected who didn’t go to a Hospital, also go to Death, to account for people dying at home or at least out of a health care setting.

So, what is still wrong with this picture?

1)  The model still has everyone in the population being able to be Infected.  There is no filtering at the Exposed stage.  It is a standard concept in epidemiology that not everyone has the same susceptibility to infection after exposure.  There is now ample evidence to suggest that this is the case with coronavirus, including increasing evidence that a number of people may have pre-existing antibodies and other immune system components that repel the virus.  So you have to account for people who are exposed but not infected.  Not doing that feeds more people into the formulas who then end up with Hospital or Death than is warranted, unless you make some other adjustment later, and they don’t.  In the materials, the modelers also say that 88% of the population becomes Exposed and Infected.  That is an extremely high number and I don’t see any justification given for it.  I have not seen any suggestion anywhere that this is the appropriate percent of the population to use.  Most people are in the 60% to 70% range, and I have shared research suggesting that variation in susceptibility and infectiousness likely leads to a much lower population immunity number.

2)  The model assumes that symptomatic and asymptomatic individuals are equally infectious.  This is also likely in error.  While there does appear to be some transmission from asymptomatic persons, it also appears to be significantly lower factor than symptomatic transmission.  That makes clinical sense because asymptomatic individuals in general would have lower viral loads and viral shedding.

3)  For a reason I really can’t understand, they treat ICU use and ventilator use as co-equivalent.  If you need an ICU bed, you get ventilated and if you need a ventilator you end up in the ICU.  That is just not consistent with actual medical practice and as I have explained several times, and actually sent the current care guidelines to the modelers, current recommendations are for far less use of ventilators.  It is also the case that there is nothing magical about the concept of an ICU bed.  The reality is that there will be plenty of beds to treat however many patients need whatever level of hospitalization under any reasonable scenario.  So at this point, we should just stop the nonsense about over-running health care resources, and the model doesn’t need to be complicated with a notion of an ICU bucket.  If you are concerned about health care resource use, you can address that by comparing hospital bed needs with capacity in a separate analysis.

4)  The entire model continues to be driven by a “contact” sub-model.   It is not possible to overstate the importance of this model in driving everything.  It determines how many contacts an individual is deemed to have and hence their ability to be infected or to infect others.  Those contact rates feed all the buckets.  And the mitigation measure modeling then makes assumptions solely about reducing contacts, and these assumptions drive the changes in cases and deaths resulting from a mitigation measure.  This contact sub-model is based on a single study from Europe, adjusted for Minnesota demographics.  The contact model has no adjustments for population density, residence type or a myriad of other factors which would drive contacts.  It doesn’t in any manner address the long-term care facility issue which is the sole driver of Minnesota deaths.  A long-term care facility population obviously has a very, very different contact mix than does the general population.  That population should and easily could be pulled out and modeled separately.  The contact model is completely inadequate for its purpose.

5) The modelers then didn’t use the contact model to determine transmissibility.  They instead just moved parameters around until they got an initial estimate of 3.87, which they thought was consistent with early phases of the epidemic in 11 European countries.  Just shows how little faith you have in your contact model.  And later estimates of transmissibility don’t support the number used here.

6)  They aren’t varying ICU or hospital stay length by age.

7)  While they don’t have a specific undetected infections parameter, they initially calibrated the model by assuming that 2% of infections had been detected as of March 22nd.  This is up from 1% in the version 2 model.  There is no longer a factor to account for undetected infections in the process of feeding buckets and all the formulas, which seems bizarre when you are saying that initially you missed 98% of them.  Instead, what now seems to happen is that you treat all missed infections as asymptomatic.  But that proportion of infections is only set at 41%.   So just a reminder, the model now says everyone who is exposed gets infected and they either are asymptomatic and recover or they die at home or recover at home, or go to the hospital, with or without ICU, and either die or recover in the hospital or ICU.  Setting the asymptomatic bucket at only 41%, is just absurdly low if that is where all your undetected infections are supposed to go.  So you suddenly went from only detecting 2% of infections to detecting 60%.  Better than version 2, but still not realistic.  And that rate is what ultimately determines serious illness and death.  (A side note, in the version 2 modeling materials on slide 12 of the powerpoint the original slide deck, the modelers said that as of May 4 (if I read the graph right), 1.49% of Minnesotans, or 85,000 people would be infected.  In actuality on that date the state reported 10,162 cases.  So there were approximately 8 times more cases than detected, according to the model.  Yet in version 3 undetected cases, which all appear to be treated as asymptomatic, are set at no more than 40% of all cases.   12.5% were appear to be a better percentage to use, if you are following what you think Minnesota experience actually was, but that would drive deaths down to, well, the level we actually see.)

This post is getting long, so I am going to continue in a second post to avoid reader fatigue (and mine).  But I think you get the picture, this version of the model isn’t going to produce outputs that are any more accurate or useful than version 2 did.  And we see this in the uncertainties in this model version’s output.  The bands are wider–which is the exact opposite of what you should see when you refine a model–you should have more data which improves your assumptions and parameters and gives you more confidence in your central estimates.

Two big picture points regarding comments made in the modeling materials.  One is that the technical paper says “In the absence of effective therapies or a vaccine, non-pharmaceutical interventions such as social distancing and case isolation provide the best available strategies for mitigating the impact of the COVID-19 pandemic.”   That is not accurate.  The best strategy  is whatever gets you to population immunity fastest, with the minimum amount of coronavirus damage.  There is no vaccine and we don’t know when there will be one.  There is a basically zero risk to almost all Minnesotans of serious illness and we know the group with vulnerabilities.  So the best strategy is one that allows the buildup of population immunity, which will reduce transmission and protect all remaining uninfected persons, including the vulnerable elderly.  That is the strategy we should have embarked on and still can.  And it gets us to safety within a couple of months.

The second point is that in the Frequently Asked Questions, the modelers acknowledge that the model does not capture economic, or non-economic harm associated with the mitigation strategies, although it is clear that “the mitigation strategies to delay and reduce its (the pathogen’s) impact affect Minnesotans’ health and economic wellbeing.”  Why doesn’t it include that information?  It wouldn’t be hard to include it.  The team is led by a health economist.  And if this team isn’t doing the work to take that into account, who is?  Surely the Governor and the legislature would want that information before taking any action.  Most of those last couple of sentences are tongue in cheek.  We know now that the Governor had no such analysis or modeling done.  The only conclusion that can be drawn from that is he just didn’t care or he was incredibly reckless in making his decisions.