The Modelers Respond to Minnesota Legislators’ Questions and I Respond to The Modelers

This is long.  The Minnesota legislature is trying to do its job and understand what the heck is going on over in the Executive branch.  They asked a lot of questions about the models.  The black text below are the answers they got.  The red text is my response to those answers.  One observation I would make is that the legislature is entitled to a little less evasion and obfuscation.  Grammar and wording issues in the questions and original answers are as they were in the original.

What potential changes in the parameter assumptions would have the biggest impact on deaths?

Because mortality is really an interplay of a range of variables, it is difficult to say what single parameter would have the biggest outcome on mortality estimates. For example, how transmissible the virus is and how fatal it can be for certain populations, are key factors in mortality estimates. With regard to what recent changes in our model had the greatest impact on reducing mortality estimates, there are two primary factors: 1) By increasing the available ICU bed capacity (to 2,200 beds with ventilators), fewer patients were estimated to be caught without needed ventilator capacity, which otherwise would have dramatically increased their probability of mortality. 2) By adopting shorter estimates of length of stay in hospitals and in ICUs from European and U.S. data, further reducing potential capacity constraints, thereby improving expected mortality. 3), Finally, by simulating ongoing and longer-term physical distancing for the population most vulnerable to experience negative outcomes, we effectively reduce their exposure to the virus, thereby reducing the need for hospitalization and ICU care for people with higher probability of mortality.

The single biggest factor in their model is actually what they call the detection rate.  This is not well applied.  A better way to think about this is what percent of the population that is exposed to the virus gets infected and what percent of those infected have a serious illness that leads to death.  Varying those numbers have the biggest impact on mortality rates.  At this point, it is clear that there is no shortage of health resources in the state to deal with any realistic estimate of serious illnesses.

How many deaths are directly related to the lack of ICU capacity?

The model doesn’t make this distinction directly in its output of estimated deaths. The way the model incorporates potentially inadequate ICU/ventilator capacity in the face of demand is by assuming increased mortality (between 1.5x to 16.x), depending on age and comorbidity status.

The issue about ICU beds is a red herring.  You have to understand that an “ICU” bed doesn’t mean anything, the question is whether a hospital has adequate equipment and staff to provide for the care needs for a seriously ill patient.  Any hospital bed can be turned into an “ICU” bed, as was shown in NYC.  So the true capacity for handling serious illness is some proportion of the existing hospital beds in the state, which is over 10,000, plus the creation of additional beds in new settings, which also was done in NYC very easily.  So there should be no deaths attributed to lack of health resources in any realistic scenario.

What happens to the death estimate if we had 3,000 ICU beds (and equipment and health care providers)? How about 4,000?

At this point, we have not modeled further expansion of ICU capacity with ventilators to assess impact on mortality. Future model runs can explore the potential impact of a more complete match of ICU capacity/ventilator demand and supply, if surge capacity beyond what has been modeled is indicated. While not a linear relationship, further expansion of ICU capacity has the potential to further reduce mortality, assuming it is appropriately paired with staff, personal protective equipment and ventilator capacity.

See the answer above.  And note further that ventilator capacity is even more of a red herring at this point.  The modelers presumably are keeping up with the research literature and are aware that the new guidance is to put fewer patients on ventilators because for many patients they were actually worsening their condition.

Regarding the assumptions used in the model for days in hospital and days in ICU – are those averages, medians, a statistic generated by a 95% confidence interval or judgment relative to any of those calculations?

Model inputs for length of stay in hospital and ICU days are estimates based on summary measures (means or medians) in the literature. With sensitivity analyses, values are sampled from these ranges during repeated simulations of the model to help measure uncertainty.

Regardless of the source, it appears that the numbers being used in the model are outdated and too high.

People without a comorbidity index of >1 probably spend less time in ICU if they need ICU at all. How does the model handle what could be a stark difference in ICU duration?

Average lengths of hospital and ICU stay used in the model do not currently depend on age or comorbidity status; however, the probability of ICU admissions and death is dependent on age and comorbidity status.

Not really an answer to the question, the model should directly adjust length of stay in an “ICU” bed by patient characteristics.

Regarding the R naught number, where does the number come from?

R0 is not model input; rather, it is calculated from model output to estimate the probability of transmission per effective contact (R0 is related to transmission probability). Empirical methods were used to calculate R0 using doubling times in the first 20 days of the model simulation (without mitigation). Model input values for transmission probability (β) were tested to find values that yielded a plausible range of R0 values from 2.5 – 4.7. These β values ranged from 0.025 to 0.045, with a β of 0.035 yielding an R0 of 3.87. The reference for the base case R0 value is:

Flaxman S, Mishra S, Gandy A, et al. Estimating the number of infections and the impact of non- pharmaceutical interventions on COVID-19 in 11 European countries. 2020;(March):1-35.

The range of R0 values was informed by:

Li R, Pei S, Chen B, et al. Substantial undocumented infection facilitates the rapid dissemination of novel coronavirus (COVID-19). medRxiv. 2020;3221(January):2020.02.14.20023127. doi:10.1101/2020.02.14.20023127

Ferguson NM, Laydon D, Nedjati-gilani G, et al. Impact of non-pharmaceutical interventions ( NPIs ) to reduce COVID- 19 mortality and healthcare demand. 2020;(March).

Liu Y, Gayle AA, Wilder-Smith A, Rocklöv J. The reproductive number of COVID-19 is higher compared to SARS coronavirus. J Travel Med. 2020;27(2):1-4. doi:10.1093/jtm/taaa021

The transmissibility number is important but even more important, and not currently addressed by the model in the correct way, is the percent of people who are exposed to the virus who actually become infected.  The modelers appear to assume 100%, the research does not support this. 

Why are we using data from other countries?

With evidence about SARS-CoV-2 slowly emerging from across the globe, reports from China provide the most complete clinical and epidemiologic data available. This is beginning to change, with evidence on asymptomatic patients, length of stay, and comorbid conditions emerging from European and U.S. settings. Current contact patterns are based on survey data from numerous European countries. We are in the process of validating these assumptions with a current Minnesota survey. In addition, the following model parameters are informed directly by data from Minnesota: demographic information, the distribution of comorbidities across age groups, and data on COVID-19 cases and deaths.

Numbers from China always should only have been used with very heavy caveats. They were known to be untrustworthy.  In just the last couple of  days we have seen China almost double its total number of reported cases, saying they just found a large number of asymptomatic infections.  Chinese numbers in particular cannot be relied on for either infection fatality rates or population death rates.

Where are we at with testing?

At this point, testing as a means to identify the rate of infection is not built into the Minnesota Model. It is a potential model extension that the research team is currently exploring. In and of itself, it will not provide additional detail on the impact of the disease on people by race and ethnicity. MDH is looking at bringing additional data to bear on that.

This seems like a complete non-answer.  The right question is what are the state’s plans for a comprehensive infection and antibody testing study so we can, as the Stanford study is doing, identify the true infection rates.

How does Minnesota plan to put protective measures to be sure the most vulnerable and folks who are most likely to die from this virus are serviced different than others? It is our understanding that the Administration is exploring this right now through a workgroup of experts. We have passed on this question to the Governor’s Office in case there is further detail available.

It is apparent at this point that solely targeting protection of vulnerable groups would be completely adequate and that most people could safely return to their normal lives.

Who paid for the model? Model development was funded in part by MDH using federal emergency funding.

Will the code be available? We are also working towards posting a user interface and programming code in April. At that point the code will be thoroughly commented, so that analysts can develop an informed understanding what sections of the computer code implement what mathematical representation of the propagation of the virus in the population.

So here is a question, why isn’t the model being updated and run more frequently and the results released to the public.  Or if it is being run frequently, why aren’t we seeing the results.

What’s the goal?

The goal of the model is to provide decision makers with a directional sense based on empirical information about how a range of options would play out in the course of the epidemic. Like other models, it is based on the most complete and current set of available information, which is detailed in the technical documentation, aligned to Minnesota data as best as possible.

One thing that is critical for the modelers to do is more clearly state the limitations of the models and the dangers of using its results for decision-making when there is such uncertainty in those results.  The early runs should not have been used and the modelers should have made that clear.

When will co-morbid statistics be available?

The age-specific estimated proportions of Minnesotans with one or more underlying conditions are included in the model technical documentation (Table S2). These underlying conditions that were used at this point include: hypertension, diabetes, cardiovascular disease, chronic obstructive pulmonary disease (COPD), chronic kidney disease, and cancer.

How does the Minnesota model differ in four-month measure than current 4 months to their 18 months?

As you know, the MN COVID-19 model and the IHME model developed by researchers from the University of Washington differ fairly radically in methods, assumptions and time horizon. These differences are responsible for the substantially different outputs both models generate. Though no model can claim to be right, many researchers are concerned that the methods, assumptions, and time horizon chosen by the IHME model created some highly optimistic projections that have not been well communicated. Given the many questions we have received about comparing model results, the research team is exploring for our next release to program a scenario where at least the timelines are more directly comparable (4 months).

I completely agree that the Minnesota model is better than IHME, by far, but it still is completely inadequate in its treatment of key variables, like the percent of people who get infected, the percent who get seriously ill and actual health resource availability.

Have you factored in treatments?

No, the model does not currently include estimated treatment impacts. At this point, we are not aware of FDA approved treatment for COVID-19. When a treatment does become available and its impact on infection or recovery is understood, the research team will be able to incorporate that evidence into the model.

I think the right assumption now is that no curative treatment or vaccine will be available for at least a year.  So the model, as it does, should run out for a year.  The modelers have taken the correct approach here.

What percent of patients die in the ICU?

In its current state, the model assumes that all COVID-19 deaths occur in individuals who progressed to the ICU. As shown in the technical documentation (Table S3), estimated mortality rates vary across age and comorbidity ranges.

How many are over 80?

The ICU mortality rate for individuals aged 80+ years is at least twice as high as the rates for other age groups.

I am going to assume that is correct but the more important issue is of patients in any age range, how many will die, whether they go to an “ICU” bed or not.  Some people are dying at home or in another setting, in part because they have an advance directive that forbids use of ventilators or other intensive care.  Correct modeling would focus on rates of infection, serious illness and death by age band and comorbidities, regardless of course of treatment.

Usual public health is to quarantine sick and vulnerable not healthy. Why change now?

The pathogen that brings on the COVID-19 disease is highly transmissible and has a high fatality rate. One concern is that if the disease can move through the population unmitigated, it can quickly overwhelm health care, resulting in even greater death, because some patients won’t be able to access needed ventilator care. Therefore, the focus has been on delaying peak infection through a number of strategies. In terms of isolating or quarantining patients, the challenge is that we can’t identify the people who have been exposed to the pathogen because we also can’t identify who in the population are currently infected and contagious – this has to do with how the disease evolves in patients over time. We only know about those who have been diagnosed or show symptoms that make it highly probable a patient has the disease. In that circumstance quarantine is not effective but isolating positive (and probable) cases is helping.

I do not understand this answer.  The question is since we know that only a few groups are vulnerable why are we not focused on protecting them, instead of isolating those who have no risk.  Given the spread of the virus in settings like nursing homes, the strategy clearly isn’t working, but is creating enormous economic damage.

What is the percentage of those who go on the ventilator survive?

This percentage is not explicitly part of the model and that data does not appear to be formally available in the literature yet. Rather, mortality rates are applied to individuals who progress to the ICU, regardless of whether they are on a ventilator or not. Estimates of mortality for COVID-19 patients who end up on a ventilator range from 66% to 97%; data released on April 10th from the Intensive Care Neonatal Audit & Research Center in London found a mortality rate around 66% for people with COVID-19 who had a ventilator within the first 24 hours, compared to a 36% mortality for people who did not, but were still in the ICU (full report available at: https://www.icnarc.org/DataServices/Attachments/Download/c31dd38d-d77b-ea11-9124-00505601089b).

Little confused by this answer, first they say there isn’t data and then they give you some.  Given the change in guidelines about use of ventilators, I would agree that it isn’t necessary to put that in the model.

Where did you get the Ro factor?

The base case R0 of 3.87 comes from the following published study:

Flaxman S, Mishra S, Gandy A, et al. Estimating the number of infections and the impact of non- pharmaceutical interventions on COVID-19 in 11 European countries. 2020;(March):1-35.

Additional studies informed the range of plausible values (2.5-6.0) used in sensitivity analyses:

Li R, Pei S, Chen B, et al. Substantial undocumented infection facilitates the rapid dissemination of novel coronavirus (COVID-19). medRxiv. 2020;3221(January):2020.02.14.20023127. doi:10.1101/2020.02.14.20023127

Ferguson NM, Laydon D, Nedjati-gilani G, et al. Impact of non-pharmaceutical interventions ( NPIs ) to reduce COVID- 19 mortality and healthcare demand. 2020;(March).

Liu Y, Gayle AA, Wilder-Smith A, Rocklöv J. The reproductive number of COVID-19 is higher compared to SARS coronavirus. J Travel Med. 2020;27(2):1-4. doi:10.1093/jtm/taaa021

There actually are studies suggesting even higher transmissibility numbers.  But again, it is erroneous to focus exclusively on transmissibility, what matters is what happens when people are exposed, and the following rates of infection and serious illness.  I don’t think the model has accurate inputs on these numbers.

Why in your slides do you use the ICU bed count of 2200 when your own website dashboard list 2770 can be available in 72 hours?

We used an estimate of approximately 2,200 beds (with ventilator) available to potential COVID-19 patients to account for the fact that there will be patients without COVID-19 requiring ICU-level care.

See my comments above, it is erroneous to focus on “ICU” beds as being some magical setting.  Beds and other equipment to appropriately treat patients is available and more capacity is easily created.  And see also the comment regarding the change in recommendations regarding ventilator use.

Why don’t we pursue the strategy in the second scenario now that we have successfully bought time and haven’t overwhelmed the healthcare system?

Both scenarios you refer to reflect hypothetical scenarios with a one-time mitigation strategy that is artificially set to “end” at a certain point. To that extent, it may not at all reflect the actual mitigation strategies under consideration by the administration, nor does it capture the range of inputs the administration would consider to craft its response. To your question, specifically, the two hypothetical scenarios do indeed produce similar “output.” But they also represent states where tens of thousands of Minnesotans would die and where hospital ICU capacity would be exceeded, neither are likely acceptable outcomes.

This is the key policy question and this answer is nonsensical.  The answer, after the gibberish first two sentences, is that they do produce the same outcome.  It is not true that “ICU” beds would be exceeded.  If the model used accurate numbers about infection rates and serious illness rates, it is also not true that tens of thousands of Minnesotans will die.  There are no mitigation of spread scenarios presented by the Governor or anyone else that suggest that under the assumptions the model currently uses, we wouldn’t eventually see that number of deaths.  So this is just an evasive, illogical and misleading answer.  There is no reason not to move to targeted mitigation of spread.

If the model accounted for deaths that may be due deaths in congregate care settings that what would the curve look like and can you send that to us?

What we are trying to relate with the observation is that the mortality curve (which we did not display) could change for a number of reasons, including if COVID-related mortality was to a greater than expected extent due to deaths in congregate settings. Of course, there is tremendous infection control work underway to prevent that scenario.

Model uncertainty (about the mortality curve and other outcome variables) will decrease over time as the model is fitted to more data points. That will give us better insight into whether assumptions about transmission in Minnesota aligns with data from other locations. In addition, we are exploring the possibility of adding a population subset that resides in congregate facilities to the model to allow us to make more finely tuned assumptions across people with different levels of vulnerability to COVID-19.

They definitely need to add the capabilty to vary infection and serious illness rates by setting.  Doing that will greatly lower overall rates of serious illness and deaths in the model.

Why is MN showing lower mortality when nationally certain races show comparatively high rates per positive tested for those same groups? Estimates of case mortality this early in the epidemic are highly variable and extremely unreliable since there is a considerable time between infection and death.

Okay, so this is a key admission.  The results are completely unreliable, so why are they using them for policymaking.

The model shows upwards of 22,000 deaths. Given the community mitigation strategies in place, shouldn’t the death rate be declining?

The MN model simulates a full year to capture both the short term and long-term effects of different mitigation strategies, allowing for recurring peaks as social distancing changes over time. Although death rates may decline while the stay at home order remains in place, most of the deaths are predicted to occur after the stay at home order has been lifted. Ultimately, mortality is related to the length of any mitigation strategy, as the hypothetical scenarios aim to demonstrate.

Also hard to understand this answer, deaths are occurring during the extreme lockdown we currently have and they aren’t now due to infections incurred before that lockdown went into effect.  It is probably true that under the model, deaths are lower when mitigation of spread measures are in place, and rebound when they are lifted.  This is likely realistic.  But the question, which was asked in another section above, is what happens in each scenario with different mitigation strategies.  And the model says you get the same outcome with a more targeted lockdown as you do with an extreme one.

Why are we treating Minnesota as one case? Should we be looking at what is happening in rural MN vs the Metro?

The current version of the model produces statewide estimates for Minnesota and has not been extended to model the geographic strata of the state. Future iterations of the model may explore this possibility, but that addition will be only as good as we can inform it with additional data on contact behavior or differences in the rate of transmission. Our hope is that evidence can emerge through ongoing science rapidly. It is important to note, however, that while some parts of rural MN appear to have low rates of detected infection, others have rates of infection higher than the Twin Cities.

Probably is appropriate to have different population density scenarios, as this may affect transmissibility.  This could be done by attaching to the population in the model different density factors which reflect the actual spread of density in the population.