BONUS / Running Out of Time / Michael Donnelly

I'm Dr. Celine Gounder, and this is Epidemic. In this episode, I speak with Michael Donnelly, whose background is in economic research, financial policy, and data science. He's worked in the US federal government in various research positions for 6 years and in the private sector for almost 7 years. Last Wednesday, Michael published an analysis to provide a broad sense of the scale of the crisis the country is facing. He found that reported cases of COVID-19 in the United States dramatically understate the current number of infected Americans. The same week, Michael published another analysis looking at the New York City metro area. His results were startling. The outlook for New York City and COVID-19 is bleak. The policy response has been far too slow and too weak to meet the needs of the moment. His analyses came just days before an Imperial College report predicting high fatalities in the U.S. if the country doesn't take extreme measures.

The place where I start with all these conversations is that I'm not an epidemiologist and I'm not a public health researcher. My background is more in economic research and financial policy and data science. Prior to my current job as a data scientist in the tech sector, I worked in the federal government for about 6 years and in the private sector for about 3 years in financial policy research. And so my specialties in all of this: quantitative communication and data science and forecasting.

How did you decide that you wanted to delve into this? I mean, this is not your regular job, right?

No, it really isn't. It was really— it was a fascination. I think is, to be totally honest, it's kind of the topic that I've always been interested in. I did some training in health economics in grad school, and I started digging into this really in January and was pretty convinced by February that this was, you know, like pretty major, even for the US. I found over those weeks that I was getting pretty exhausted trying to convince friends that this was not the seasonal flu. And so because of my background, my skill set, I turned towards quantitative analysis and empirical research to answer these questions first. And that's, you know, that's really where I started going to begin with. So really in March, after becoming pretty tired of trying to convince friends, I decided it'd be a lot easier to compile all of my thoughts in written form. And so like, and also providing that really as a resource for people to feel empowered to explore the existing scientific research on the topic, rather than just believing some friend of theirs who thinks he's got a good opinion. So I started writing up about a week and a half ago, all of my notes, and it quickly turned into like a 20-page Medium post with the research and my own quantitative analysis and modeling. Before I put it out there, because it had some disturbing conclusions to it, I ran this by a bunch of friends. I think probably about 6 to 8 of my friends who I had review it had PhDs, JDs, or some sort of doctorate in a variety of fields between public health, economics, and political science, and law. So I was really making an effort not to put out bad information, but my main goal was to put out information at a level that you didn't have to have a degree in science or in statistics to understand. So that was my goal with my first Medium article. It was really kind of a national overview. Shortly after that, I think I released it a week ago today, I started thinking through the consequences in my own life. And part of that meant thinking about my rec sports league, which was about to restart its season. And so I ran, as I realized this, the executive board meeting was starting in about half an hour. And so I ran down to the board meeting to ask them to cancel the upcoming season. And as I was waiting for my turn to speak, I replicated my national analysis for the New York City region. And got pretty disturbing results about what that meant for New York's healthcare system and its capacity to absorb, you know, the forecasted severe and critical cases into its hospitals. That was the realization that I needed to do a more formal analysis for New York, and by the next day around lunch, I had finished the analysis and was ready to post it on Twitter after, you know, getting some other friends to review that specific analysis as well. That's when I started reaching out to all the government officials that I could possibly reach out to because the conclusions were so jarring. It really told me that we had very little time left to take much, much stronger governmental action to increase social distancing before we had essentially locked in too many infections that would overwhelm New York's hospital system.

Some follow-up questions about that. So having looked at sort of the national scenario as well as the New York City scenario, how divergent or different are those?

Well, I'll start off by saying that there are a lot of models out there that do a really good job dealing with different geographies and densities and the way that people move around in different ways. Based on different cities. Those models also are unfortunately very complex and difficult to understand. The models that I used are essentially simple, almost 100-year-old models called SIR models, and they just work at the top line, summed up the whole country or summed up the whole city. Because they have the same structural form, if you are saying that the the virus progresses in the same way, they kind of look very similar in terms of how quickly they progress under these simple models. So that's kind of the short answer. The wrinkle in that is that it matters a great deal about where you estimate your current position to be versus the relative total susceptible population. So in the New York City region, it really seems that Well, I haven't done a formal analysis of all the cities in the US. It seems like it's the most, the highest priority city to be addressing the problem right now. You could also think that maybe Seattle would be close on its heels.

To some people, that would be surprising because Seattle was hit first. Why would that be?

I'll try and point out throughout this conversation where I've done what I think is firsthand quantitative analysis and where I I feel like I've got greater uncertainty. In this place, I haven't done as much firsthand quantitative comparisons, but I think it's important to point out that it seems that most of Seattle's cases, at least detected cases up front, were in a nursing home among a much more susceptible population, a much more vulnerable population among older people, whereas in New York, you had its first cases detected among 50-somethings, 40-somethings who were very mobile throughout the city and didn't have a clear lineage to trace back how they had become infected in the first place. And so since it could really have been, you know, if you know New York City, anywhere along the Metro North corridor or in the subways of Manhattan, it stands to reason that there were substantial numbers of undetected cases that were circulating around America's most dense city. And so for that reason, I think it's reasonable to assume that the overall reproductive rates that have been detected in a variety of studies based either on the experience of China or the experience on the Diamond Princess cruise, which I— those studies I leverage heavily in my papers because they really could do a full population study in those papers. I think they give you a really good picture of what to expect in New York without intervention.

How are you able to guesstimate, estimate, you know, how far back coronavirus may have been introduced into New York City, when the first cases may have started circulating here?

Yeah, I didn't do that analysis, but the last time I updated the analysis was the end of March 13th, which was last Friday. As of that time, we had about 400 cases in the New York City region that were reported, and we were estimating anywhere from 1,300 to 2,300 cases overall in the regional population, that most of which hadn't been detected. That suggests that it had been New York City for, for many weeks, but any greater specificity than that, I couldn't, I'd just be guessing.

So maybe if you can just talk us through, first of all, what you found, you sort of gave us a very, very broad overview of what that was, but maybe a little bit more specifically.

Sure. The numbers that I found in New York City, which are now 5 days old, about a third of the analysis is just focused on trying to understand the actual current number in the population of infected and potentially infectious cases. As I mentioned before, we had reported at the time 400 cases, and then there's a real question of, okay, how do you kind of ballpark the right number of cases in the population? You know, the place to start is to just acknowledge that directionally the reported number of cases is an underestimate, assuming also a fairly low false positive rate in the testing results, although that, that may be somewhat of a weak assumption given some reporting that I've seen on that. But regardless, even with a relatively high false positive rate in testing, it's almost certainly an underestimate because of the limited number of tests out there and the strong restrictions that even still hold to this day about who could be tested. But it was even stronger restrictions on who could be tested at the time last Friday. So the approach for coming up with the estimate, which I detailed a lot more in my first paper, was to attempt to estimate 4 different parameters. And those parameters were the asymptomatic rate, mild case rate, and then 2 detection rates. So I'll start first on the why we want to look at the asymptomatic and the mild case rate. Mild and asymptomatic cases are just a lot harder to catch. Because they present, as you know, as a kind of as a cold, and people don't get tested for colds regularly. And so we kind of need to know first how many people who get COVID-19 are going to experience these mild and asymptomatic versions of it. So I went to a few different papers. There's a couple of papers on the clinical characteristics of the virus coming out of China, one in The Lancet that I used and one in New England Journal of Medicine, and they both placed the mild case rate at around 80%. I think one had 81%. And both papers had about a 1% asymptomatic rate, if I'm remembering correctly, whereas there were two papers coming out of the Diamond Princess cruise, which was unfortunately for the passengers a pretty good natural experiment for research purposes, because you can test everyone on the cruise, they came up with a couple of different estimates for the rate of asymptomatic cases. One paper placed that at the lower end, around 17%, and another paper placed it at the higher end, around 35%. So why does that matter? It matters if we believe both papers, that the easiest way to reconcile them is to say that the clinical papers coming out of China just weren't getting into their sample the asymptomatic cases because those weren't lab-confirmed cases. And makes sense that they would be missing quite a few asymptomatic cases based on what I know about the Chinese testing protocol. So how to adjust then this mild case rate? It's actually not that hard. You just assume that that if the overall population actually has, say, 35% asymptomatic cases, then of the remaining 65%, 80% are mild. So then you can just add up those percentages and you can get the overall mild and asymptomatic rate. Since both mild and asymptomatic cases tend to be harder to detect, we've been separate out the detection rates for these non-severe cases and for severe and critical cases. By the way, the calculation of the severe and critical case rate is much easier. Once you've figured out the mild and asymptomatic rate, you can just subtract that from 1, and you get the remainder is going to be severe and critical cases, which ends up being somewhere just under 10%, depending on, depending on which estimates you use. The next step, once we've kind of got that, is to just understand how quickly people transition from one population to the other.

So there's also been a lot of discussion, as you know, about school closures, mass gatherings. Those are two slightly different questions. Based on your modeling and your looking at the evidence, and I think you've been involved in the discussions, at least in New York City, around school closures, what do you think the evidence shows and what's been your involvement in those discussions?

It does seem to show in various studies that I've been reading that, you know, approximately a third of transmissions happen at schools and at workplaces. And so closing down schools, while children don't seem to have as many severe side effects from this, they do seem to be a vector. And there's one interesting analysis that I saw coming out of South Korea where they've done really extensive testing of their population and A lot of people have pointed to that extensive testing as being one of the reasons they've been able to get the spread of the virus under control there. But the finding was that there was a, quote unquote, over-indexing of cases among 20- to 29-year-olds. And so even though they don't tend to show as many symptoms or as many severe symptoms, it seems that because it's a more social group in the age pool that they're one of the biggest vectors for the disease spreading. And so while that's not the age group for, say, public schools, it is a part of the age group for universities. I think between the Imperial College study that found that a significant part of transmission happens in schools and the finding coming out of South Korea that a significant part of the transmissions are coming from 20-somethings, it does make sense that we're going to want to continue doing restrictions on movements and restrictions on schools and universities.

How different are your methodologies? I mean, it sounds like some of it may be that you're just trying to approach it as simply as possible, especially for laypeople to be able to understand what you're doing, but how different are your methodologies from, say, Ferguson, Neil Ferguson or Marc Lipsitch?

The Ferguson paper— this is the Imperial College paper— uses what I believe is an agent-based model, and so this effectively means that they're attempting to model the individuals in society and how they move about and how much they contact each other. This is a pretty new approach to modeling, at least in my background in financial stability research, and it can yield some interesting results that are different than what top-line statistical analyses yield. But on average, it turns out that they yield very similar results. And so what I can point to is the conclusions that both Neil Ferguson and I reach and what parameters we use to reach them. The Neil Ferguson paper uses an R-naught of 2.4, which is almost the same as my 2.3. They, I think, use essentially a serial interval, so this is how frequently that compounds, of just over 5 days, which is slightly more than my 4.7. They find for the U.S. that without intervention, about 80% of the U.S. population would become infected. I find higher than that, but it would make sense that there's slightly different results, particularly on that factor. And then finally, the total number of deaths, which is a super important metric, I think we would all agree, they find about 2.2 million deaths peaking sometime this summer, which is slightly later than my timing estimate for that, but it's also just under twice as high as my estimate for that. From a forecasting and statistics standpoint, all of these results are, you know, what we kind of refer to as within an order of magnitude. So while they approach the modeling from a pretty different mathematical standpoint. What's interesting is that the results come out very similarly. And when you're using models to forecast, it's actually a good thing to have people pursuing a lot of different model forms because if they're all coming out with roughly similar conclusions and they're not collaborating, then you can have a much higher degree of confidence in the findings.

So in other words, you're approaching the problem in different ways, but you're all sort of— your results are all pointing in the same direction.

Yeah, that's exactly right. And part of the reason I chose a simpler model, aside from the fact that I'm not an epidemiologist, was also I think it's important for people to feel like they have some agency over understanding why political leaders are taking such strong actions when it seems like the death rate or the critical case rate is still pretty low in the country. And so I hope that this math and this explanation of the math gives people a better sense of why it's important to act now.

You know, another thing that's been up for debate is duration of the social distancing measures and when we can lift them. What does your model show, and how does that align with the Ferguson model?

I haven't done what the Ferguson model did. My analysis went up through the first part of the Ferguson model, and it aligns very well with their hypothetical no government intervention and no individual response to the virus. Our results are within essentially an order of magnitude of each other and have very similar peaks. In terms of the interventions, I think that there's limited information right now to estimate exactly how big an effect each of these interventions have on the virus. And so the Ferguson approach using agent-based model is some of the best ways that we can deal with that right now. I do think that there's going to be more evidence coming out of the change in the reproductive rate in each of the countries that we've seen— China, South Korea, Japan, France, Spain, and Italy— to see how much we can attribute change in reproductive rate to these policy implementations. There's just very limited data on that right now, so I haven't attempted that work yet. One of the things that we're beginning to discuss among some other U.S.-based data scientists is trying to come up with a way of tracking at least directionally in real time changes in the reproductive factor in different regions of the United States. But in order to do that, we're also going to have to estimate changes to the detection rates, and that's all related to the number of tests as a proportion of the number of infections. So in order to do that, we're going to have to continue to forecast. So there's a lot of unknowns right now, And it's a big mathematical problem, essentially, and a data problem that a lot of people have just begun to work on. So I'm hopeful to have a better answer for that in the coming weeks, but right now I think that the Ferguson paper is one of the best ones I've seen on it. They seem to suggest that the first period of lockdowns is going to be the longest, because we need to get the disease spread under control. Before we can think about lifting the lockdowns so that we don't overwhelm hospitals immediately after lifting them. And I think they place it somewhere around 2 months for the initial period.

Then you lift the measures, but then you have these subsequent waves of disease. So how do you decide when to reimpose measures, and can you do so in a more local fashion as opposed to doing so across the country?

I think there's a couple points there. One key point to get first to understand the intuition behind the subsequent point is that ICU cases or severe hospitalized cases of coronavirus are a lagging indicator. So if you stop people from catching the virus by imposing a lockdown when you're at 100% of your hospital capacity, well, you've got all the transmission that had been happening a week before then that haven't been showing the symptoms. And so you're going to have all of those cases that haven't showed up at the hospital yet arriving at your hospital after your hospital's already full over the next week. So what the paper does is it says we're going to need to estimate capacity triggers for this. And I think that there's a real data problem here. So let's say that the capacity trigger is like a third of the, of the available capacity in the hospital. And that seems kind of reasonable to me. But in order for that trigger to be in place, we're going to need to know what the capacity of the hospitals are in the region and what the number of hospitalized cases are in the region. And as far as I can tell, and I've been speaking to epidemiologists and data scientists across the country, there doesn't seem to be a good centralized source for that information. And so it's going to be a lot harder for even sophisticated state and local governments to do that. And I think as unsexy a problem as it is to say we need to invest in data right now, I think that's got to be a high priority.

How well do both models predict how things have played out in Italy and how things have already started to progress here in the U.S.?

You know, I haven't seen many backtests of the Italy case. I can only speak to my own on that one. And when I ran it on about 2-week-old data coming out of Italy, my model was actually somewhat underpredicting the 2-week-out number in Italy. Now that is essentially within what we expect to be the error for a model. And so it's suggested that these models and thus the parameters that we use for the models that the R-naught, that reproductive rate, how frequently that compounds, those sorts of things, it really suggests that these models are right. And left broadly unchecked like they were in Italy, the future extrapolations, even out a month or two, are right. And I think that's really why people— why governments are beginning to take this so seriously right now, because people have really convinced national leaders that if they don't do something now, these really horrific scenarios could play out.

How far out can you accurately forecast, and how, you know, how much can we trust these predictions? You know, some people are saying 3 months, 5 months of these kinds of interventions. You know, how long are we looking at this? You know, are we going to have to really wait 18 months for a vaccine? You know, what can you say about the you know, how far out you can really project?

I think with almost any forecast, the underlying fact is that the farther out you try and forecast, the less accurate you're going to be. So a year from now is always going to be harder than a month from now, and a month from now is always going to be harder than a week from now. In this case in particular, the biggest source of uncertainty, at least from my perspective, is the policy response. The models that I've done and the models I've compared my work to, particularly the Imperial College model done by Neil Ferguson and his team, they've been criticized in what I think is a productive way. For one instance, they haven't considered the case tracing that could happen alongside some of these other policy implications, and that in fact could reduce future infection rates. Similarly, in their paper, they suggest doing essentially on-and-off kind of lockdown or suppression tactics in order to deal with both the economic and the health policy consequences of these policies of keeping people indoors and social distancing, and that would essentially flatten the curve by having the curve kind of go into waves. But those waves over time may not be evenly spaced, and those waves may not be evenly spaced for a variety of reasons. One is increasing technology, increasing test loads, but also increasing acquired immunity because more people have been exposed. So those sort of dynamics are very challenging to forecast, and I think what a lot of people who are involved in trying to forecast this right now are doing is they're trying to do, you know, two steps forward, one step back. And I think we're just going to keep leveling up the quality of these forecasts over time as we decide which forecasting approach is yielding the best results.

Do you think New York City has acted quickly enough, and for that matter, the U.S.?

The short answer is no. I think we knew enough last week to have taken the action that the mayor took on Sunday, which is to close restaurants, bars, public venues, movie theaters, and schools. Now, schools, it doesn't matter as much over the weekend, but bars, clubs, I think they were reduced to quote unquote 50% capacity, I believe, by the governor, but I don't believe that most of those venues were observing that order. At some point, I think some clever researchers are going to figure out just how many illnesses are to blame for that slow movement. I know more about the New York situation than the national one, but I think New York has some of the best health surveillance systems in the country, and I think that's probably why we know more about New York. We also probably have more because it's a more dense city, but I could see a similar pattern playing out across the country, and it certainly hasn't helped anything when you have national leaders telling people that it's a hoax and that it's not even as bad as the national— as the seasonal flu. People will take action to protect themselves, but they won't do it if they don't think it's real. Even just those words have been a detriment to the spread of the virus in the country, and those sorts of things are going to cost people lives.

Yeah, I mean, I can't agree with you more. How soon do you think we might see hospitals in New York overwhelmed by this?

That's a tough question. It's hard to know how much the decision to close bars and restaurants and other public venues over the last couple of days has slowed down the virus. I— the model that I published on March 14th using data through March 13th suggested that we were going to hit enough infections by March 23rd or 25th, that by the beginning of April, we would be at hospital capacity. Some recent analysis that I've done using data from yesterday and today suggests that I was undercounting the starting place in the New York City region. The trend was accurate, but it's generally bad news. I'm very nervous for the hospitals of New York City. I think the news that the Defense Department sent a hospital ship to New York City, which is frankly unimaginable a month ago, is actually very good news and will hopefully help with the surge capacity. But from what I hear, the outer boroughs of New York have only seen a 50% reduction in subway travel when you look at the, the turnstile data that comes out of the MTA. I don't think 50% reduction is enough if we are already at capacity in the number of infections we have today, which we may be at. So we may be adding beyond capacity every day that goes by that New York doesn't adopt stronger lockdown procedures.

Michael, before we close, is there anything else you wanted to make sure to get across or wanted to add?

I'm impressed by people's desire to get involved and from a wide variety of walks of life. You know, I think there's a lot of volunteer work that's coming out. I think we're resilient. I've seen people who work in gyms, who are personal trainers, who've lost their income source for a long time, immediately hop online and find ways of making income for themselves by doing video fitness classes. I think we're just going to have to be creative going forward to get through this, and honestly, I think there's some opportunities for really good things to come out of this. It's going to be painful in the, in the meantime, but I'm not fully pessimistic about the next Few months.

Epidemic is brought to you by Just Human Productions. Today's episode was produced by Jordan Goss-Poree and me. Our music is by the Blue Dot Sessions. If you enjoy the show, please tell a friend about it today. And if you haven't already done so, leave us a review on Apple Podcasts. It helps more people find out about the show. You can learn more about this podcast, how to engage with us on social media, and how to support the podcast at epidemic.fm. That's epidemic.fm. Just Human Productions is a 501(c)(3) nonprofit organization, so your donations to support our podcasts are tax deductible. Go to epidemic.fm to make a donation. Starting next week, we'll be releasing Epidemic twice a week on Tuesdays and Fridays. But producing a podcast costs money, and we've gotta pay our sound engineers. So please make a donation to help us keep this going. Also check out our sister podcast, American Diagnosis. You can find it wherever you listen to podcasts or at americandiagnosis.fm. On American Diagnosis, we cover some of the biggest public health challenges affecting the nation today. In season 1, we covered youth and mental health. In season 2, the opioid overdose crisis. And in season 3, gun violence in America. I'm Dr. Celine Gounder. This is Epidemic.