Business Over Breakfast Webinar Series with Emory Executive Education featuring Patrick Noonan

Good morning, everybody. I'm Nicola Barrett, Chief Corporate Learning Officer at Goizueta Business School. Thank you for joining us for Business Over Breakfast today. Today, Emeritus Professor Patrick Noonan is back to make even more sense of the models, data, and predictions around COVID-19. With the current rising crisis cases, this is again front page news. And given the uncertainties around school reopenings, state reopenings, and different approaches that countries around the world are taking to try and combat the virus and keep economies afloat, we thought it would be helpful to deep dive again into decoding analytics of COVID As a reminder, Patrick Noonan is Professor Emeritus in the Practice of Information Systems and Operations Management. His field is decision analysis, which draws on a wide range of skills and areas, including the wisdom and analytical methodologies of economics, psychology, statistics, sociology, and engineering, to try and provide prescriptions for improving decision-making in business, in professions, in health sciences, in politics, and in life. So he is the perfect person to guide us through a discussion on what goes into the epidemiological models that are guiding national, state, and organisational responses to the global pandemic that we are dealing with. Patrick is going to kick off the webinar with some information and observations, and then followed by answering your questions, so you can put those in the Q&A section at the bottom of the screen. If you find this session helpful, there's a couple of chances to continue learning with Patrick in a short course on how to approach problems like a management consultant, that he teaches with Lynn Siegel in late August, as well as a workshop on crafting communication for influence and impact. And also, we are in executive education at Emory Goizueta Business School. We are being very focused on trying to bring learning opportunities that will be relevant and helpful for people to cope with the pressures and the challenges that we are all facing today. So there are some other short programs that we are delivering and providing in August, one around building a data-driven strategy. And, you know, critical now to think about how we do that given the rapid adoption of technology over the last few months. And also Peter Topping is taking a deeper dive into leading remote teams. During a crisis. So that's in an upcoming workshop. And just to note, these are not replicas of traditional in-person classroom programs. These are really designed to leverage technology and really provide an interactive and high-impact learning experience for everybody. So I encourage you to try these out. So enjoy the morning. I look forward to the conversation this morning and then also next Thursday morning when we are going to be looking at how do we improve intergenerational communication. So over to you, Patrick, and thank you everybody for attending.

Good.

Well, thank you and, and welcome everyone. Welcome back in the case of some of you. I understand that Some of you might have been regular participants in this wonderful Business Over Breakfast series. Some of you may have been with me about a month ago, early June, when we had part 1 of our conversation about decoding COVID. Let me share the screen and get us set up here. Let's see, let me get to the PowerPoint. There we go. You should be seeing a lovely A lovely shot of our— Is what a business school building at this point. The setup, the framing of this entire conversation is, I mean, it's about analytics and it's about COVID but come on, this is breakfast time. We're just starting our day in many cases. Those of us on the East Coast, we want to keep it a little lighter, a little more casual, and we also want to really want to try to focus on the things that we can do something about. And one of the things we can do something about is our own understanding so that we can make better decisions. And so the framing here really is we're not epidemiologists, most of us, and we're being confronted with this very complex public health situation with infectious disease, and we're being bombarded, inundated perhaps, with various kinds of data and models. And we have to make decisions. We have to make decisions as individuals, as members of a family, as members of communities, in our businesses, and as citizens. What should we do? How can we make some sense of this? Well, I can't demystify that, and I couldn't demystify it if I tried, no matter how much time we had. But we've got some time together this morning where I think we can try to demystify some of these things and, and disambiguate, as my co-author Lynn Siegel and I like to talk about. Disambiguate some of these ideas and terms so that it makes it easier for us to connect and make, and make good decisions about them. One of the things we'll do a few times today is try to make it a little more interactive. We've got a large number of you and you're scattered around the world, and you do have the opportunity through the Q&A feature to enter some questions, and I will respond to some of those during the session and at the end. But we've got an opportunity to actually get some data from you and get to know each other a little bit. So several times we'll use this feature called Menti. And what I'd like you to do right now to get started is go to a browser. It can be on your phone, it can be on your desktop. And today's session is code 327975. And when you go there, you should be served up a question, and I'm going to flip over to that in a second, and we'll start to look at your At your answers. I'm just going to change what, what it is that we're sharing here. And we'll get a sense of who we have with us today. You should now be seeing a dynamically moving word cloud. We have Atlanta in the center, quite a few folks here from Atlanta and Marietta, thereabouts. East Cobb, also in the metro Atlanta area. Welcome Chicago. We have a pretty good presence from Omaha. Omaha and Marietta are duking it out. There for, for the listing. We've got Philadelphia, we have London. Good afternoon, London and New York. Manila, I think I know who that is. Shout out to Joe in Manila. Jacksonville Beach, Chattanooga. Wonderful. We've got a great geographic diversity and of course a lot of people close to us here in Atlanta. Thank you for your quick response. Let's ask a slightly different question now. Let's get a sense of who you are, not from where you are, but the role that you play. I'm going to advance this and you're going to see it serve up a new question for you now. The new question that you're going to have is, what's your professional focus? In these past sessions, we've had a wide variety of different backgrounds, but it's always fun to see who else is here. We've got people from around the world, and my prediction is spread across a lot of different fields. We always have a fair number of people in consulting. For this topic, we get some folks in healthcare as well, and of course a lot of people in general management. And on that topic, I'm not an epidemiologist. I'm an analytics professor, partly retired, which means that I want to try to keep in my lane. I'm going to try to Do what we all should do, which is to think about from our own training and our own expertise, how can we make sense of this? Where there are some experts in these other fields who have their own, the same way that we do in our fields, have their own course of study, in many cases years of education and many years, decades of experience. How can we figure out what it is that they're trying to say and tell us and be able to make good decisions and even help others understand it, but without getting out of our Our lane. So I'm going to try to delineate pretty clearly what sorts of things I think of, how I think about this as an analytics guy, and, and then, you know, where I have to, in many cases, just kind of defer to the experts in those fields. But this might help. I hope it'll help us read what they have to say, listen to their words, and connect it to the kinds of things that we— many of us are business people and business trained— so what it is that we know. And try to— I'll try to serve as that That bridge today, but there'll be several occasions and some questions that you're going to ask where I'm going to say, I don't know, here's how I think about it, and here's what I'm listening for and looking for when I, when I listen to the experts speak. Great, looks like our government and nonprofit people are not here today. They're probably out trying to help save the world. We appreciate that. Maybe they'll watch the recording later. Let me go back to the PowerPoint again and let's And let's see where we are here. Well, here's, here's where we were. This is the chart some of you might have seen if you joined us in June, where at that particular point we had over 7 million cases globally, global deaths at about the 400,000 mark. U.S. cases were still under the 2 million mark, but just barely. Here's what it looked like today. 12 million cases around the world. U.S. confirmed cases over 3 million now. Global deaths well over half a million. So it's been— there's been tremendous amount of growth in the past month. Let's take a closer look at the U.S. and let's start looking at some numbers to just set the stage for today's conversation because there is a There is a context here. This is the chart that we looked at a month ago, where the blue line represents, and on the scale on the left, the the number of cases cumulatively in the U.S. And as you can see, what the trick of this one is is that I blocked it off for the date that we had in early June where we last met on this topic. Here's the early weeks of the pandemic on the right scale, the percentage daily jumps. This is the point at which we were getting 20 to 30% compounding daily in the number of cases. That's the exponential growth phase that we saw, and then as a term— terms of percentage, it was of course a declining number as the cumulative cases rose. What has happened since, since then? Well, you know what's happened since then. The cumulative cases have continued to grow, and in fact The growth rate has increased. We see the blue curve tipping upward again, and in fact, even in percentage terms, we're seeing an increase in the number of new cases on a daily basis. This is perhaps the more revealing chart. This is what it looked like in early June. Now we have the total, the total cases on a log scale. Notice the exponential growth, straight line when we look at day-to-day percentage, constant percentage change. It flattened out, and that corresponds to us reaching a peak of cases in April and a very long, slow, hard fight to get the daily case rate down to around 20,000. A lot of noisy data. First lesson about analytics here is look at averages. The way that the data was collected and reported was very lumpy, to use the technical term. It was chunky. aggregated from a number of different sources with irregular timing and large and small. So look, always look at averages. And in fact, there was a weekly variation. I flagged the Fridays in the dark triangles and the Sundays in the yellow circles because those are typically the largest and the smallest days each week. Looking at these moving averages, a long hard fight from April through May into early June. That's where we were when we last met. What's happened since then? Wow, our last session was literally the best that we got it, and it has been rapid growth. Notice the slope of this, the increase in the number of new cases each day. The slope is a very familiar one. It's a very rapidly increasing number of cases where we're now in the 50,000 to 60,000 range. I've had to readjust my right scale several times in recent weeks. So we've had this, we've had this explosion, and so it's a good time for us to revisit another question that we asked last time. I'm going to go back to Menti in a moment, and I'm going to serve up a new question, and it's going to have to do with the same question I asked last time, the impact that COVID has had on you. Let me switch back over to That to, to Menti. You should be seeing Menti, and you should now have a new question. 3 parts: the impact of the COVID virus— has it hit close to you, your friends, and your family? Has it significantly reduced the daily quality of life? Has it materially harmed your company and organization? So we've got 3 sub-questions there. Please enter that in, and we will look at the live results here as we, as we go.

And Patrick, while we're waiting for those results to come in, we did get a quick question. Can you repeat the source of the data from the graphs you were showing in your slides?

Yes, there's 2 things, and they're from the same source. The data that I've shown so far is from Johns Hopkins University. They have a coronavirus dashboard, which is one of many excellent sources of data where they aggregate data from around the world, various public health departments, counties, and states in the US and countries around the world. And then the charts where I was showing the kind of the peekaboo, here it was in June and now it is in July, those are Excel spreadsheets that I've built, but the source data for those is that Johns Hopkins dashboard.

Patrick, the indicators on Friday and Sunday being the largest numbers and the smallest numbers, is that a factor of when the data posts or when people are actually taking the test?

It's not clear because it's a— that kind of aggregate data is a kind of a sausage-making process because they're aggregating it from so many disparate types of sources. There's not a consistent way of capturing, capturing, validating, reporting, and so forth. And so it's very, it's very dangerous to rely on any single day's numbers because we just, we don't really know who reported when. It's just what's turned out just by observing it after a couple of months, we could see that there was this pattern about, and it could be that the largest public health departments have a weekly reporting schedule that happens to drop numbers on Friday. We don't really know what the source of that is. So again, I think what we do is look at the averages. I went to a 7-day moving average so that every time a new Friday would come on, the previous Friday would drop off. I also use something that some of you may know called exponential smoothing that puts the most weight on the most recent data points. Even then, when you have a sharp upward trend, those averages are going to lag behind what happens. But rather than trying to explain or dig into the cause of that, what we're trying to do is blur our eyes a little bit and make sure that we don't get indexed too much on what's basically some kind of noise of unknown source. Well, I don't have the summary from last time. We'll have to take a look at that later, but it looks like my recollection is that the average amount of impact is up from June in terms of the people who have actually experienced it themselves and their family. I know that I would answer that differently since then. I've had a member of my family recover from COVID I've got a good business school classmate of mine who's suffering terribly from it at the moment. Medically seems to be doing pretty well, but it's not fun. And a number of other friends who have recovered from it. And I think the material impact on daily life and companies also edging up a bit. It's a very different kind of a phase and a different place than we thought we would be a month ago where we are today. Well, thank you for sharing that information. That's some good context for us to think about. Who's here? We've got a wide variety of people in business mostly, and this is a very real thing to us. It seems to be getting more real. Here's a topic that I mentioned last time. I'm going to just hit it very briefly today. There is a little bit of context about this that's important in understanding what's happening, particularly in the US, and that has to do with whether this pandemic was out of the blue, and the shorter answer— the short answer is no, it most definitely was not. For some completely different reasons, partly professional interest— my field is decision and risk analysis, of course— but for other reasons, it has to do with writing a novel, have been researching for years the various kinds of existential threats to civilization. And time after time, we find that pandemic is something that's mentioned. It's been talked about as one of the most predictable of the unpredictable, events that can take place. It's just a question of exactly what and when and where, but the if has never been a question. And we know the reason for this and the dynamic of this pandemic. This is some maps that I showed last time. Here's a map from about 100 years ago that talks about the travel time. Notice how long it takes to get— and this is a London-centric map— how long it takes to get from London to other parts of the world. You literally were counting things in tens and twenties of days, scores of days. And the World Bank made a report last year that said— they had this astonishing statement that 90% of the surface of the planet is now within reach of any large city within 48 hours. And so the nature of infectious diseases is that they exist, they will spread, and the ability for them to spread globally in a remarkably short period of time before we even know it has never been greater. And honestly, I think that's going to be a part of our lives going forward. As long as we have global trade and global travel as we do, pandemics are going to be the most predictable of global crises, and they will keep coming. So thinking ahead about what do we do now, thinking about how do we prepare our— not just our businesses, but our societies to be able to be resilient to these kinds of things. Which will recur. There's this idea about the models. I've heard a lot of times the models were wrong, the models misled us. I want to push back on that a little bit by just explaining how I think about what models are. I'm not an epidemiologist, but I do create models for a living, mostly in the business context, and so I think there's some things that those of you who've studied analytics in business and myself, we can make some sense of what's going on. So when I see some things like this, this is one from yesterday, a news article predicting 200,000 deaths based on the current scenario, but if we wear masks, it'll be down to 162,000. There was another forecast, it would be at 175,000 if we wear masks, but it would fall to one— I'm sorry, without masks, 150,000 with masks. And then how the modeling works, there's a little bit about the assumptions. What assumptions are they making? They're anticipating the reimposition of strong Social distancing mandates when the deaths per day reach 8 per 1 million versus no preventative action. So this article actually gives us some good information, and it's the kind of thing that we want to be looking for. Every number by itself is just kind of a hanging number. We want to think about compared to what. This gives us a double comparison. It's telling us, predicting if we do this, We'd have this many deaths compared to if we wear masks, a different one. And it gives us another study to give it a little some context. And it peels back what are some of the assumptions. There are probably many. What are some of the assumptions to help explain what these things are about? Now there'll be a slide in a little bit that will come back to this, and I'll ask you, I'll give you a little quiz about, about this article. But this is the sort of stuff that we see all the time. What is going on here? It comes down to this. A very wise statistician once said, all models are wrong, but some are useful. Models are wrong. They're intentional simplifications of the world designed to help us get insights about particular things. And so the issue— Michael Steele, J. Michael Steele, also said this in this way— the issue is, what is the model intended to do, and what is it not intended to do? So understanding why a model exists, What assumptions it makes and what it's trying to provide insight is important to knowing whether or not we're going to be using it properly and how to interpret it. So this statistician says he feels ripped off only if, you know, if a roadmap, for example, doesn't correctly answer the questions that it claims to answer. So we want to be very careful about interpreting these things. And to demystify this, you know, they look like a bunch of numbers, and if we were to look at the mechanics of it, There's a lot of switches and dials. Let's pull off the COVID and understand what's going on a little bit inside some of these things so that we can think more critically about it. First of all, there are some models that are, in this particular situation, generally inappropriate. And that would be purely statistical models where you're just trying to look at the data and try to find some kind of internal dynamic from the data as to what's about to happen next. And in the study of data analysis, we always say you have to have some kind of a context in order to be able to interpret it. Interpreting a mystery set of numbers, what happens next, is much harder to do well than if you know, for example, that that happens to be production of steel. Why? Because you can think about the other things that might have an impact or might have had an impact on the production and consumption of steel. And you might be able to have good solid predictions about those dynamics, those factors that are driving what we're observing. And so understanding the— I call it the physics of the problem— is important. So if we're just doing just raw extrapolations from data, we get all sorts of crazy things. This is one that I showed last time where we look at the number of blades in a razor, and depending on what kind of assumption you make, you can get sort of crazy extrapolations of how many blades there would be on a razor by 2020, and it's all nonsense. Just to give you an example of how this works, let me go back to my own model for a second. Very famously— there we go, you should be seeing Excel at the moment, and this is the chart we were looking at earlier. Very famously, a few weeks back, somebody predicted that the number of cases would go down to zero soon because looking at the first part of the data, they fit the data from March till about middle of May, and they just used an Excel trendline to fit a cubic function. Why cubic? Well, they couldn't explain that. It just fit the data well, and the cubic function had it going back down to 0. Well, we can play that same game here, and notice the problem we have. I'm gonna select the data, and I'm going to— Insert in the chart. I'm going to insert a trend line and insert the trend line. And my trend line, if I do a straight line, you should now see a straight line. Well, that doesn't appear to fit very well. We could look at the trend line and we could say, give me the equation, give me what's called the R-squared. That's the goodness of fit. And most of you remember from regression or time series, that doesn't fit well, but I can, without any— without thinking, I could go to a polynomial. I can go to something that has a little curve to it. I can make the order of that go to a— here's a cubic. The R squared goes up. If I keep going up, there's a cubic, there's a quartic, 4th power, the R squared goes up. And it's going to say— and it says, oh well, it's going to go through the roof. But if I go to an even better fitting one, a 5th order polynomial, all of a sudden it, for some reason, It decides to curve downward. If I go to a 6th-order polynomial, it goes up again. The point of that is that purely statistical extrapolations of time series data without understanding what's going on inside that's driving the data is going to give us nonsense. So when you look at a model, particularly with predictions, you want to understand the assumptions that went into it. A general framework for thinking about models is we're looking at some outputs that derive from some calculations, but there's a host of inputs, and those are key. There's some hard data that they use, there's a lot of assumptions, there's also some decision variables, policy variables. What kind of assumptions are they making about what people do or what governments do in response to that? And peeling back and understanding what is inside these models, that what logical relationships are they building into it, as well as what assumptions about how those things work, are critical. Otherwise, you can't really tell what's going on. So in epidemiology, now this is where I'm out of my lane, and I've seen some of these models, but I'm not an expert in the art of building a good one. But let's evaluate it. We can understand it in the context of business modeling, The simplest models is you've got a susceptible population and some of them become infectious, and they've got these— there's some parameters that describe the rate at which the susceptible population becomes infected and the rate at which they stop being infectious and come back. And more and more elaborate models have more elaborate different states or different pools of people in them. And so these epidemiological models are taking a look at subpopulations and really thinking about the way in which the virus is transmitted. They're thinking about surface contacts. They're thinking about vapor droplets. They're thinking about things being airborne. They're thinking about all the different ways in which the mechanisms by which it can be carried, and they've got to try to estimate what those parameters are, what those rates are, and build sort of the physics of the model. They also need to know If somebody's infectious, how long are they infectious? How long are they infectious before they show any symptoms, if at all? How long do they remain infectious? Once they're back to the susceptible pool and they're not with the disease anymore, do they still— are they still susceptible, or do they have some sort of enduring antibody protection? At the start of a pandemic, we don't know any of those things. And that's the biggest problem and the biggest challenge we have at individual level all the way up to the government level. We just don't know. One way of thinking about it, again, the layperson's view of it, is that viruses have a number of different genetic components that we might think of as the wheels in a slot machine. And the mutations that take place, this is true of colds and influenza as well as corona, The mutations that we see spin the wheels. In some of these wheels, some of these genes affect its ability to bind to various kinds of surfaces. Some of them have to do with how durable the protein coating is. Some of them have to do with how aggressively it can take over cells. They have all the different parameters of how the virus functions are unknown to us. and they impact all of those parameters. When we get a novel virus, as we have here, we don't know, and so the virus detectives, the epidemiologists who are trying to understand this, are working very rapidly to understand, is this a killer virus or not? Is it one that's highly transmissible or not? Can we build immunities or not? We know nothing at first, and the key is to start building models simply. That's the way we all build models, with simple models to get some kind of insights, But recognize the limitations and do everything we can to try to update using data to estimate what these parameters might actually be. Well, I showed you an article a few minutes ago from a press— some press about a model that just came out yesterday, and it had some numbers in it, and it had a lot of other caveats about assuming this, assuming this, assuming that. Here's one of the problems that Where we contribute to this. Famous Far Side cartoon, one of my favorites. What we say to dogs: Okay, Ginger, I've had it. You stay out of the garbage. You understand, Ginger? Stay out of the garbage or else. That's what we say to dogs. What the dogs hear: Blah blah blah blah blah, Ginger. Blah blah blah blah blah, Ginger. Blah blah blah. I often use this in the context of what we say to clients and what clients hear. What we say to our bosses, what bosses hear. You can caveat things to death, but what tends to stand out? What, what if we had a chance to quiz everybody, what would stand out from that article I showed you? The numbers.

Mm-hmm.

Numbers are very, very, very sticky. You can never call back a number. And once that number's out there, it's stuck in people's mind. So preliminary numbers, preliminary model estimates are very dangerous, and they come back to bite us all the time. Because we may have said all the right things about here's the assumptions we made, and this is the kind of things that would need to take place for this to be true. And on the other hand, this might be true compared— but a lot of that stuff just in one ear out the other. You told me that there were going to be 10 million cases. Well, here's my own model from a few months back. If we looked at what was happening, this is the data, if we looked at What was happening in March where we were getting exponential growth, right, a straight line on a log curve. If we looked at things right here on March 24th, as I did, you say, if we continue at this rate, we will have 10 million cases by some date. Well, the if is everything. The if is everything. What you're trying to explain is, this is what exponential growth means, Folks, is that we, we strip out all of our healthcare capacity in X number of weeks if we don't change, if we continue at these rates. It's like saying, if we don't turn our wheel in our car, we're going to hit that tree. And when you turn the wheel, you don't hit the tree. The model wasn't wrong. The model— well, we can't really tell whether you would have hit the tree or not. But when you look at models, think about what are they trying to represent, what assumptions have they made, and what was the purpose? There's a— for those of you who are model geeks, by the way, there's a lot of interesting work being done with what we call system dynamics models. We might see these things in a business context where we think about lots of different decisions and quantities and these feedback loops that connect them and interconnect them. Those are very difficult things to build and to calibrate. But in order to capture complex dynamics, you need them. So for example, here is a system dynamics model. For COVID-19, notice what happens over here. Here are the transmission events. It has to do with the number of people, but also it has to do with— wow, lots of different factors. The number of people who are infectious, the amount of environmental contamination drives the risk of transmission, but there's also these other, other loops going on, right? Public performance of protective behaviors can drive the risk. They're driven by the effectiveness of public health communication, which is driven by trust in authorities, which is driven by accountability. All of these different factors having to do with the various actors' and agents' behavior and how they influence each other drives the dynamics, and that's why we can end up with some very complex kind of dynamic behavior. These are very— dynamic systems are very, very difficult to predict, Because of the interconnection of all these different feedback loops. So that's an interesting topic for you to look at if you're interested in dynamic models. One question that we got before we even started that I had anticipated talking about, it's a question we got last time too, is how do we even keep score? How do we measure what's going on? And the shorter answer to that is, Using multiple metrics. So in addition to always looking at averages and checking the reliability of sources, this is a complex multidimensional kind of a problem that can't be boiled down to a single number. So if you wanted to come up with a single index, you're probably throwing out a lot of valuable information. So for example, just measuring the, the actual public health aspect of this. We can count cases, hospitalizations, deaths, and because this is not— does not appear to be a virus that you get and then you either recover or you do not, it has implications for future health states and health quality. It's important to look at when they're talking about QALYs or DALYs, I'm not sure how the pros pronounce it, but thinking not just about whether there's a death or a recovery, but thinking about how many life years are lost, and not just the number of years, but maybe quality adjusted, because there appears to be— again, I want to stay in my lane— there appears to be long-term health consequences among some people who have this virus. There appears to be organ and other tissue damage that could have longer-term impact. And by just saying, well, you survived it, It's not the same as not having had it at all. And so the metrics are really complicated. And then, of course, we get into the issue of the economics, the costs associated with this, and the costs associated with mitigating it. And the costs are complicated, because what costs incurred when, in what form, and by whom? And so it's important to think about multiple metrics. Particularly given that we have this— a lot of data issues, it's difficult to even tell the trajectory of things by looking at one thing alone. For example, cases is problematic because the definition of a case and what constitutes a confirmed case is not the same in one place or the other, and it's not even the same over time. And so it leads— it might lead one to conclude, incorrectly in my opinion, that that the increase in testing is responsible for the increase in the number of cases. Well, if you look at multiple measures and you look at, for example, hospitalizations, you see them also climbing too, and you think, well, those probably aren't the consequence simply of just more tests. So looking at these things in a multidimensional way is more likely to give you a fuller picture of the complete dynamic and depth of what's going on. Another thing we need to think about when we're thinking about comparing different geographies, different states or different countries, is that there are different approaches that have been used, and we need to think about the economic costs of various kinds of approaches. And so there are some trade-offs between various health outcomes and economic outcomes, and one way to think about it is when you're When people are trying to present comparisons about Italy versus Sweden versus the U.S. versus New Zealand or what have you, dig a little deeper and try to understand at least one metric on the health outcome side and at least one metric on the economic side, because that can permit us to do some efficiency analysis. What I mean by that is looking at this chart, suppose we've got— in this chart, we've got the horizontal axis is a is a reduction in GDP per capita. So that's a cost, and the more to the right, the worse you are, because you actually had to— somehow your society has had to pay. The vertical axis, the higher the number is, the more COVID deaths. Again, substitute whatever metrics you like for health outcomes and economic outcomes. If we were to compare these different countries, we might say, well, you know, C has done very well because it had low deaths per capita, But they may have taken a bigger economic hit. Likewise, we could say A was able to save most of its economy, but at the price of a higher amount of deaths. And so we have, when we look across different geographies, a wide variety of performance. What are the things we want to look for? Well, there's trade-offs made between those two, and there's better and worse ways to do that. There are some countries that are just plain inefficient. So for example, we'd say that D is dominated by B. To use the economic speak, B is Pareto superior to D because it's better in the lower cost and it's better in the health outcomes. And so we would say, you know, we'd rather be B than D. We can't say for sure whether we'd rather be A or B because those involve some trade-offs. Country E, I'm not naming any names, but, you know, this would be one that would be having a big economic impact and high deaths. We definitely don't want to be Country E. We would want to be more on the efficient frontier, or even the most efficient countries are the ones that are able to manage relatively low death rates and low costs. When we think about different kinds of policies and interventions, it's the same kind of thing. We want to think about these trade-offs, and as a public, as a country, we need to be able to have that conversation about what we're willing to trade off. How much are we willing to pay and by whom to be able to get certain kinds of health impacts. How good is the data? Terrible. But in a way, it's almost always terrible. Some general ideas for thinking about data quality are this. First of all, evidence comes in a lot of different strengths. And if you look at site after site after site about how do I evaluate evidence and think critically about evidence, The answer is always, well, it depends on who's doing it and how it's done, but there's a general hierarchy. Expert opinions, I guess, are the lowest that they talk about. Non-expert opinion, we get a lot of that as well, but individual case studies, some cross-sectional studies, randomized trials, and then systematic reviews of multiple randomized trials are the gold standard, but they take a lot of time, and we're doing a lot of science and technology on the fly. And so we're often in the very messy middle where we've got cases and small studies that are done very quickly under field circumstances. But looking at data and looking at the, the way in which the research is done, as opposed to just relying on experts and non-experts, is generally the way to go. Think critically about the entire process of data Analysis. Those of you who have studied this with me know that it's a process. You think about what you're trying to measure, you acquire that data, and you've got choices as to what the metrics are and where to get it. You have to build a database and then do some cleaning and clarifying. A lot of the data caught on the fly is noisy or misunderstood or misinterpreted because it's just of the haste of it. There's a lot of just trying to explain things and just use it descriptively, and then In the form of prediction? These are the kinds of questions I ask myself all the time and that you should ask yourself as well. Thinking about the data: who, what, when, where, why? What's the source? What's the type of evidence? Was this an experiment? Was this just some field data that was collected? Exactly how was it measured and gathered? What was the purpose of the data collection? It can be collected for one purpose and good for that and a few others, but not for others. How was it collected? How was it checked? How was it validated? Is it a complete census of some population, or is it just a subset, just a sample? And if it's a sample, what determined what got in the sample? There's a lot of sampling biases that have to come in when we're doing data on the fly. What are the statistical margins of error? What kind of conclusion is being asserted? Is it being predictive or causal? And I've got an example of that in a moment. How was it aggregated or categorized? Is it being cut down into too small categories, or is it being aggregated so that it's— so as to lose meaning? What kind of expertise was behind the analysis? And the people communicating it, and also what interests do they have? So thinking critically about data is key. I don't have time to recount this story. Some of you may recognize this. Some of you are students of mine quite a while ago might recognize that even the story of Challenger was was one about management and not about technology. And the key issue, which I can talk about some other time, maybe we'll come back, is how to have a good fight about analysis and have a good fight about data. And the punchline of this is that NASA had all the data that they needed in their hands, but they weren't talking about the right issue, and they were distracted by bad communication of the data. It was the wrong kind of chart. There were some some data problems having to do with some misclassified data. Even then, it was not even a close call. Challenger shouldn't have been launched because it was going to be so blooming cold the next day. It was off the charts of all of their data, and the costs and benefits of delay versus a quick launch made it very clear that they should have delayed until it was warmer. When you think about sample data, you've got to worry about margin of error. As I said, remember, I'm not going to go through a statistics class with you here. This is Business Over Breakfast, after all. Remember that many cases the data you're looking at are just a sample from some kind of unknown or unknowable complete population. And so understanding how that sampling takes place, there's this black box, this population that we would really love to see. We would love to know how many people in the, in the US, for example, have antibodies for COVID. We can't measure everybody, but we draw on samples and we try to then go backwards and make inferences about the population. We need to be very careful about that process. We need to particularly understand how that sample was derived, and we need to be careful about margins of error. A couple of quick final points. Does the virus discriminate? There's a lot of interesting analysis about does the virus behave differently in different populations geographically, ethnically, racially, gender, age. Here's what I understand. I understand a lot about the difference between predictive and causal relationships, and some of you may be familiar with correlation is not causality, and that is, that is true. But the problem is these other variables that might be in the population that might be the real causal factors, and And it takes a special kind of analysis to tease that out. It's hard to do on the fly, but here's an example. There's a lot of reporting about racial disparities in outcomes in the US and differences in the way that, say, white and Black populations within the US are impacted by the virus. In this recent article by Graham Wood, he says Black people die from COVID-19 at a higher rate than white people do. Recognizing that we've got a very diverse population who's exposed to this. And Wood's point is that the virus knows no race or nationality. Sure, it can't peek at your driver's license, but what he says is society checks for it and provides the discrimination on the virus's behalf. That's a very powerful statement. We just unpack it a little bit. I haven't looked at the source data, but here's what I believe the analysts are saying about this. Is that you can look at the data in an aggregate way. For example, overall population, you can look at by race. But if you aggregate in the wrong way, you can confuse things. You can end up messing up some of the variables that really matter. In this particular case, here's what I understand to be the case. So every statement I make on this slide is, here's what seems to be the case based on my reading as a non-medical professional, but trying to relate this to our business understanding of causality, is that one, that the death rate for COVID does vary by age, and it's particularly— it's more lethal for people over 70. Okay, I have my 65th birthday coming up in 2 days. I've already got my Medicare card. I am actually, you know, at elevated risk. I'm not at 70 yet, but I'm taking extra precautions because I'm aware of the what seems to be differential age-related risks. So that's true, but here's what also seems to be true, is that the distribution of age within the population varies by race. And so it turns out that there are, in the 70-plus people in the U.S., the white population has a greater percentage. There's a higher survival rate to those ages, and so the people in that age group who are more susceptible to death by COVID is different from the general population, and what that means is that the death rate for whites is high or higher overall than the death rate for Blacks, but the death rate for whites in every age group is lower than that for Blacks. How can that be? This is something called Simpson's paradox. Not that Simpson, but you know you look at these results and you want to go dope because how you aggregate the data influences what conclusions might appear to be reasonable. And so what we see is if we look age group by age group, that the the survival rate, the survival rate is lower for Blacks than it is for whites in every age group. But because the mix of cases is skewed, right where there are more white people in the the victim group in the over-70 group, and that group has a higher death rate overall, that it skews the overall results. So we have to be very, very careful. So again, I haven't looked at the source data yet. I put that asterisk on this analysis, but I understand this argument and believe that it's a valid argument that we have to be very careful when we look at aggregated data. And particularly, we have to be very careful when people are making causal statements About what causes higher or lower survival. We really have to look very carefully at the aggregation and what the various causal factors might be. And in particular, we know there's something about age. So try to look across within different age groups. If we think about other gender effects, look within age groups. Are there race effects? Are there income effects? We need to be very, very careful about that. Well, I'm going to flip over to question mode here. I was going to talk a little bit about false positives, false negatives, Bayes' rule and stuff. But again, it's It's breakfast time, and maybe we'll come back and talk about this again. We'll see how things go in the next month or two. I'd like to turn things over to, to you guys, and I have one question for you, one final mentee, and then I'll answer some questions on the fly. Your questions— your prediction about daily life. We've got a large and wise and experienced group of people here in the, in the group. So let's do this. I have one final question for you, and that question is this. When do you think your daily life is going to return to a close approximation of pre-pandemic? I'm thinking about ability to socialize, to travel, to shop, to go to school, to have our kids go to school, for us to do our university educations and what have you. Let's, let's get this wise panel of participants what you think is going to be the case. You should be seeing the results now. Looks like you've got that— those results there. The mode currently of this particular group, this non-random sample— we don't know how representative our group is, it's probably very non-representative of the general public— the mode appears to be 12 to 18 months. Fewer than 50% think that it will be Close to normal with less than a year. And we've got more than 25% think it's going to be at least 18 months, 18 to 24, with 10% thinking that it may be 2 years or possibly never. I'm going to take off my expert hat now and for a moment and tell you my opinion. My wife and I are mentally, psychologically, and every and every other way preparing ourselves to be operating in life as we currently are for at least a year. We'll see. My personal opinion is that it's going to depend a lot on the development of vaccines. The studies that I see seem to indicate— preliminary studies, right, and models— seem to indicate that in places like Spain that have had very Rough bounce with the, the pandemic, the percentage of the population who have antibodies is still quite low. And so this idea of herd immunity, many of the professionals are starting to conclude— asterisk, you know, early on— that until we get a vaccine, we will really not be able to successfully put the control rods back in. So that takes a long time. Crash effort, big uncertainty. Could be in a few months, could be years. Build contingencies into your life, be careful, and listen to the experts. Let me stop there. We've got a bunch of questions that have come in. I've answered a few of them, I think, along the way, but, but let's just see what else we've got. Christine, we've got some other questions I might be able to address in our remaining time.

Yes, thank you, Patrick. As we talk about Children going back to school. Is there a model you recommend when analyzing the data in school-aged children considering the reopening of schools, and what factors should we focus on— transmission, testing, death, recovery timetables?

My kids, my kids are, are well beyond school age, and I don't have any grandkids yet, so that to To me, that's an issue that I haven't had the need to look at very carefully. But the way I think about it is, again, is we've got these tightly coupled systems, our society, where everything's connected to everything else. And if it were possible for children to go to school and be isolated amongst themselves, it's a relatively low-risk group in terms of In terms of death, we still don't know the long-term health impacts. And, you know, the long— a long-term health impact for somebody starting at age 9 is very different than somebody starting at 65, by the way. Um, You know, that might be one thing, but, but there's a complete ecosystem of, first of all, the teachers and administrators and maintenance people and other people who support, support education and who transport people. And I would look really carefully at more holistic analyses. I think the simplistic— to me, again, I'm just looking at it as an observer. I think the simplistic analyses are the ones that somehow imagine that the children appear in a classroom, somehow educate themselves, and then come home and worry only about student-to-student transmission. I think that bringing a lot of people in close contact for those periods of time is very, very risky, and because of the interaction with others, I would be pretty skeptical of— I would want to look very hard at those analyses. Again, that's just an opinion from an outside observer on that particular issue.

Thank you. Given that confirmed cases are Largely indicative of people who have experienced symptoms and have been tested, and that there's been a significant incidence of false negatives. Is there any way to chart an approximation of total COVID cases, symptomatic as well as asymptomatic, using random sampling?

I have seen— I don't have a citation right now, but I know they're pretty easy to find. That's an important question. For a lot of reasons, policymaking particularly, how many people have actually had this? Because for the reason that I mentioned earlier, trying to understand if there are a lot of people who are not just pre-symptomatic, but for the duration they are asymptomatic and just don't even know that they've had it. Those numbers, if those numbers are not appearing in the confirmed case totals, Then, then a couple of our parameters are off. The, the, the death rate is lower than we think, because the ratio of, you know, deaths to cases is— the denominator's off. But also the, the ability to transmit it is much higher, right? So we, you know, it's, it's, it's, it's easier to catch, lower death rate if there's a bunch of asymptomatic people. Trying to figure out how many people have had it and what kind of herd immunity we're building, what's it going to take to build up enough natural control rods for transmission is a very important question. So yes, there are a lot of people tackling that subject right now. It's live science, but it's a— it's an easy thing to track. And if you want to look at things like estimating true prevalence— prevalence is the word that they use, I believe, to describe what percentage of a population has a virus or a particular condition, and looking for, you know, true estimates of herd immunity. The live science on those terms is crackling with activity, and I'd recommend going there. I don't have an opinion about— because I'm not an epidemiologist— I don't have an opinion about what the right number is.

I think we have time for one more question. Can you comment on some of the differences between US and Canada's approach and results, is it an unfair comparison given Canada's size is roughly 10% that of the US, or is it a fair comparison given differences in the healthcare systems and government policies?

Again, partly out of my lane because I'm not an expert in health policy, but from an analytical perspective, what I've mentioned is the kinds of things that I would want to look at. I'd want to look at things on a per capita basis. And so I'd want to look at a variety of different health outcome measures on a per capita basis, so controlling for size, and also some of the economic types of metrics, looking at a number of them, also controlling for size. It becomes difficult to comment on these things or to draw conclusions without providing additional context, because the data is not the only thing that we know about these. We observe leadership, we observe policies, we observe on the streets, we observe how the populace behaves in these situations, and we know that there are also— we know that there are some major differences in those kinds of behaviors, as well as the existing, the pre-existing condition, pun intended, of the healthcare delivery system from one country to the next. And so I guess the final point that I'd want to make about this is that a good analyst is going to pay good attention to the data and be using the data the best we can to make sense of these things. But it's never the only thing that we have to make sense of things. We have context. We know a lot of other things about our societies. Apply the same kind of care and attention and caution about biases to those things as well. But the data, you know, we let the data do the talking, but we— and I taught that for many years— but at the end of the day, it's we as humans who do the deciding. It's our judgment. And the point of the models and using the data is to improve our judgment, to give us insights about what's going on so that we can make responsible decisions. And so it's not only fair, I think it's essential that we use our other human skills, which is to understand, you know, the character and the capabilities of the people and the institutions that we see. And in that, I think the countries vary greatly. And if you look in terms of the data, as well as the approaches and the nature of healthcare and so forth, the United States is an outlier. And I'll leave it for you to draw the conclusions With the data as well as everything else that you know. All right, well, thank you all for your time this morning. It's wonderful to, to see you, and I hope that we'll be back again sometime. Again, be careful, look at the data, listen to the experts, take care of yourself and your family. Collective action is the opportunity for us to continue to fight this pandemic and get back to regular life sooner rather than later. Thank you.

Thank you so much, Patrick, and thank you for all of that wonderful data that you shared with us. Us this morning. We want to thank you all, our wonderful participants that come back every Thursday to join us for the Business Over Breakfast webinar series. A copy of this recording for today's session will become available on the Emory Executive Education LinkedIn page. You can find the link for that inside of the chat box there. We do have a couple of things that we want to share with you that we have going on inside Emory Executive Education. Next Thursday, July 15th, we'll be hosting a quick information session if you want to learn more about the different offerings that we have in executive education and how Emory Executive Education can partner with you and your career goals. That'll be next Thursday, July 15th. If you want to join us over lunch from 12 to 12:30, the registration link is located in the chat box. Additionally, as Nicola mentioned, Patrick will be returning for a short course beginning August 19th, Approach Problems Like a Consultant. With his colleague and other faculty member at Goizueta, Lynn Siegel. And next week for our Business Over Breakfast webinar series, we'll have Goizueta faculty Molly Epstein joining us for intergenerational communication. So we hope we'll see you all next Thursday at 9 AM for that. In just a moment, you're going to see a brief survey pop up on your screen. We ask that you just take a couple of minutes to give us some feedback on today's session. Again, thank you all so much for joining in every Thursday morning at 9, and we look forward to seeing you next week. Have a wonderful rest of your day, everyone. Thank you.