A casual conversation concerning causal inference

The big picture with causal inference, essentially it's exactly as it sounds. I guess I'm going to use the word and the definition, which I know you're not supposed to do, but if you're trying to answer a causal question, so does something cause something else? Which often most of the questions we're interested in are in that framework, although it is distinct from something like prediction, which would be a different kind of space. But in the inference space, when we're trying to determine the relationship between different factors or different variables, a lot of times we talk about associations as a way to sort of describe relationships that we know maybe are correlated, but we don't want to go as far as to say they're causal. But I would argue in most cases the human instinct is to want to talk about things causally. That's most of the time when we're studying relationships between variables, it's often because we want to know if there's a causal connection. So in randomized trials like the ones that we're talking about with Pfizer and Moderna looking at different vaccines, they're not just interested in whether or not getting a vaccine is somehow related to whether or not you get Covid. They want to know if getting the vaccine will actually cause you to not get Covid. So this is sort of the relationship that we are focused on in causing inference. Bandwidth for Changelog is provided by Fastly.

Learn more@fastly.com Our feature flags are powered by LaunchDarkly. Check them out@launchdarkly.com and we're hosted on Lino cloud servers.

Get a hundred dollars in Hosting credit@linode.com.

Changelog what up friends? You might not be aware, but we've been partnered with Linode since 2016.

That's a long time ago.

Way back when we first launched our open source platform that you now see@changelog.com Linode was there.

Help us and we are so grateful.

Fast forward several years now and Linode.

Is still in our corner behind the.

Scenes, helping us to ensure we're running.

On the very best cloud infrastructure out there. We trust Linode.

They keep it fast and they keep it simple.

Get $100 in free credit@linode.com changelog Again.

$100 in free credit@linide.com changelog.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changeaw.com community and follow us on Twitter. We're RacticalAI FM.

Foreign.

Welcome to another episode of Practical AI this is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined, as always, by my co host, Chris Benson, who is a principal emerging technology strategist at Lockheed Martin. How you doing, Chris?

I am doing very well, Daniel. How are you today?

I can't complain. No complaints from this end. I had a good, restful weekend. As listeners might remember, I recently downloaded the new Tony Hawk Pro Skater for Xbox and that's been filling my waking moments as I try to increase my combo scores to ridiculous numbers.

I'm so jealous. You had a much better weekend than me. I had a weekend full of honey do lists and stuff. I mean, I was exhausted and happy to return to work on Monday morning.

Oh, I've got a pretty cool wife. She. She played with me. So there are things on my honey do list. But it was okay because we were, you know, we were spending that quality Tony Hawk time together. So.

Yeah, I have like. I kept hammering my thumb and things like that. It's just terrible. It was, yeah, horrible.

That's rough, man.

I'm just wounded. I'm the walking wounded now, man. That's terrible. Anyway, go ahead. You keep talking.

Yeah, well, I won't, you know, go into the. The cause of you hitting your thumb with the. With a hammer, but we are going to talk a about causes today and causal thinking today. We're really lucky to have with us Lucy D' Agostino McGowan and she is an assistant professor in statistics at Wake Forest University. She's also very involved in the R conference and giving a talk at the upcoming Government and Public Sector R conference, which actually Practical AI is a media sponsor of. So. Welcome, Lucy. It's really great to have you here with us.

Thanks. I'm excited to be here.

Yeah. It's wonderful to collaborate with you on this. We should say, too, that you are also the co host of a podcast. I think you're the co host. There's another host of podcast, Ellie Murray and I. Yes. Of causal inference.

So casual. Casual.

Casual inference. I'm sorry, I'm getting mixed up with our topic for the day. Good call.

That's right.

Casual inference. Yes, yes.

It's a pun. We gotta.

Yeah, all good. I kept trying to pitch pun names for our podcast and none of them made it through the filter. So I don't know. I don't know what the reason is for that, but I'm all for pun names.

So they just weren't very punny.

Oh.

Yeah, I love a good pun. We have some segments that have pun names as well. It's a good. A good time.

Yeah, yeah, yeah, for sure. So we will put a link to that in our show notes, definitely. I encourage our listeners to check out the podcast and see what they have over there. Some really good content. But before we jump into all of the great things that you're involved with and that you're passionate about in terms of data science and R and causal inference and casual inference, if you could just let us know a little bit about your background, how you got interested in data science and R and other things, and how you ended up where you're at right now and what you're working on.

Yeah, so my background is in biostatistics, so I have a PhD in biostatistics from Vanderbilt. So that's essentially statistics for the medical field. But while I was doing the work there, I had an internship with RStudio for six months. And so as part of that, I mean, biostatistics is a pretty applied statistical field, and so there's lots of kind of programming and data science, like, things involved with the training there. But then the software component was something that really interested me kind of further than even what we were doing already, and so I pursued it more. I got heavily involved with R Ladies and did this internship under Jenny Bryan at RStudio, which really kind of built up some of my coding chops and being able to actually do software development and things like that.

And R Ladies, if you could describe.

That, that's what I was going to ask.

Yeah.

Oh, yeah, yeah. So R Ladies is basically an organization, a global organization that there's the global kind of aspect of it, but then they're also. It's mostly run by local meetups in different communities. And the idea is to sort of increase the gender diversity in our community. And so I had started one in Nashville and they exist kind of all over the world and they do really excellent work in trying to sort of increase that diversity. And it's been great. It's been actually really successful. There's some really fun plots that you can see as sort of the first couple. It started in San Francisco and then it really has taken off since then in the past couple years.

Yeah, it's kind of all over the place now, isn't it?

It is, yeah.

Yeah. It's so wonderful.

Yeah, it's definitely international. So that was excellent and really formative for my interest In R, I was interested in R kind of before that, but that really kind of solidified that this was a community that kind of, in addition to being, you know, a language that offers the type of statistical tooling that I'm going to need, it also had that community aspect where there was a lot of support for learning and for developing and things like that. And so that kind of was some of what branched me into, into this interest in sort of data science more broadly. And then I went on to do a postdoc at Johns Hopkins in Jeff Leak's lab. And so my dissertation is very causal inference related and a big arm of what I do is still causal inference. But then through my postdoc I added a bit on data science pedagogy. And so thinking about how to teach data science and how that can integrate in with kind of medical applications in particular. And so I worked on some of that and that sort of branched into this thinking about human data interaction problem, which is another big arm of my research right now, where we think about kind of how people interact with data and how they conduct data analyses and sort of how we can potentially think about interventions to nudge people towards conducting a correct data analysis as opposed to one that maybe would be incorrect in different ways or maybe getting kind of alignment between stakeholders and producers of data analyses and things like that. And so this was all kind of what brought me into the more data science type space.

Yeah, I think that human data interaction is, yeah, it seems to me to be like the really the tough problems that data scientists deal with kind of live at that boundary with the tooling. Now there's so much great tooling around and it's fairly, I mean, of course, you know, you need some context and some background knowledge and some domain knowledge and all that to solve problems. But. And tooling wise, it's fairly easy to actually create somewhat sophisticated systems. But that communication part and that interaction, the human data interaction all is for naught if that's not properly taken into account and managed.

Right.

It's such a big deal too. As you were talking about that a moment ago. Every single day in my job I run into issues where there are challenges between how people are interacting with data and stuff. So I'm very excited to hear what you have to teach us in the minutes ahead.

Yeah, it's a huge, it's kind of neat because it builds on, I think, the kind of foundation for this field of human data interaction. Kind of builds on what people would talk about before they were talking about human computer Interaction. And I feel like in some ways we have mastered that in a lot of ways. There's still a lot to learn, but I think that was happening a lot when things like Apple were coming about and Google and sort of thinking about how we can help people interact with computing systems to make it that they're. For some people, using a terminal is a great way to interact. And I personally love interacting via a terminal. But to get the Internet and everything into the hands of the average consumer, someone had to figure out that, no, we need to have something that has pictures and words and people can click things and they don't have to be just interacting via this text module. That really launched a huge revolution, I think, when we were able to bring what computing had to offer to the average person. And I think that this human data interaction is sort of the next step on that. So now we've got the computing and we know how to get that into the hands of people who, you know, maybe otherwise wouldn't have been able to implement these different things. But making sure that now that they can do it, that they're kind of doing it correctly, they're interacting with the data. The way that is going to get to kind of a result. That is correct and also aligns with the incentives of everybody involved, I think is really crucial.

Yeah. And I think it's not only that's deeper sort of data interaction, but there's a lot of communication about data now in the media as well. And how people perceive that is very different depending on who you talk to. I know even in the last week, I think it's happened all in one week. There was two or three companies that announced these statistics about how good their Covid vaccines are going to be. This happened days ago, or maybe it was even. I forget if it was it this morning that the last one came out or yesterday or something.

It's yesterday.

Yesterday.

Okay.

Yeah, yeah.

95, I think was the thing they were throwing around in the news. 95%.

Yeah. So I think one was 95% and the way it was communicated on the other one that I noticed was like 90% or higher or something like that.

Above 90%?

Above 90%. Okay. Yeah.

And just as a real quick clarification for listeners that we, at the point where we're recording, we just heard the first two vaccine initial results and those, and everyone's talking about that. So you're probably beyond us and you know more than we do by the time you hear this. Yeah.

This is late November 2020.

There you go.

Yeah.

When you hear that sort of communication come out, do you get the sense that people are consuming that in a way that they are understanding the implications of those types of numbers or what is your thought on that?

Yeah. Oh, this is such a good question. And it actually, like, it bridges essentially all of my research interests.

Perfect.

Yeah. Because we're talking about. So the Pfizer and Moderna, I think, are the two big ones that came out relatively recently. And these were both randomized controlled trials. And so obviously the question of interest is a causal question. It definitely covers my causal inference kind of framing. And then we also. It covers the human data interaction and also communication of statistics, statistical kind of concepts. And so it bridges it really nicely. But one thing to mention in all of this. Well, first, I think you said that it was 95% for the. For Moderna, which is what a lot of outlets are reporting. The actual interim analysis reported 94.5 and the round. I know also what's very interesting about this is that there's been, you know, just from a communication perspective, even that has been a little bit controversial because they're giving a sense of precision where there really isn't that kind of precision. So the number of participants in this trial that received the placebo, that got sick, it was. There were. I think there were 90 that received the placebo and five that received the treatment that ended up getting Covid. And so 94.5 implies that there's this real precision. But we actually don't even have 100 participants that have reached that end point of getting.

Right. It makes you think that there's like tens of thousands and they were able to get like really, really granular data.

Right, right. Yeah. And giving. I mean, I'm sure that there were. Well, I shouldn't say. I'm sure. I don't know, maybe there wasn't someone that thought about if they said 94.5, that everyone would round up to 95 and then that this would sort of give some kind of sense of things that maybe wasn't exactly what was being represented by the data. And the other thing with these that's really challenging to make sure is well understood is that both, in both cases, these are interim analyses, and so they're not the final result. And we wouldn't, as, you know, as a statistician involved on data safety monitoring boards or these different groups that actually come out with these estimates. First of all, it's not common. I mean, in other circumstances, often interim analyses don't get the same kind of press that these are getting. I mean, sometimes they will get a press release and they will see a little bit of a market change, but only for people who are really, really paying attention. And of course, now it's very different because the whole world is paying attention. And so kind of how we communicate, it has to be even more clear. But, you know, interim. And it's not unusual to get a different result in an interim analysis as a final analysis. You know, certainly you would expect, especially in both cases, because they're closer to the kind of what their final endpoint is, then they're closer to that than they are far away. So this isn't like a first look. It's, you know, they're closer to what the final result likely will be. So probably it's going to be in the ballpark of what we're seeing, but it probably won't be exactly 94.5%, for example. It would be really surprising if it was. And some of the concern is when those kind of precise numbers are reported in the news and then it comes out that, oh, in the end it was 80% effective, which is actually really great, people kind of can start getting concerned.

Yeah.

And then they can kind of come up with a storyline of their own and one that maybe they're not comfortable with taking the vaccine for whatever reasons because these people can't get their numbers right. Or, you know, whatever it is. Yeah. Which could cause actual health and safety issues.

So, having said all that, it raises kind of an ethical question in my mind, at least to, you know, clearly this is done for, you know, kind of the marketing benefit. You know, the first one comes out, the second one gets out there to persuade the public. And yet, you know, as you just instructed us, it might reasonably fall to 80, hypothetically. But that does change the way people are thinking about it. And so, you know, any thoughts on using data in that way? I mean, there's a certain amount of manipulation potentially involved in that.

Yeah, for sure. And it's really challenging to know what the right thing to do is because, I mean, I think one thing that can certainly help, and I know some of the reporting that I've read has been, you know, they've been careful to include in their text. Like I was reading the New York Times piece on Moderna yesterday, and, you know, they did include that this is an interim analysis and we expect these results to change. But unfortunately, and, you know, maybe to no fault of the author, but that ends up being like paragraph four, and, you know, the headline says something very different. And people tend not to read past headlines. And so it's kind of hard to figure out the right balance of getting people to be able to consume this information. Because it is good to sort of have people see a light at the end of the tunnel. I think there's definite benefit.

There's a need for optimism.

Yeah, yes, there's definite benefit to that. But then being able to balance it with the reality and kind of being able to articulate the uncertainty in such a way that when the numbers slightly change it doesn't cause people to sort of lose trust. And I don't know the perfect answer to this. This has been something that my lab has been studying. I have a student that's been running a study that's been looking at kind of if we randomize people to see kind of health recommendations qualified with uncertainty versus ones that are sort of said definitively. When a follow up recommendation is made that maybe reverses the previous one but gives kind of the full context, are they more or less likely to trust it if they saw things with kind of the proper uncertainty the first time or kind of with just like a clear certain statement? And it turns out we're still in the process of analyzing this data. But it's challenging because the first result when you just look at people randomized to seeing a certain versus kind of qualified recommendation on average, people do tend to prefer the certain recommendation. Like they just want to be told what to do. They don't want to be told the like, you know, roundabout. This came from a small study and we aren't actually quite sure. And this is the best that we know right now. But we're going to be taking in more information as it comes. So if you knew for sure that the recommendation or the numbers wouldn't change, that actually is the easier way to go to get people to do kind of what would be in their best interest from a public health perspective. But of course we don't know for sure. If we only had a small study, we really don't know that that evidence isn't going to change. And so if you have to think about kind of hedging against that possibility and in the kind of secondary piece, if you've said something with certainty and then you turn back and say the opposite, the lack of public trust I think is a real potential to be lost and that can really be negative. And so sort of thinking about this in the long run as opposed to in the short run is also really important.

So, Lucy, I'm curious. We talked a little bit about these numbers that are thrown around and obviously coming from statistical analysis of some type. And what I was thinking in my mind was there have been those times in my data science career where I felt maybe a little bit uncomfortable with how some of the numbers that I've communicated kind of up the chain to stakeholders have been used. Like, you know, oh, maybe I have this new speech recognition model and for this particular language, like it performs whatever, you know, 5% better than the Google one from. And then, you know, up the chain, maybe it starts to be used in marketing that our speech recognition is 5% better than Google's version. Or there's situations where there's just like, I never intended for my work to be sort of taken in that way. Do you have any general recommendations for data scientists and statisticians and analysts in terms of how they deal with those interactions? Where you've done some analysis and now you're communicating to a different sort of audience and it's maybe going up the chain. Any recommendations or general principles that you think should be kept in mind in that type of situation so we don't.

Have to be hostage to the marketing machine?

This is such a good question. And it's so hard because in your example, the marketing team is at least within your own company, so you have kind of constant communication with them. And often what I see on the medical side at least is like someone will publish something. And actually there are a lot of examples from this, from early kind of COVID data where there were papers that were published in our very top medical journals that some of the reviewers missed things. And I think the scientific community came together and found them pretty quickly and the articles were either updated or retracted. And it worked, but it worked on the scientific side. But the media ran with some of that. And the articles that ran headlines based on the original data often would get much more press than the updated ones, for example. So I think that this problem is not unique to kind of the setting where that you described, where you're trying to kind of help a marketing committee understand the numbers in such a way that they can market them accurately. But it also happens sometimes where it's like not even in your control in the sense that it's like someone outside, totally outside of your organization is the one kind of running with it. I have this talk that I've been working on for a while. I have some aspirations of sticking it in like a short course or something, but basically it's I think of statistical communication and probably just scientific communication in general as on this kind of two by two grid where you have, like, you could imagine this like a XY axis where you have maybe on the Y axis whether or not something is true or not, and on the X axis, whether it's interesting or not. And so you have this quadrant of interesting and true, which is kind of where we want all of our communication to be. So something that kind of is interesting to the marketing folks but also conveys the actual truth. And kind of next to that, you have a quadrant that is not true but still interesting, which sort of falls into what you were describing where you saw that there was a 5% improvement in your speech recognition versus this other one in this one specific case. And the marketing committee kind of ran with it and sort of implied that that was. But exactly generalized it beyond it anyway. So the kind of picture that I try to paint is moving from interesting to. Or from not interesting to interesting is one kind of dimension that we want to move across. But then the other one that sort of describes what you're talking about is moving from not true to true. And sort of what are the pieces there? And so the ones that I've sort of defined as on this, like, journey to truth would be that it has to be mathematically correct, marketed correctly, disseminated correctly, and the audience has to interpret it correctly. And so those are kind of the four main pieces that I think determine whether or not it's true. So that one that you just said there fits in my second category of being marketed correctly. And I guess from my perspective, even just naming that these are the things that need to happen with statistical communication is moving us forward. Because I think that I haven't answered your question on how to get the people to market it correctly, but I've at least named that these are things that can go wrong when you're trying to communicate statistics. And I feel like that's potentially a step in the right direction is just knowing that that could happen. And so being cognizant that even if the thing that you do is statistically correct, you need to make sure when you're passing it on to people and when you yourself are marketing it that you're marketing it kind of in the right way.

Will you take a second and rename those four things? Because you started down and I got caught on the mathematical and I missed the other three. So I want to do.

Chris is always stuck in the map.

I constantly. It's terrible. And then I also wanted to see. I actually want to, like, follow up on Dan's Thing, because I know you're gonna do it anyway. But can you take your grid and kind of give us an example of how to use it? Because selfishly, when we are done recording this podcast and this is gonna go up for release, I want to use this as a tool myself. So you've just given me hope.

Yes.

So can you kind of walk us through, like, how you would use the tool itself, what the four principles are, and then, like, how we can use it day to day so someone can listen to this and then they can go off and actually use it? Because I have the same set of issues. I work for a big company, and there's lots of different audiences, and they may use information in all sorts of different ways like any other company. So I am all ears.

Yes. Okay. So the four that I had mentioned are that it's mathematically correct that it's marketed correctly, so that you have kind of given the correct marketing, that it's disseminated correctly. And so that's slightly different than the marketing. So you could sort of yourself market it correctly. But then the way that it gets kind of disseminated after you've provided that.

Marketing could be kind of delivery outward. You know, the mechanism. Okay.

Yeah, exactly. And then the third part, or the fourth part rather, which I think this is one of the hardest parts, is that the audience interprets it correctly. And the example that I like to give for that is that just because you've done all these other parts, right, if your audience misinterprets your result, then maybe you really need to be thinking about communicating it differently, even if you've actually communicated it in a correct way. And so somewhat relevant example for that. That made some rounds. You know, it's hard to know. I think I'm in an insular Twitter sphere, so it's hard to know how much this made outside of my world. Yes. But in my world, this made kind of a splash that back a while ago, back in July, there was a dashboard that was going around from Georgia where they plotted these figures, a map of the cases per 100,000 people.

Yeah, I'm in Georgia. Trust me, that impacted us in a bit. That was what we were talking about.

Okay, so it's not just my Twitter story. This is good to know. And so the intention behind this map was it was looking at kind of which counties relative to the others are the worst. So that's what this map was trying to do is one of these maps that had, you know, it highlighted, basically the ones that were in the top Percentile. But how audiences were using this. A lot of people were visiting this site every day and taking a screenshot of this map, and they were comparing this over time. And this particular visualization is really not meant to be compared over time because the bins are going to change every time because they're percentiles, they're quantiles. And so those bins are calculated relative to all the other ones. And so they're not telling. This map doesn't tell you if things are getting worse overall in the state. It just tells you basically which counties are getting worse or better by these different. And when I first saw this, people were a little bit outraged because they were like, look, they've changed the bins. And so, you know, in the past two months, it looks like we're the same, but we're actually way worse, even though these two plots look the same. And when I first saw this, I'm like, yeah, but the people that made those maps did the right thing. That's how you make those maps. How could you possibly guess what Bruins to use two months in the future? How could you guess how many cases there might be? So that you could set the colors in the past based on what you might see in the future? So that you could be showing this graph over time, Unless you retrospectively did it. How could anybody do that? But the fact that so many people, the way they were consuming this information was via looking at this once a day, taking a screenshot and comparing over time, that was an important piece of information.

Yeah. Would it be fair to say it wasn't being interpreted correctly because it wasn't being disseminated with enough communication of how it should be consumed to ensure that interpretation was occurring correctly.

Exactly. Yeah. And I think that the solution here as an analyst, is that either you need to very explicitly say these maps are not meant to be looked at over time and you know somewhere where it's like in the screenshot that will end up on the screen. There was also a problem with if you were viewing it on mobile or a desktop. If it was on a desktop, it was a little bit clearer that these weren't meant to be viewed over time. If you're viewing on mobile, there was a line that said you can look at things over time that was corresponding to a different plot that ends up not being in view on mobile. So there were other pieces, communication wise, that made this challenging. But. But what I saw from this was that either you need to make that explicit or you need to just take away this map and use a different form of trying to show this result because people are going to be doing what they're doing. People are going to consume information as they consume it. And I think outside of Georgia, I've heard from lots of people that the way that they've been consuming information from the Johns Hopkins maps and things is that they take screenshots and record this over time because there's some kind of fear that we're going to lose information and so they need to be holding it. And so this is like it's sort of a sociological kind of study, but just looking at this is how people are consuming this information that we're putting out. And it's not necessarily how the people who are putting out the information would have consumed it themselves, but they need to recognize that if the masses are consuming it like this, we need to be adjusting because the whole point of putting out these dashboards is to make them consumable.

So, yeah, so we totally need to get you on a consulting job as a side gig in Georgia for the Georgia Department of Public Health because we have 12 million Georgians who are. We're in now. We're doubting that. So we just need you there. We can fix Georgia.

I know it's so. It's a hard problem. I think it's not just Georgia.

Get ready to come down here.

I'm not too far in North Carolina.

So there you go.

It is interesting what you said about like, people don't trust that they'll be able to, like they think that they're going to lose this data or something or they're going to lose this information. It's like this thing was generated by some experts somewhere and who knows how long this website's going to be around or whatever. So I want to. They're basically trying to do some data gathering of their own and come up with their own sort of self service, you know, data dashboard or something that they think that they need. Yeah, I mean, that's a really interesting psychological thing. I don't know what the solution to that is in terms of giving people more trust that they'll have access to things. If it's like giving people the ability to, you know, self serve themselves data more, more frequently or I don't know.

So it's funny. I'll just note that in this particular example that we're discussing, that has happened that that lack of trust has happened where like lots of local media stations and other organizations in Georgia have been pulling data from like John Hopkins because they weren't trusting the Georgia data and doing a whole bunch of graphing instead of pointing people at the institution that's supposed to be doing that. You know, there have been a whole bunch of surrogates out there. So it's been interesting to see how trust affects that in a pretty, pretty big way.

Yeah, well, and I know, I mean, I, I'm not sure about in Georgia, but I know in other states there have been cases where things were on a dashboard one day and then they were not made available the next day. Day. And obviously that kind of. Oh yeah. So I guess it's gonna happen there too. And so I think that that kind of thing happening also can make the trust kind of, that can erode it a little bit too.

Changelog is the best way for you to directly support practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of practical AI into the future. Simply follow the changelog link in your show notes or point your favorite Web browser to changelog.com Once again, that's changelog.com.

Changelog.

It's better.

All right, so we, we started talking into these, these Covid related numbers with the, the vaccines and then also talked a lot about data communication. All of that super useful. But I do want to get a chance to talk a little bit more about that kind of third piece that you mentioned was mixed into that original problem we talked about of the vaccine numbers, which was causal inference. And I know that at the upcoming R conference, this is the first R conference that's going to be focused on government and public sector, which is super exciting. Chris and I are going to moderate a panel there which will be a lot of fun for us to join. But you're giving a workshop there on causal inference. Of course, that name has also inspired your podcast, and I see that sprinkled around throughout your web presence. So could you just give us a little bit of context for what causal inference means, why it's different than some of the other types of inference or prediction that we might perform as data scientists. I'd love to hear that, because I definitely think that we have not had that specific conversation on this podcast as of yet.

Yeah, great. I love talking about causal inference. So the big picture with causal inference, essentially it's exactly as it sounds. I guess I'm going to use the word and the definition, which I know you're not supposed to do, but if you're trying to answer a causal question. So does something cause something else which often most of the questions we're interested in are in that framework, although it is distinct from something like prediction, which would be a different kind of space. But in the inference space, when we're trying to determine the relationship between different factors or different variables, a lot of times we talk about associations as a way to sort of describe relationships that we know maybe are correlated, but we don't want to go as far as to say they're causal. But I would argue in most cases, the kind of human instinct is to want to talk about things causally. That's most of the time when we're studying relationships between variables, it's often because we want to know if there's a causal connection. So in randomized trials, like the ones that we're talking about with Pfizer and Moderna looking at different vaccines, they're not just interested in whether or not getting a vaccine is somehow related to whether or not you get Covid. They want to know if getting the vaccine will actually cause you to not get Covid. So this is sort of the relationship that we are focused on in causal inference. And so randomized trials kind of often are what people think of as the gold standard, although there are several ways that causal estimates can get skewed or biased in a randomized trial as well. So there are ways that you can actually need to do some more sophisticated analyses to get at a causal effect, even in the randomized setting. But then where my work mostly is is more in observational data, where we don't have a formal randomized trial. We're just sort of observing things. For example, in electronic health records, or you can think of all different types of data sets that have already been collected, but you want to try to determine if there's some kind of causal effect between different elements. And so to do so, you kind of have to build this framework that involves both kind of statistical modeling, but then also a lot of assumptions. And so a lot of times, like the way that we build these kind of causal. We build up this causality is being able to kind of state assumptions about our data that we're making. And should these assumptions be true, then we can assume that the effect that we're seeing is actually a causal effect and not just kind of association between two things. Does that kind of answer your broad question about causal inference?

Yeah. Yeah, it definitely does. It's very useful because I do think that it is the natural human reaction when we're doing any sort of modeling, to assume, like, if these features that are fed into Our model allow us to predict, you know, whatever it is, Y1, Y2, whatever those things out, then those things are somehow causing that response or those labels or whatever it is. It's a very natural reaction to think that. One of the things I'm curious about is. So you mentioned this sort of process of defining your assumptions, being very careful about how you do that. I'm sure there's a number of things, but what are some of the kind of common tools that people use in causal inference that maybe. Are there kind of gold standard tools or very common tools that people use in this case?

Yeah, so there's a lot of kind of underlying assumptions and there's ways that people try to kind of help get at them. So my work uses something often called propensity scores. And so what that basically means it's in the kind of observational setting where we don't have randomization to an exposure or to some treatment. And so you try to kind of construct what we call a counterfactual framework. Like, all I know is that you got, for example, if I'm looking at diabetes drugs and heart disease, all I know is that you received diabetes drug X. And I don't know what would have happened if you had received diabetes drug Y, for example, but I could try to construct what I think may have happened. And so I could basically look at all of the different, your baseline characteristics. I could adjust for all of those and then kind of adjust for other people who have the same baseline characteristics you could imagine on the other drug and sort of assume that those two, if you were to measure all potential characteristics so there's nothing unmeasured that might be confounded, then you could compare those two kind of groups, or maybe those two groups on average to each other to sort of be able to build that counterfactual that we couldn't actually observe. And so in the randomized setting, because we're randomly assigning you to one or the other, the counterfactual is much easier to deal with because we assume that all of those baseline characteristics kind of on average are going to end up being just. They're going to be balanced because we end up kind of randomly assigning to one group or another. But in the observational space, you don't have that luxury. So actually kind of constructing something that can help you achieve that balance between the two via something like propensity score is the tool that I use most often. And that's essentially it's just a summary score of your baseline characteristics. And so you essentially are estimating the propensity that you would get one treatment versus the other. And then you can use that as an adjustment tool in various ways, like weighting or matching to be able to get comparable groups. And then once you have comparable groups, then you can start making some more causal assumptions. But of course, the big piece here is that there's. You can't have anything unmeasured, which in a randomized setting, you're less worried about because we assume that things are going to kind of be balanced in the long run. In an observational setting, you have to either feel very certain you don't have anything unmeasured or do some sensitivity analyses to see kind of how bad things would be if you were missing an important variable.

Where do most people go wrong with this? Maybe they don't aren't thinking explicitly enough about it. They might be an experienced data scientist, but aren't really focused on implementing a counterfactual framework that's not part of their thinking. Where do you see people go wrong? Where it kind of takes that process off the rails a little bit.

I think the first place where people kind of, it depends because people kind of of all levels of experience go wrong on this. And so it's not even just beginners, but I think that the folks on the beginner side, it tends to be this unmeasured, confounding piece. And thinking about the plausibility of it, I think it's really easy to do kind of a sophisticated analysis that adjusts for many things and assume that you're really capturing all of the variability. And you see this a lot with electronic health records. That's where, because I do biostatistics, that's kind of the data source that I tend to be thinking about. But you've got tons of information in electronic health records. But that doesn't mean you have everything that's important. And it also doesn't mean that you're going to end up with an unbiased result. And so I think that there can sometimes be confusion between I adjusted for lots and lots and lots of things, and I have everything that's important. Like those are two kind of potentially distinct pieces of information. So the example that we talk about in healthcare a lot is for a long time there were studies that looked at hormone replacement therapy and heart disease, and it to used people. It used to be recommended that hormone replacement therapy was actually protective against heart disease. And this was based on several large observational studies that were all kind of conducted around the Same time. And that did adjust for several things. And they were seeing kind of these consistent results. And so you could do a meta analysis across all of these studies, and they all were sort of showing that it looked like it was probably protective. And then a randomized trial came about and it showed that it wasn't really protective. And in fact, there was a chance it could have even been harmful. And so this sort of threw a bit of a wrench into things and more studies were done. And it turned out there's some nice plots that you can sort of look at for this. But the observational studies that adjusted for socioeconomic status were showing null effects or even potentially harmful effects, and the ones that didn't were showing protective effects. And so essentially, the whole effect of, or the large part of the effect of this hormone replacement therapy on heart disease that was being thought to be protective was mostly driven by socioeconomic status, which just wasn't adjusted for in these original models. So while they adjusted for other things, they didn't adjust for that important variable. And it turned out that was actually a huge driver. So women who had access to kind of. They were from kind of higher socioeconomic categories, they had access to different health care, and so they were less likely to get heart attacks or have these cardiovascular events that it didn't have anything to do with the hormone replacement therapy itself. And so sort of this. I think that type of example is what the first relying on previous knowledge in an area doesn't always save you from this unmeasured confounder piece. And I think that people who are first, new to a kind of discipline your default might be. Okay, well, everybody that's fit this model before has included these variables, and this is what we do, and it looks like we're getting the same kind of effect that people see, so it must be right. And I think that's kind of not always the case.

I like the term for that, too. Unmeasured confounder.

Yes, Measured confounder.

I know that'd be a good podcast name if anyone else is. I don't know.

That's what I'm going to be next Halloween. I'm going to be an unmodernated confounder for Halloween.

Yeah, that would be a great Halloween. Yeah. On that note, not the Halloween costume, but the confounder note. I'm curious. I could see myself getting into a state where I'm like, I'm a little bit gun shy in the sense of like, oh, I'm like, in this situation, I'm Trying to do some causal inference. But I'm like, how do I know when I, you know, I could see myself always thinking there's going to be another confounder out there. How do I know when I'm ready to pull the trigger and like actually give some results to someone? There's so many different, you know, things at play in here, in this situation. And I'm just, you know, I think I've accounted for everything, but I don't know if I've accounted for everything. How do I know when I've accounted for everything or at least enough things to where I can have some confidence? You have any thoughts there?

Yeah, it's a great question. And unfortunately there's not like a test or something for have you accounted for all the things, you know, Unmeasured confounding is one that's specifically onerous because there's not a way to know for sure. One of my dissertation papers was on building tipping point sensitivity analysis for unmeasured confounding. And so the idea is that you can do your analysis and you do the best you can. You state the assumptions that you're making, and then at the end you can do one of these tipping point analyses that we basically show Mathematically, it's a pretty simple formulation. There's an R package called tipper that can do it for you. But essentially what it will do is it'll take your effect that you're assuming to be causal and it'll tell you the size of an unmeasured confounder that would be needed to tip that analysis. And so I think that that's kind of the best case scenario for what you can do. If you've done all that you can to account for what you have, then you can just explicitly state that, you know, we think that this effect, we feel like we've done what we can. These are the assumptions that we've made. If there were an unmeasured confounder out there like this that, you know, was related to the exposure and outcome in this manner, it would make our result no longer significant. It would make it, it would nullify our result or make it inconclusive. And so I think that's kind of the best that people can do. And I think just doing that would really move the field forward in a lot of ways. You know, one thing that that can do is once that's been stated explicitly kind of in a paper or write up or whatever it is that you're doing about this causal estimate that you're working on then content matter. Experts that come in, you know, from all different areas can see that. And some may say, hey, that actually is plausible because I actually have seen that this particular variable that you didn't account for can have that kind of impact on your exposure or have that impact on your outcome that that really would tip your analysis. And so sort of doing that work for them where they can see what a confounder like that would need to look like, and then they could sort of map that back to their own content expertise, I think, is what can help with this. And so that's my best recommendation on that, friend.

That's very helpful.

It's pretty good. Excellent. So I guess as we wind up, what are things that you're excited about? You know, it could be an R, it could be about causal inference, it could be trends that you're seeing. What are the things that you're looking forward to doing over the next couple of years versus the stuff that you have been working on?

Let's see. I'm excited about so many things.

No, that's fine. Let's hear it. Go for it.

Well, so I think on the causal side, I'm excited that this has been getting a lot of attention recently. I think that we had Roger Pang. He hosts a podcast called not so Standard Deviations with Hilary Parker. He also hosts one called the Alpha Report with Elizabeth Matsui. And I had him on our podcast a little bit ago talking about his thoughts on causal inference. And he talked about how he sort of at one point he had implied that maybe it's a fad, but essentially that like, people are interested in this. And I think that it's true. I think that it's something that people are gaining more and more interest in in terms of understanding the methods as opposed to just trying to make causal claims. I think people like, as we've discussed, human instinct is to want to make causal claims. And so that's something that people have always been interested in. But the interest in sort of incorporating the more rigorous methods has been going up. So I'm really excited for that to continue. I think in that space, I think we're starting to see better and better kind of introductory level information on how to conduct causal type analyses. And I think the gap still is sort of in that intermediate spot. I think that we have lots of people who are very competent on the heavy, heavy methods and lots of people who are working on the introductory. And I think we've got this nice middle spot that has a lot left to be contributed. So I'm excited about that because I think that there's potential there. And then kind of going back to the science communication piece that we were talking about before, I'm also excited, I think that a lot of the scientific process has been brought to the forefront with just the pandemic response and sort of thinking about how this was done and how it was communicated. And so I think that we have a lot of data and information now on kind of how people have tried to communicate things and where that has potentially failed or maybe where it succeeded. And so I think going forward, we have a good horizon for being able to sort of improve on how we're communicating results, which I think is only going to be something that's better for everyone, both on the scientist side and on the general public side. So those are two things I'm excited about. Could probably come up with others.

That's good.

You've made me definitely excited about those things as well. And if our listeners are, I'm sure they are also excited about causal inference and these things that we've talked about. I would encourage you very much to check out the upcoming R conference. You can go to rstats AI Gov and find out all of the info there. Lucy's giving a workshop there on causal inference. So it's a really great opportunity to dig deeper than we can during this period on the subject. The conference is December 2nd through the 4th. The workshops are on the 2nd, and the conference is December 3rd through the 4th. And our listeners have a special discount code. So make sure you use PracticalAI20 is the discount code. PracticalAI20. And you'll get 20% off all of the ticket types, including the workshop that Lucy's giving. So make sure and check that out. Chris and I will be there as well moderating a panel. So it's going to be a great time. I would encourage everyone that our community is so welcoming and awesome and I really encourage people to check that out and hopefully see Lucy and both of us there. Thank you so much, Lucy, for joining. It's been a pleasure.

Yes, thank you. It's been great. I'm so excited that you mentioned that workshop, the propensity scores that we talked about. You'll learn how to fit those types of models. You'll also learn about counterfactuals and things exactly the methods that we talked about today. You'll be able to actually implement from the comfort of your R console.

Awesome.

That's so perfect. Yeah, Great, Great timing. It was great timing this discussion because you know, those numbers came out just then for the, for the vaccines just a couple days ago. And then we've got the R conference coming up and people can just follow this whole story arc and get, get trained up in causal inference and go do all of these exciting things that we talked about. So thank you so much, Lucy.

Thanks for.

Come hang out with Daniel, Chris and hundreds of other AI practitioners in our community. Slack It's a cool place to be. Not a lot of noise, some great signal, and best of all, it's totally free. Check it out@changelog.com community and don't forget to follow the show on Twitter for AI news and links, highlights from past episodes and more. We are at Practical aifm. We'd love to have you for following along. Thanks to Daniel and Chris for hosting Practical AI week in and week out. To the mysterious Breakmaster Cylinder for the excellent beats you hear on all changelog podcasts. To our sponsors who have our back Fastly, Linode and launchdarkly. And to you for listening. We appreciate your time and attention. That's all for now. On the next episode, the guys chat with the team at Unsplash all about their huge open data release. So stay tuned for that one next week.