#21 The Fight Against Cancer

Hi, I'm Maggie, VP of Curriculum at datacamp, and I know firsthand how overwhelming it can be to keep up with the rapid changes in data and AI. But staying ahead doesn't have to be hard. That's why our curriculum team has curated the best learning content to help you and your team stay ahead, covering everything from data and AI literacy for busy executives to cutting edge generative AI for developers, and everything in between. Whether you're upskilling yourself or your team, we make learning practical, engaging and flexible. It just works. Power your data and AI transformation journey on datacamp.com.

In this episode of Data Framed, a Data Camp podcast, I have the great pleasure of speaking with Sandy Griffith from Flatiron Health, a healthcare technology and services company focused on accelerating cancer research and improving patient care. Sandy is principal methodologist on Flatiron's quantitative sciences team and is tasked with leveraging data science to improve lives by learning from the experience of every cancer patient. How can data science help in the fight against cancer? What are its limitations? Find out in this conversation from the frontier of research. I'm Hugo Bauen Anderson, a data scientist at Data Camp, and this is Dataframe. Welcome to DataFrame, a weekly datacamp podcast exploring what data science looks like on the ground for working data scientists and what problems it can solve. I'm your host, Hugo Bown Anderson. You can follow me on Twitter Hugo Bowne and datacamp at datacamp. You can find all our episodes and show notes@datacamp.com Community Podcast hi Sandy, and welcome to Data Framed.

Hi Hugo.

Thanks so much for having me.

It's such a pleasure to have you on the show and I'm really excited to have you here today to talk about your work at Flatiron Health and how data science can help in the fight against cancer. But before we get into that, I'd like to know a bit about you.

Oh, great.

How did you get into data science?

Sure. So I think it wasn't a straightforward path at all. It was a bit of a long and windy route to get here, I guess. Starting back to college. I originally thought I wanted to be a scientist. I think I started as a biochemistry major, but. But I did lab work and I really don't think it really suited me. It wasn't my favorite. I'm very clumsy. I think the precision of lab work wasn't really for me and so I still wanted to do science, but I didn't quite know how. And over the course of that time when I was in college, I ended up taking some time off. And I did a sort of different things that were completely unrelated. I had jobs like waiting tables. I worked in a video store. I was a 911 dispatcher for a few years also. All kinds of things that were completely unrelated to data science. And then eventually I ended up going back to school and back to grad school for biostatistics. And I think I chose that because I was good at math and it was a way to do science without actually doing science back in the lab. And so it appealed to me for that. But I don't think I had any idea of the potential of biostatistics and statistics when I went back to grad school for.

Seems like an interesting place, though, to be for someone who's interested in scientific research, scientific progress and the scientific method, but doesn't necessarily want to pipette or do lab work.

Yeah, absolutely. I think it's a great way. And I didn't know this quote at the time, but I had since learned it. I think it's from Tukey. Actually, this quote about statisticians is the really great thing about statisticians jobs is they get to play in everybody's backyard. That's the thing I love about biostatistics, is I don't have to pick one specific area of science. So my area of expertise is statistics, but I get to work on all different kinds of fields.

And that's one of the kind of very enjoyable aspects about being a data scientist as well, I think.

Absolutely.

So what happened then?

So, well, I was in grad school for biostatistics and, you know, the path for job options. When I was in grad school, it was pretty clear that there were three main categories of options that people I knew went into. So there were government jobs like the FDA or the nih. There were academic jobs and there were jobs. So for biostatistics, a lot of the people went into jobs in pharma after grad school. And so between those three options, it was really clear to me that I wanted to be an academic and something I had always thought about. It had a lot of appeal to me. And so that's what I did after grad school, I went and I became assistant professor of faculty at Cleveland Clinic. And that was great. That was really exciting. I did that for a few years. I got to work on really great research involving electronic health record data and patient reported outcomes. Outcomes. I worked a lot with neurology and behavioral health and it was really fun and exciting. But I don't know, I then hit, you know, I had a kind of a serendipitous moment in my life, and that was actually when I visited a friend of mine, Hillary Parker. Do you know Hillary? I do, yeah, of course. So Hillary had been a friend of mine for many years. We'd actually applied to grad schools at the same time together for biostats PhD programs. And then, you know, we were friends. We continued to be friends over the years, and we both really were into cats, and so we bonded over that. And this idea of our cat ladies, which was kind of a hashtag type of thing that we started once at an R conference. And so we were friends. I visited her in New York. She worked at Etsy at the time. And I went to work with her one day and I was just so impressed and amazed by the different kind of way that she worked in tech. It both seemed like so much fun, but then also just the pace that people were working and the way they were collaborating cross functionally and the way they were sharing ideas, it was just. It seemed very cool to me. And, you know, that actually that weekend when I was visiting her in New York, I also went out with another friend of ours, Hannah Wallach. So Hannah had been, you know, a very accomplished academic at the time in fields of computer science and machine learning. Just recently left academia to work for Microsoft Research in New York City. The three of us went out that evening and we went out to dinner, and it was in New York City, it was in Brooklyn. We went to this vegan restaurant in Gowanus. And it was just. I saw the promise of a different life for myself that was both in New York and working in tech and industry, but still getting to do really exciting things. And so that night I was just sold to this idea.

I suppose the question then is if you want to work in tech, but you love working in biostatistics, where. Where can that meet?

Yeah, exactly. So, you know, that that night I decided. But I didn't really have any options in front of me because I still wanted to do this biostats science research, but it wasn't clear what that would be. And so I just, I mentioned to. To people that I knew. And I think this is a great way for people who are looking to kind of change directions in their careers is just to tell people what you're looking out for. And then odds are, when people know what you're looking for, something might just pop up into your life when they know you're looking. And so, in fact, a few months later, actually Hilary had heard about this newish tech startup in New York called Flatiron Health, who was working with cancer research. And she sent me the link, and I ended up applying.

Incredible. And how big was Flatiron at that point?

So at that point, it was still pretty small. When I applied or when I first heard about it, it was probably. It was certainly under 100 people. And the time there was an idea that they'd be working with oncology data from electronic health records, but it was still really figuring out exactly how they would do that. But I came and I interviewed and I met with people, and the mission really resonated with me. And the people that I met, I was really impressed with. So I decided to kind of take a plunge.

Fantastic. So why don't you tell us a bit about what Flatiron's mission is, what Flatiron Health does.

Sure.

So part of what they do, it's helpful to kind of understand why there's a need. So the problem is that in oncology, only something like 4 or 5% of cancer patients are actually on clinical trials right now. And so those patients, they're getting different treatments, and their outcomes are recorded and used for research and used to generate new knowledge and to change practice. But the other 95 or so percent of patients with cancer, they're seeing their doctors and their oncologists, and their information is recorded electronically in their electronic health records, or EHRs. But for the most part, and especially several years ago, when this was happening, it wasn't really being used for research. And the reason why is because the data that's being recorded electronically, it's not recorded for research purposes. So it's actually complex and not straightforward to use directly for research in many different formats that aren't necessarily standardized. It's siloed in different locations. And so, for the most part, all of those experiences, those patients, whether, you know, they're receiving treatments and some of them work and some of them don't work, but for the most part, it's lost.

And so it sounds like there's. It's a huge mess of heterogeneous data.

Yeah, absolutely. And so that's the challenge. And that's where Flatiron Health was really coming in at the time. And, you know, the business that they were building was that they were both partnering with oncology provider, and so those were community oncology clinics as well as academic cancer centers to work with them, to give them tools to help with their practices in seeing patients. And those could be. There's an electronic health record that's oncology specific that we develop and license, as well as things like tools for building support and understanding their patient populations. And so that's one side of it. At the same time, we work to process the data coming from those electronic health records at scale. And that's easier said than done, which is why when I started, there were over 50 software engineers out of that, less than 100 people. Because it's a really massive technological undertaking and really a tech company at heart. And, you know, part of the way that we do that, we process that electronic health record data at scale, is through a combination of structured and unstructured data. And so the structured data is complicated enough in itself, just because, like I mentioned, things are not standardized, things are in many different formats. And so there's a lot of processing and harmonization involved with that. And we have a great medical informatics team here as well as the engineering team that helps with that. But then actually, especially in oncology, a lot of the real information in the data is in unstructured fields. And so those could be clinician notes or radiology reports or handwritten notes, and those both are not in an easily accessible format to begin with. But also there's a lot of clinical complexity there. And so we use something called technology enabled abstraction to process that unstructured data. And that's really. It's a combination of both technology and data science and also humans that combine to process that data. And so how that works is we actually have a large network of clinically trained abstractors. And so those are tumor registrars or oncology nurses, and they're working on software that we've developed to actually look through some of the documents and try to do it in a smart way based on the technology, and actually extract and curate a lot of these complex clinical data points out of those unstructured documents and turn them into an analytic database that is then suitable for research.

And it sounds like you have a vast amount of data, but you also have a lot of people working on it who have domain expertise that you can work with.

Yeah, absolutely. I think that's one of the things that makes this really special is that we have quite a few oncologists that work not as consultants, but are working embedded on all of our teams. And so almost every day talk to an oncologist in some capacity as part of my work.

So what do you do yourself at Flatiron Health?

Yeah, and so, you know, a lot of what I do is working on the research side. So after we have that analytic database that has research grade quality data. It's both understanding what's contained in that and then also actually asking research questions on that data and performing research. And so my role right now is on the quantitative sciences team. And so right now I'm a. My title is I'm a principal methodologist on the team. And what that means is right now is actually a lot different than it was when I started over three years ago, partly because the role and the team has evolved a lot over that time. So when I started, there was only really one other quantitative scientist on the team. And it's grown to about 20 people or so over the last three years. So it's been a big growth.

Do you feel like you have worked for several companies?

Yeah.

So my job has definitely evolved a lot over the last three years as the team has grown. So early on, my job was very much a generalist, just like you'd imagine for anybody working at a smaller startup. So I did a little bit of biostats, some study design, a lot of just really digging deep into the data that we had and immersing myself in it so that I really understood it as well as this is a tech company at heart. So I certainly really had to flex a little bit more on the data engineering side, though certainly not as good as is what the actual engineers would be doing, but definitely flexed more in that direction. And over time now, as the team has gotten a lot bigger, now my work is a lot more specialized, more on the biostatistical method side. A lot of doing research, writing publications, and also mentoring other team members.

Great. So I saw you speak at RStudio conference recently.

Oh, great. That was a fun conference. Really, really good.

It was. And your talk was. Was fantastic. And one of your slides, one of your opening slides in particular, I found really inspiring. What I want to say is that we've spoken a bit as to Flatiron Health's mission, but your slide said, our mission is to improve lives by learning from the experience of every cancer patient. And what I'd like your help in is unpacking that statement because I think there's a lot of interesting, interesting content in there. We've discussed this a bit, but I'm wondering how I want to think about the word or the term every cancer patient. How can data science help you or us to learn from every cancer patient?

Yeah. So, you know, like I mentioned up front, the traditional paradigm has been only learning from a small percentage of cancer patients. And as we know, that's not ideal because we're Throwing out a lot of information. And also those small percentage of cancer patients don't necessarily represent the experience of every cancer patient. We know that patients in clinical trials, cancer tend to be typically younger and healthier and have different demographic characteristics than the patients that are actually out in the community seeing their oncologist on a regular basis. And so I think it's important to both learn from more people, but learn from a more generalizable and diverse group of people.

And how do you do that?

Yeah, and so, you know, part of this, you know, I don't know. Have you ever heard of the idea of a learning healthcare system or a rapid learning healthcare system?

Not a lot.

So, you know, it's this idea in the, you know, the healthcare services delivery world of that, you know, our healthcare systems shouldn't just be about treating patients because we can, especially with the advent of electronic health records, learn so much from each experience. And so the promise and the idea of this rapid learning healthcare system is actually to take this rich patient focused data that we're collecting at the point of care and then actually aggregate that data to generate real world evidence that tells us something about that collective patient experience. And then at the same time, we're actually transforming the way that we deliver care to patients and thinking more about a personalized medicine approach as well as then after we transform that care, evaluating the outcomes and seeing actually did the things that we do make a difference in how patients actually end up doing in the end. And then the idea is that this is just a system that loops around and that we keep feeding the learnings back into the system and continuing to change care and change treatment for patients.

Great. And that iterative process I think is essential. So I have a question revolving around this rich patient focused data and the collection. How do you guys go about thinking about getting as much data as possible or do you?

Yeah, so I mean certainly we, like I mentioned, we look at structured data and unstructured data. And so I think it's, you know, it's about as much, but I would say more than just the number of patients, it's the depth per patient in terms of like the longitudinal context over time. So, you know, it's, it's one thing to just have a large number of patients, but I think what actually gives you so much more information is to be able to follow them over a long period of time and really annotate their experiences over time in their clinical context. And so, you know, we think about that in a few different ways because, you know, from a Data side, you have just all of these different, you know, tables in a relational database that might correspond to each patient. But we think about, like, for example, how do you visualize a specific patient journey over time? And this is where a lot of the close collaboration with oncologists come in that I mentioned. Because, you know, it's one thing for me to say what I think is important, but, you know, really you want to know what are the clinically important things that are happening to patients over the course of their care. And I think it's really hard to do that without a lot of that really deep domain knowledge. So we've, you know, we've looked at ways to visualize at a patient level, like, for example, when they're starting their first set of treatments, you know, when certain events happen, like, for example, if their cancer gets worse or if their tumor starts to shrink and how that relates to different biomarkers. They might have different genetic testing.

Yeah, and that actually really speaks well to this idea of learning from the experience of every cancer patient, because I'm really interested in the fact that, you know, we have data points and aggregates of data and that type of stuff, but that doesn't necessarily tell us about this idea of human experience. So when you have this patient journey visualizer or really think about the entire longitudinal experience, that really speaks to that aspect of the human story, I think.

Think absolutely, that helps you see the human story and then it also tells you something so much more about the data. Because just looking at an isolated data point out of context only tells you so much. I think, especially when you get into this. This clinically complex stuff, you really need to look at it all together.

Yeah. So then that next step, how is. How is this knowledge that, that we learn in this process transformed into the actionable of in inverted commas, improving lives as. As is your. Your mission? And how does data science play a role in. In this improving of lives?

Yeah, great question. So, you know, ideally, what we want to do is we want to use this information to learn from patients to accelerate cancer research and the treatment options and the knowledge that is out there for each patient as you imagine, you know, if someone that you. A loved one is going to their doctor, you want that doctor to have, you know, first of all, all of the information available to really know what happened to patients that. That are similar to them when they get different treatments so they can make a really educated decision based on the data of what is most likely going to work for that patient or not work for that Patient on the drug development and regulatory side, Another benefit of actually taking all of this real world evidence and generating knowledge from it is actually thinking about, when we think about the process of the really lengthy and expensive process of traditional clinical trials, if there are ways to speed that up and help get, you know, good treatments approved faster for patients than it would typically take. And so we think about ways that we can both supplement existing evidence from clinical trials with data that are collected in the real world so that we understand more about how those treatments are working and in what populations they work and thinking about getting these treatments to patients faster and then also thinking about different ways to do, for example, traditional clinical trials, are there ways to do them in the real world with more representative patient populations that are more reflective of how care is delivered?

And once you perform this research, how and where is it implemented?

We work in a variety of different ways in terms of the research. So some of it we might be working on internally, we might be working on publications. So we have a lot of publications in process. We have a lot that have come out recently. Our first joint publication manuscript with the FDA just came out about a month or two ago. And so, you know, when those publications start to come out, they start. We start disseminating that knowledge to the scientific community. And that's one way that it's implemented. Partner with, for example, life sciences companies as well as academic cancer centers, and I mentioned the fda, also the National Cancer Institute, to work on developing this research and then using that for, like I said, different ways to get treatments to patients faster.

Great. We'll jump right back into our interview with Sandy Griffith after a short segment. Now it's time for a segment called Blog Post of the Week with Spencer Boucher. Hey there, Spencer.

Hey there, Hugo.

Spencer, today I want to tell you about a blog post that I really like.

Oh, okay.

I'm usually the one bugging you with the nerdy blog posts. Hugo, so what's got you excited this week? It's a post by Alan Downey, a professor of computer science at Olin College and the author of many great books related to computer science and data science. The post is on Alan's blog. Probably overthinking it. The post is called There is Only One Test. It's from 2011 and still as relevant as ever. All right, I'm excited. So what's it about? It's about statistical hypothesis testing and the fact that although there seem to be a bunch of different tests such as the student's T test, AB tests, and chi square Tests. These are all actually variations on one test and that there is only one test. Alan also provides a compelling and to my mind correct argument that we can utilize the power of modern day simulation to see that there is only one test. All right, Hugo, so why don't you break this down for me by first giving me an example of one hypothesis test? Sure. I'll actually give you one of Alan's examples. Let's say that you own a casino and think that a gambler has replaced a die with a crooked one. You confiscate the die and roll it 60 times, say to get some data on it. Your hypothesis is that the die is crooked. Crooked? In the language of hypothesis testing, this is called the alternative hypothesis. You also have the null hypothesis that the die is fair. You then figure out the probability of seeing the data you got under the null hypothesis. And this probability is called the P value. If this P value is small, say less than 5% in many fields, you conclude with 95% confidence that the die was crooked. So what type of statistical test test would you use in this case? Spencer? There is only one test, but I get where you're coming from. In addition to a null hypothesis, a statistical test requires the construction of a test statistic, such as the difference between the number of times you would expect each value 1 through 6 to be rolled under the null hypothesis and the numbers you saw when you rolled it 60 times. In this case, you may want to use a chi squared statistic. And don't forget that when constructing your test, you also need to specify the null hypothesis itself, itself, in this case, that the die is fair. And if you were computing the mean value of a control versus an experiment, you would maybe use a student's t test statistic which measures the difference between the observed mean and the difference expected under the null hypothesis. Yes, given certain other conditions are satisfied, and the point of Alan's post is that these are all the same test. We're just choosing different relevant hypotheses and statistics to compute before the power of simulation. These different tests were created as efficient ways to compute the relevant p values. So tell me, how exactly does simulation help here? It essentially allows us to skip these different tests and to merely simulate the data given the null hypothesis lots of times to get the distribution of possible values under the null hypothesis. We can then read off directly how likely it was to get a test statistic at least as extreme as what was observed in the experiment, such as the difference in the numbers rolled with a possibly crooked die. And this applies to so many questions. So whether that be crooked dyes or click through rates or differences in drug efficacy. Exactly. Are there any other really strong takeaways you got from this post, Hugo? There are actually several and we don't have enough time to cover them all here, so we'll definitely link to Alan's post in the show Notes I do want to focus on Alan's point to that the exact p value is irrelevant and what you're really interested in is the order of magnitude. To quote him, if the p value is smaller than 1%, then the effect is likely to be real. If it's greater than 10%, probably not. If you think that there's a difference between a 4.8% usually considered significant and 5.2% not significant P value, you are taking it too seriously. Moreover, Alan provides several further compelling reasons as to why simulation beats analytic tests. I'll tell you one of them once again, in Alan's words, analysis often dictates the test statistic. Simulation lets you choose whatever test statistic is most appropriate. Alan makes clear that the choice of the null hypothesis is also arbitrary and specifying it is absolutely necessary for the test.

Wow.

Okay, well this sounds like a great post, Hugo, and I don't even think I've quite done it justice. I'll also add that Alan has a great follow up post called There is Still Only One Test. Now I want to close out with some words from Justin Boyce, who has taught us statistical thinking in Python courses here at datacamp, in which he teaches this simulated approach to hypothesis testing, among many other things. Justin recently wrote to me, using named tests with analytical solutions means that we can only test available hypotheses with available test statistics, whereas a resampling approach frees us of these constraints so long as we can simulate it. We get to pick the hypotheses we want to test and the test statistic we want to test it with. See you next time, Spencer. No, Hugo, I will see you next time Spencer, you cheeky bugger. After that interlude, it's time to jump back into our chat with Sandy. So we've spoken a lot about what Flatiron does. I want to step back a bit and think about this fight against cancer in, in general. And I'm wondering, in this discipline, what are the biggest challenges in the fight against cancer?

So you know, I think you know from a, certainly from a data science and statistical side, part of what I see some of these challenges are is that, you know, we so much has happened over the last many Years in terms of understanding the biology related to cancer, the causes and the treatments. But you know, when you actually go from that, from that science that's happening in a lab and try to translate that into clinical practice, into something that's actually going to help patients directly, you know, I think there's a lot of open questions, questions still about how you take those scientific findings and actually translate them into real benefit to patients.

And I suppose there's that translation challenge, but there's also translating technical data, scientific results to clinical practitioners who may speak another professional language, right?

Oh yeah, absolutely. And I think that's one of the ways that our cross functional teams here are so critical. Like I said, software engineers working with oncologists working quantitative science is that, you know, everything, that nothing that we do is in isolation. So very few things that I do are just for a statistical or data science audience, almost everything I'm actually really, you know, speaking to these different groups of people and trying to make things that, that are accessible to others.

And you've spoken to already the fact that a lot of the data out there is heterogeneous and very different. Presumably this is a place that data science can play a huge role as well.

Well, yeah, absolutely. And honestly that's what I think is one of the things that I think is so fun about working with real world data that's generated from electronic health records and other data sources is because it does have that heterogeneity and variability inherent in it. And to some people that just seems like, well, that's too much. It's not actually going to be useful to answer our questions because of that variability. But, but from a more data science side, I actually see that as the fun challenge is that because of that you're not going to see maybe the same strength of a signal that you might in a really controlled laboratory experiment or even perhaps in a highly controlled clinical trial. But that doesn't mean that you can't get any information from it or come to any conclusions. I think especially when you pair that with rigorous study design, high quality data and rigorous methods, methods that then that's where even in the face of that complexity and uncertainty, you need this data science and methods more than ever to make sense of it.

Yeah, and in a sense you're saying that you're actually really excited by that challenge as well, right?

Yeah, I love to say that. I like complicated data. Actually one of my biggest areas of interest from a statistical side is in incomplete and missing data just because it's, it's not always straightforward to analyze, and it presents a lot of challenges, but it also has so much potential that can be unlocked if you know the way to unlock it.

Yeah. And once you start finding the signal, it can be pretty exciting.

Yeah. I mean, one of my favorite things about working with data is the moment when you get to, I don't know, look at a plot that you've created or a set of output from a regression model, something like that, and you. You get to see something about the world that you know that exists in the world that no one's ever seen before. And you're the first person. You're sitting at your computer and you're looking at this output, and you're the first person to ever see something about the world. And I think that's one of the most exciting things about my job.

I couldn't agree more. But I do love that, you know, you're actually excited and interested in the job isn't difficult enough, and you love having incomplete and missing data. So that's. That's huge. I love that a lot.

Yeah.

And I mean, you hear a lot of people talk about data that's more complex like this, and they say things like, garbage in, garbage out. But I would put it really differently. I think it's really. It's more like it's complex data in, and I think it's nuanced answers out for sure.

And I think as you spoke to before, the fact that we're dealing with clinical notes that may be taken in the margin of certain reports, there's a lot in there to be extracted.

Oh, yeah, absolutely. And, you know, often, you know, it can be conflicting data where you really need to look at the clinical context. I mean, one of my favorite examples is, you know, an example of a. I show this sometimes in talks of an actual redacted patient chart where you can see that the patient has undergone imaging to see if their tumor or their cancer has gotten worse or progressed over time. And there's a radiologist report from the imaging that actually says, well, yeah, the tumor increased by this amount of centimeters over time, and their cancer is getting worse and progressing. But then there's a note below it, and on the original radiology report, the oncologist actually goes through and crosses that out by hand and says, no, actually it decreased in size to this many centimeters and that the radiologist was looking at the wrong report and they didn't have the full clinical context. And so that's the kind of thing that we often see in these data. And so that's why we really need to think about the clinical context when we're making sense of it.

Yeah, that makes sense. So are there any initiatives, and I'm sure this is actually a very difficult challenge, are there any initiatives with respect to opening these types of data records to essentially crowdsource data analysis and data scientific techniques?

Yeah, so that's one of the things that we're thinking about now in terms of both the trade off between respecting patient privacy and abiding by HIPAA regulations and making sure that we're working with de identified data, but then also ways to make the knowledge accessible to others.

We've discussed a lot with respect to the fight against cancer, what data science can do, but what can't data science do in these types of questions and what other disciplines are essential?

So I kind of touched on this a little bit with that, you know, some of those examples of those complex handwritten notes and conflicting data that we have. And so I think our approach to the technology enabled abstraction is a great example here. And so we do use, for example, machine learning to help with some of that processing. And we use technology to build the right software to do it and to find the right documents. But at the same time we recognize that we do need humans to curate data, at least some of the time, especially for the really tricky and complex different data points. And both we need the humans to help with that, for both getting high quality labeled data to then use for modeling later if we need to, but also for those cases where the deep clinical context is needed.

We haven't spoken much about what type of languages you guys use or what programming setup you have, and maybe you could tell us a bit about kind of the technological or technical side and the infrastructure you use.

Yeah, absolutely. So right now the Quantitative sciences team is using primarily R for most of our analyses, though I would say we use different variants of SQL for things because all of the data live in different relational databases. Some people on the team use Python some of the time, especially for certain projects, or collaborating with some of the software engineers who use Python primarily. But R is definitely the standard for our team, but it wasn't always necessarily going to be that way. You might have seen some of this at the talk I gave at the RStudio conference.

I do recall, and I actually recall that you yourself were a driving force for the adoption of R. Yeah.

And you know, that's also one of the great things about being at a small startup is you can really help to form some of the direction of the team over time. So when I started, my background was mostly with R and I was certainly using R and wanted to use R, but there were others here that had actually come from more of a SaaS background. And it's not crazy the idea that a company that's working in healthcare and partnering with pharmaceutical companies would be using sas, because that's actually the standard in a lot of these places. And also thinking about, for example, there's been a long standing myth that for example, the FDA would only work with people who were using sas, which just isn't true. We've done collaborations with them and used R. And so early on when we thought about this, there was really a question of, well, should we be primarily using SASS or R? And one of the reasons why R won out in addition to, I guess maybe my enthusiasm is we are a tech company so we use things like we use Git for version control, we use fabricator for code review. And SAS just doesn't really work with those things.

So we've seen so many advances in data science in healthcare technology. What does the future of data science in healthcare technology look like to you?

Well, these are always the hard questions though, I guess. Yes, I guess as a statistician I should be able to predict the future. So you hear a lot of people talk about big data and just amassing larger data sets and all the potential for big data. And I think certainly there's a lot of merit to having larger sample sizes and richer data, but at the same time big data doesn't necessarily solve all the problems. I mean there's of course like well known from a statistical side is that if you have a biased estimate of something that increasing your sample size isn't going to reduce that bias. That's just going to increase the precision on a bad estimate. So I think that the big data is not the only answer here. But what I do think is that well curated smaller data set sets that you compare with larger data sets is really what's going to help us make sense of them. And you know, I think about it almost like a calibration data set type of problem where, where you can have a larger data set with less information that you can, can learn more about through a smaller data set.

And this speaks to kind of, I always say, stop throwing models at your data initially. Check it out, have a look what's happening, spend a lot of time with it before throwing whatever, you know, convolutional neural nets or whatever it is at it, right?

Oh yeah, I 100% agree. Like I said, one of the things that I spent a lot of time on when I first started here wasn't in a lot of statistical methods or fancy modeling. It was actually just really getting deep into the data and just understanding it in and out.

Great. I suppose that also speaks to your love for missing and messy data.

Yeah, I mean, I think that, I don't know if that's the future of data science, but I do think that certainly any future will need to be comfortable with those sorts of things and recognize that the kind of data that we're seeing in the future is not going to be data that were necessarily collected for research purposes and will fit into exactly the specifications we expect, but that it's going to be complex, there's going to be missingness, and that, you know, we really need to be able to be well versed in methods to handle that and also be comfortable with. With understanding that.

Yeah, right. So that made me think, do you have any challenges with data lineage, like getting a data set and not being even sure which clinic it came from or anything?

So that's certainly something that we think a lot about. And I think that really speaks to a lot of really, metadata type of considerations. So you can look at a data element, but that only tells you so much. What I really want to see with a data element is I want to have metadata understand where it came from, what some of the variables of associated with that are. So, you know, like what clinic it came from, what electronic health record they were using when they started using that electronic health record, some other factors related to that, and also understanding metadata about the quality of that data element and how it's changed over time and really importantly, what we're benchmarking it to. So I think that's really key is to have that information to understand more about the quality of the data so you can actually then use it in context. Context. And you know, from a lineage perspective, certainly it's something, you know, that we think a lot about. And I think that's where the, the tech company piece can really help with that and thinking about partnering with software engineering to have better tools there.

Yeah. We'll jump right back into our interview with Sandy after a short break. Now it's time for a segment called Tales from the Open Source. Today we'll hear from Chester Ismay, a curriculum lead here at datacamp. Chester helps instructors build courses for datacamp, builds his own courses and projects, and also develops R packages and open source textbooks. Hi, Chester.

Hi, Hugo. Thanks for all the work. You've done on this podcast and for having me on the show, it's an absolute pleasure.

What are you going to tell us about today?

Do you remember the definition of the p value from your introductory stats course? What about which hypothesis test to wrote run in a given situation? In my experience as an educator, I found these topics to be tricky to cover and challenging for students to grasp. The Infer R package I co authored is designed to assist beginners with understanding the underlying processes of confidence intervals and hypothesis tests. Hypothesis tests are frequently taught as being analogous to a U.S. criminal trial. We initially assume the null hypothesis is true, just like being not guilty and only strong evidence from what we observe to the contrary makes us change our mind about this assumption. Similar to beyond a reasonable doubt in a criminal trial, confidence intervals provide a range of plausible values for an unknown quantity where we use what we see in our sample to make some clarifications as to how wide that range of value should be and where the range should be centered. Further, we wanted the package to come at inference from a computational tidy perspective made popular by the tidy version, so that both beginners and practitioners could use the package in their analyses. If you aren't familiar with the tidyverse, maybe the simplest way to put it is that R functions there frequently correspond to English language verbs, making code easier to read. The Infer package fits into this already existing ecosystem of tidyverse packages.

So how exactly does the Infer package match up with this tidy perspective?

The package provides five main functions that give a framework known as There is only one test made famous by a blog post by Alan Downing. These five functions, in order are specify, hypothesize, generate, calculate, and visualize. You'll see how this could be applied via an example from the TV show Mythbusters. They tested to see if a person can be subconsciously influenced into yawning if they see another person yawning. In the study, participants were randomly assigned to two one where they saw someone else yawn the seated group, and another to a control group where there wasn't a yawn seed. We can use this experiment to see if we have significant evidence to conclude that yawning is contagious. The null hypothesis is that there is no relationship between the response variable of whether the participant yawned or not and the explanatory variable of seed or not. We now have the arguments for the specify and hypothesize infer verbs. Next, we permute the values in our data to simulate there being no relationship between the two variables. We repeat this process many times to generate many different simulated samples assuming independence of yawning or not with seed or not. We then calculate a statistic summarizing our sample for each of these generated repetitions. Lastly, we visualize the results of these statistics and observe how what we saw in our original data falls on the null distribution and complete our hypothesis test.

So did the Mythbusters also perform a hypothesis test on air?

No, unfortunately they did not. They concluded that since the proportion of yawners in the seed group was larger than the proportion in the non seed group, that they had evidence showing yawning was contagious. If you actually complete the analyses with their data, you'll find that the sample provides non statistically significant results that could be easily explained by randomness. Tisk Tisk. Jamie and Adam Tisk. Tisk.

I see. Is there anything else that prompted you to create this package?

We found that while R has a lot of inferential techniques built in, the syntax was not consistent. We felt the frustration beginners had with this and wanted to help fill in this gap. This was true of even traditional hypothesis tests like the 2 sample T test and the Chi square test for independence. To further complicate matters, students that had learned the tidyverse first struggled when trying to navigate how to perform methods to build intuition. Our hope is that this framework links data science and statistics, motivating the concepts behind statistical inference.

So where can our listeners learn more?

ModernDive.com is an open source introductory data science book that I've co authored that we'll discuss Infer soon. There are also currently two launched courses on datacamp called Inference for Regression and Inference for Numerical Data, with two more to come soon that use Infer.

Thanks for this modern dive into inference with R. Remember to check out infer.netlify.com for more information. Thanks once again Chester.

Always great to chat with you. Hugo.

Time to get straight back into our chat with Sandy. So what's one of your favorite data sciencey techniques or methodologies? It doesn't need to be one that you use in your daily work, but just something you love doing.

Yeah, so I mean one technique that I kind of like to pull out sometimes and you know, I think this is interesting because it's not, I think it's not that fancy or bleeding edge in a lot of ways, but it's more about, you know, some of the practical issues that I face. So a lot of the work that I do, it's not necessarily pure prediction problems. You know, I work with a lot, like I said, a lot of oncologists, we really want to understand what the science is behind what we're seeing. We want to have interpretable results that really help us make inference and understand a problem better. And so in those cases, things like some of the newer machine learning models might not necessarily be appropriate if we want these really interpretable stories that we can tell at the end. But at the same time, one thing that I've used that I've paired together is I really like to use, for example, random forest for variable selection when for example, we might have some clinical context about which variables we really should be including in our model. But there's some things that we're actually not sure about. We want to use the data to help us understand. So it might use a random forest for that, but then actually take the results of that, really help help me understand from that perspective, help visualize some of the things that I see in terms of the functional form for the variables as well, and then use them in a conventional regression model so, you know, a logistic regression or some other type of regression model to then have interpretable results. And you know, one thing so in oncology, something that we use a lot here is we use a lot of survival methods and so things like Cox proportional hazards models where we have a time to event outcome, because that's particularly important in oncology. And so, you know, one thing that I'd like to do is there are random forest methods that are actually available for these time to event outcomes with censoring. And there's some nice packages in R that I really like. There's a random forest SRC package as well as there's a ggrandom forest package that is a nice pairing to that to help with some of the visualizations.

Cool. So something there was a through line there which is the need or desire to provide interpretable results. And there's a trade off between how accurate a model is and how interpretable it is. And I'm wondering the type of stakeholders in your line of work, where's the middle ground or which side would you prefer to be on more accurate or more interpreted, interpretable?

I think most things that I'm working on need to be interpretable. There are some pure prediction problems out there, but a lot of what I'm working on really does need to be interpretable. So I think I probably tend to be on that side of the divide. But that said, you still want accuracy. But what I would say is it's important to know the limitations of accuracy of whatever you're doing, you might not have the best accuracy or the best prediction, but the really important thing is, I think, to. To be able to understand and quantify it.

Yeah. Could you give us an example of a case in which you'd really want it to be interpretable?

Yeah. So give an example of a project I worked on recently, and it was looking at patient characteristics that were associated with a certain outcome in oncology. And so that outcome is called something called pseudo progression. And what that is is. I'll just unpack that word. It might sound a bit little. A little foreign, but it's basically. So if a cancer is getting worse, we could say it's progressing. But in the case of pseudo progression, that's when patients have been taking more recent therapies called immunotherapies. Sometimes instead of the tumors getting smaller, they might actually, for a small period of time, actually get bigger. But it's not the cancer getting worse. It's a temporary increase in size of the tumors that will then shrink down and actually the cancer will be getting bigger. And so that's pseudo progression. But the thing is, because these immunotherapies are so new in practice, and the way that oncologists are using this is quite variable, we don't always necessarily really. We don't necessarily always really know what people are thinking in terms of in practice. Like what. What patient characteristics are associated with either clinicians thinking that a patient might be having this outcome or. Or that patient actually having this outcome. And so when we did an analysis on this, we. We looked at some of these different factors, but we weren't necessarily trying to. It wasn't a predict problem more. We were just trying to basically understand something about what was going on both in the treatment of these patients and also in the way that oncologists were thinking about this. And so in that case, you absolutely want to be able to have interpretable results at the end. This is actually an example that what I mentioned a few minutes ago about using this combination of random fars for variable selection, and then in this case, it was a logistic regression model for the interpretable result at the end. It helped us put it into context because actually, kind of a. It's a tricky story to unpack. And this is also why working with an oncologist to really go through what some of these results are and make sense of them was really key.

Yeah. And as you say, working with an oncologist really, in this case demonstrates the necessity of domain expertise running through this.

Yeah. And I would really Emphasize that. So, you know, over time, I've been, you know, working at flatiron for over three years, and I've certainly developed a good amount of domain expertise for someone coming from a statistical background. But I'm never going to have the same amount of domain expertise, no matter how long I work in this field as an oncologist who's actually out there and treating patients. And so actually a lot of the oncologists I work with here are still in the clinic, usually one day a week, actually continuing to see patients.

So my last question is, do you have a final call to action for our listeners out there?

Actually, one thing that comes to mind actually, at the RSTUDIO conference you brought up earlier, I know someone asked a panel there, Hadley Wickham was on the panel, and they asked what he thought some of the most underestimated tools in data science were. And he said adding and subtracting was his answer. And I actually think that's a really great answer. I think a lot of the time just counting things, figuring out the denominator is a huge part of the problem. But one thing at the time I remember I tweeted back to him that I would add is really framing the question. I think that's one of the most underutilized tools in data science is thinking about what are the questions that we're asking and how we frame them and making sure that the answer that we get at the end really answers the question that people want to know the.

Answer to and how it changes our actions as well.

Right.

So framing the question, understanding that depending on what answer we get, what then our actionables are.

Yeah, absolutely. And I think another thing I'd love to see from data scientists is just also being involved in the asking of the questions and not just being in the position to answer them. I've always thought of myself as a scientist who just happens to have expertise in statistics. And so, you know, I love it when I get to see other people who are really part of the asking of the questions as well.

Sandy, this has been so much fun. Thank you for coming on the show.

Oh, thanks so much, Hugo. This has been a. This has been great.

Thanks for joining. Our conversation with Sandy about the role of data science in the fight against cancer. We saw that data science is increasingly well positioned to deal with data generated by real world clinical practice in all its heterogeneity and variability. We saw that with these emerging techniques, there is the promise of a rapid learning healthcare system that will feed learnings and knowledge back into the system system. Big challenges now are having humans in the loop to curate at least some of the data, particularly calibration data sets. And the challenge of missingness and messiness in data still remains. Make sure to check out our next episode, a conversation with Michael Bettencourt, physicist, statistician and one of the core developers of the open source statistical modeling platform Stan. Mike and I will be talking about doing robust data science and model based building with statistical modeling. Part of the bread and butter of doing data science is building models of the world to both understand phenomena and make predictions. But building models isn't always just fitting and predicting. How do you do robust, reproducible data science when building models of the world? Join my conversation with Mike in next week's episode to find out. I'm your host Hugo Bown Anderson. You can follow me on Twitter, Twitter hugobown and datacamp. At datacamp you can find all our episodes and show notes@datacamp.com Community podcast.

Sam.