DeepLearning.AI Learner Community Event ft. Neelesh Kamkolkar

Good morning, good afternoon, good evening wherever you are. Thank you so much for taking time out of your day and joining us here today. Let me start by sharing my screen. So welcome. Welcome to the DeepLearning AI Online Mini Event Series for AI for Medicine Learner Community. I'm honored to be your guest speaker today. Look forward to sharing and learning from all of you as a community. As part of Ground Rules, we're going to go through the presentation for about 20 to 25 minutes. Please go ahead and post your questions on the Slack channel. If you see a question that's already been asked and you want to ask the same question, just go ahead and upload it so we can go through and answer those questions ahead of the queue as well. So without further ado, what I'm going to cover today is operationalizing ML and healthcare and the real world challenges that you see. You've all read my. You've all read my. Let me minimize that window here. So you've all read my bio, but I thought in the spirit of community that I share a little bit more about myself more personally. So I volunteer at Teals, which is a nonprofit organization sponsored by Microsoft Philanthropies, where we teach computer science to high school children and it is one of the most fulfilling and amazing experiences that I've had. I volunteered at a foundation called Esha Foundation. I occasionally get on stage and we have opportunities to sing and perform and I love to be in the mountains. I'm on the picture on the right, that's somewhere in the Himalayas in the Lila dark region, about 15,000ft or so, just enjoying a hot cocoa. But it's above and beyond all of the professional things I love to give back and I think this is a great opportunity that DeepLearning AI has created for me to give back to the community as well. So I appreciate that. Let's jump in. Today we're going to talk a little bit about context and background. I wanted to frame the conversation, take five minutes and frame it so that we can all see the problem in a similar way. I'll touch on what are some of the differences between traditional software and machine learning software as you go through the lifecycle. And then we'll talk about the role of ML code in production scenarios and cover real world challenges and best practices that I have learned along the way doing this for the past few years. So as background we built healthcare machine learning solutions for providers and payers. We built a lot of models along the journey and a lot of times we would do A proof of concept. We would get going on a proof of concept and then the client would want us to turn that into a pilot or go to production. And that life cycle had a lot of learnings along the way, which I wanted to cover. We work with all kinds of data. We work with big data. We had over 10 plus years of data from some of our customers. Seven, seven plus years from some other customers. Nine plus million encounters. That doesn't include all the labs and the vitals and other data points that you might have, which really explodes the amount of data that you receive and have to work with. And of course there is a lot of small data too. I mean you get an Excel spreadsheet with 10,000 rows of data and someone says, hey, create a mod. So a wide range of data, variety and volume if you will. The kinds of models we built for providers, for example, is around things like length of stay, how long do you anticipate or expect this patient to stay in the hospital once they've been admitted? Discharge disposition, right? So where do you think at discharge time? Where do you think this patient is going to get discharged to? Are they going to go to a skilled nursing facility? Are they going to go home? Do they have to go to the pack and elements of patient care through the patient continuum that tie very closely to the business in terms of reimbursements from cms. And particularly for those of you that are in the United States, you will know that Medicare is a big part of how payments are made through providers which are hospitals. And of course for those of you that are not in the US I'm pretty sure hospitals go through these stages where somebody comes into an ER and they have to make a decision very quickly of whether they need a bed in the emergency room or do they need to be transferred to an inpatient sort of a setup, inpatient being the hospital itself. So lots of different models, lots of use cases. And I think one of the double edged swords with healthcare is the fact that there are an incredible number of use cases that you can go after that it sometimes can be challenging to decide which use case should you go after. Of course we also build recommendation systems. These are programs, for example, that government of Singapore has publicly talked about in the open domain where they have been able to provide wellness programs to their citizenry and being able to provide direction and nudges to citizens to say, hey, there's a walkathon happening in your neighborhood. Or they have all these programs that allow them to eat well, you know, shop healthy as well so part of that, you know, we were delivering millions of nudges to many, many people. So that's the background and sort of, that's sort of what we have been doing. And to frame this, right? We weren't the Googles, we weren't the Facebooks, we weren't the Netflixes. So I think when you think about some of these lessons we're learning, think about this more from a perspective of a startup trying to get into this space and how that journey went. So just to frame traditional software and ML AI software, traditional software, you all know you have some data in the shape of variables and you have a set of rules that you want to apply to that data and then you write an algorithm that, that uses the data and the set of rules and out comes the result. And that in the most simplistic sense it could be a functional programming method that you apply. On the flip side, if you look at the way we build ML and AI software, and all of you are very familiar with this, is we provide data as input and we provide results, essentially the labels as input to the process of creating a machine learning model. And what we do is we try to match the data and the labels and hypothesize and come up with a hypothesis function that best fits the data and the results and that becomes a set of rules. That's the image to the right. And of course there's a lot of iteration and experimentation that happens inside that black box when you're building an ML model. So let's just say you've gone through and you prototyped and built a machine learning model that you think is really high value. So what do you do now? You need to go and deploy this to production. One of the seminal papers that I would encourage all of you to read is called A Hidden Technical Debt in Machine Learning Systems. If you have read goes through a lot of details around some of the things that companies like Google have seen in terms of the technical debt that gets created over the lifetime of a model. Essentially that paper points to the fact that only a small fraction of real world ML systems is composed of the machine learning code. So as members of the community that are building these models that want to change the world, that want to do good and, or solve a business problem, right? Whatever use case you're pursuing, for that to be usable by end users, for that to be scalable to millions of users, you have to go to production. And this recognition and the framing of the fact that ML code is a portion of the larger ecosystem so you might be wondering, what are the bigger ticket items. This is an image I created inspired by that white paper that I mentioned earlier. If you read from left to right, imagine a well timed conveyor belt, which is really your process management, that is driving and dictating everything that happens through the lifecycle of your machine learning model. You're going through and you're collecting data, you're verifying data, you do feature construction, you do feature validation based on all of those things. Maybe you get to a point where you have your model run an API call or do a scoring for a particular use case. But surrounding all of that you also have infrastructure that you need to manage and host. You have service configuration, you have to monitor all these services that you are hosting for that ML code to really create an output and usable by lots and lots of people for your machine learning algorithm to be adopted. In other words, there's a lot of other moving parts that somehow as data scientists, AI engineers, doctors, nurses, we need to be aware of that. We can, we can enable our organizations to make that adoption easier. Let's go through the journey of if you say you have a model and you're ready to go to production, operations, infrastructure, security compliance, monitoring, management, there's a lot of these areas and your operations team might start asking you questions like, well, what kind of infrastructure do you exactly need? Do you need an 8 core machine with 64gb of RAM or do you need a 16 core machine? Is your model secure? Is your model encompassing any private data that you may not want to get leaked outside? Is your model compliant with our corporate security policies? Right, and essentially the list of questions goes on. So hopefully by the end of this presentation, you'll get a sense for how to tackle some of these questions that you get ahead of time as you're going through your journey. Right, so what did we do? Right, so in my experience, we've built out this platform very simplistically put where we target providers which are hospitals, payers or insurance companies, and wellness use cases. We provide a secure data pipeline, we provide a feature construction layer and we allow automatic creation of features so that a lot of iteration and machine learning can be accelerated. Essentially all of this was built on top of Azure, Kubernetes and databricks. This ultimately gave machine learning data scientists and AI engineers the ability to develop, test and then roll out their models to production. These are the areas that I want to touch on in terms of real world lessons in healthcare data correctness and validation, machine learning, feature construction. We look at model Development and the lifecycle of the model and what some of the challenges there are. We'll look at performance of the model and how do you think about the health of the model and the features that are coming in to your production deployment. And also if we have time, I'd like to go into the need for having a service mindset and what it is. We'll talk a little bit more about that. Let's jump in. Hopefully that framing just gives you a perspective of a lot of the hard work that we do early in the lifecycle for a machine learning model is also important to educate the rest of the organization and the rest of the teams that surround that model and its deployment into production. Data correctness and validation. If you look at some of the areas, this area is huge. Given the time, I'm only able to cover a small set of these things, but there's syntactic and type correctness issues. When you get data from a customer, either in an Excel spreadsheet or through Azure, through a blob store somewhere or in aws, you get data in a certain shape or form and that shape or form has to match whatever it is you're going to get in production. One example of the thing that happened is during an early poc, we got some data that had age as a feature. So most of you that are building models in healthcare know the demographics, you know, tend to be a good set of features to have have in your model. So, so of course we built and used that age metric or age as the, as the column and used that to validate or build and validate a model which was performing very well. And now the next step was, hey, can you roll this out into production? And of course when we went to production, what we found out that the production data set didn't really have an age column, it actually had something, it had the date of birth and because of HIPAA and personal protected information pii, we essentially had to make that field. The date of birth field, excuse me, was also de identified. So it was shifted by left or right and there's all these techniques that you have in terms of protecting P. So that distinction essentially meant that the model we built during the POC was really not appropriate. So we had to literally go back and rebuild the model. So thinking about not just the data that you get early in the project, but working with operations team and say what set of the data that you're looking at now actually exists in your production environment becomes important. Of course, that's if you want to take that model to production, ontology mapping is a very challenging problem. For those of you that have been working in the space for a while, you know that mapping ICD codes, snowbed codes, they're all different across different deployments of EMRs and also different institutions and organizations that you deploy. So if you have one customer tell you that the demographic or the sex of the patient is male or female, and we've learned some of this in class, right? We've sort of changed you. We use one hot encoding to change some of those columns from M and F to 0 and A1. Some other customers might encode that as the full worded letter, right? Male and female spelling the whole thing out. So if you're getting different ways of expressing the same data, how do you manage that? What do you do about that? And create production. So it's an important pre planning step where the teams can engage to identify and understand what ontology mappings are needed and how to factor those into production. Morphological correctness of data. You know, it's interesting data, data values can be just completely off sometimes. Data can be really dirty. For those of you that have worked, I mean, we heard from Roger Stone last, last speaker that was here, he said he spent months trying to clean the data up. And he's not alone. In the industry, we find that 80% of the time that is spent on building a machine learning model is really around cleaning data and massaging data and getting data in the right shape. So having the data and knowing some of the semantic insights into what the data might mean is important. A blood pressure reading of 800 is absolutely not meaningful if you're manually looking at it. If you're looking at it visually, for example, creating some plots, you might find these outliers and read those, those out. What do you do in production, right? What do you do if your daily batch of data or your stream of data contains just the wrong number because of a bug in some ETO pipeline? Right? And that's important for data scientists to think about because data in prediction out, it's garbage in, garbage out. So how do you put data validation tests and create unit tests for data, if you will, through your machine learning lifecycle becomes important. I'm going to go through faster here because I'm looking at my clock. Semantic correctness, if you look at a blood pressure reading, systolic, diastolic, the numbers, the quote unquote, normal for adults may not be the same as normal for pediatric patients. If you're building a model for adults versus building a model for pediatric population, you might think about these features differently. Feature construction, availability and scale. Right. So what do I mean by availability? The fact that you need an MSDRG or a diagnostic resource group for a patient when they're getting admitted, because that's the data field that's getting filled in the EMR and flows through the data pipeline right to the prediction algorithm. Assuming that those DRGs are available may not be the right assumption. Sometimes they're just not there at midtime. Or sometimes a triage nurse might provide an early diagnostic that might go get updated at a later time in the patient's care continuum. So how do you handle those kinds of situations? We had one situation where, and I think this is more for the product managers also on the line, where a customer really passionately argued for a particular item in the EMR they were filling in that was very critical. They thought that was the most important thing. And to the extent that if you're not using this in the model, your model is probably not flawed. Is probably flawed, I don't know how to trust your model if you're not using this. Interesting journey not to point any fingers, but I think what happens is the passion of the customer translates into a lot of work. But also at the same time, when we looked at the data, the data told us a very different story. The data told us that while that field was being populated by nurses, it wasn't actually used anywhere in their operational pipeline. I think looking at how do you bring feedback from end users into your machine learning model and incorporating that in a data driven fashion becomes important in terms of just availability of features and not all features may stay the same. Like I gave you, the example of age and date of birth scale is another factor. So we worked with very large data sets and when you're looking at millions and millions of encounters and lab and vital data, our pipeline would take hours to go through it. And initially in the prototyping stage it was like, it's fine because we're trying to figure out what the model is and what are the right set of features that are required for that problem definition. But also over time, this takes a long time. But what happens from a customer perspective or the problem definition perspective is these, these predictions have to be available in a mission critical way to your end users at some point in time. So in terms of scale, while we were running it for three and a half hours, that may or may not be sufficient in your use case to say, well, it's okay for the pipeline to run for three and a half hours or maybe it's not. Okay. So I think factoring those is an important. Where does scale come into picture is we had this situation where the pipeline was taking forever to run and it wasn't going to meet our customers sla. We had to figure out how to optimize the runtime for that whole pipeline for that particular customer. Of course, as part of triage and performance, cloud systems are so easy to scale. You can easily scale up a cluster or scale down a cluster. Now, in an organization where data science and operations teams are working closely together, that's one thing. But if you're disjointed, you could fire up and scale up your cluster significantly larger, imparting a significant amount of cost without being aware of the cost that is coming out of that scale that you're trying to use the clusters for debugging performance. So easy to auto scale, but easy to burn money. So ideas like cohorting become important. And in our case what was going on was was the machine learning pipeline was doing what it was supposed to do. But the data ingestion pipeline had a bug that got introduced over time which was causing a Cartesian join between two tables and it was exploding the data set. The big lesson for us was if we.