A Chat with Andrew on MLOps: From Model-centric to Data-centric AI
Andrew argues that improving data quality beats chasing new model architectures. For datasets under 10k examples, systematic fixes to labels, targeted collection, and error-driven updates deliver bigger gains than model changes. He walks through the ML lifecycle and shows how MLOps keeps data consistently high-quality across stages.
Chapters
Hey. Hi. Thanks for hanging out with me this morning to chat about ML Ops and going from model centric to data centric AI. There are some ideas that have been fleshing out with a couple of my teams landing AI and deep learning AI over the past year, plus that I hope will be useful to a lot of us in the AI community. Hopefully you too, in terms of helping us be more systematic and efficient in how you might build and deploy AI systems. So, want to share those ideas of you and maybe also get your feedback. Here's what I mean. AI systems are made up of data and code. Let me share my slides. All right, so AI systems are made up of data and code. And by code I mean the model that you might implement in your network architecture, what have you. Excuse me, by code I mean the model including the neural network architecture you might choose, that is then trained on the data set. The way that machine learning has developed over the last few decades, a lot of progress has been driven by teams downloading and trying to do better on standard benchmark data sets. And so we spend a lot of time improving the code or the model or the algorithms. What I'm finding is that for a lot of problems, it'd be useful to shift our mindset toward not just improving the code, but in a more systematic way improving the data to then help us get the performance you want your learning algorithm to have. Let me start with an example. I'm going to make this presentation a bit interactive, so I'll ask a few questions that you can answer online in this presentation as well. But here's an example of inspecting steel sheets for defects. This is a project that one of my collaborators, Ivan Zhou, was working on. This is a picture of giant rolls of steel, a little bit like the rows of aluminum foil you might have in your kitchen. But instead of super thin sheets of aluminum, these are giant thick sheets of steel. So, you know, one of these sheets of steel may be the car door of a car you may drive in the future. It turns out when these sheets of steel are manufactured, it can have different defects like this, like this, like this, like this. And in this Factory, there were 39 types of defects that the factory operators wanted to detect using computer vision. If you train a neural network, train a baseline system, well tuned hyperparameters, you get 76.2% accuracy. The factory operators wanted to target 90% accuracy because that's how well human inspectors were at spotting these types of defects. This is important. You don't want to buy a car and then find that there's a wrinkle on the sheet of steel that was used to make the door of a car you're driving or just sport. So I know I've given you relatively limited information so far, but I want to ask you a question that you just answer according to your gut feeling. If you were the one leading this project, given what you know now of the project, you know 39 classes, so 76% accuracy is not bad. But given what you know now of the project, which isn't that much, if you were leading this team, would you ask the team to go and improve the code or would you go and improve the data? So if you're watching the live stream, go ahead and go to this URL and pick one of the answers. Just ask if you've got a feeling I'm curious what people are thinking and then I'll share people's share what people are voting for in a second. All right, going to stop sharing my screen. Cool. Okay, so there are many of you on the Aha Slide. ML Ops slide voting. All right, let me, let me, I'll take another five seconds. I'll share what everyone is voting. Okay, four, three, two, one. Okay, cool. So this is all right, well over a thousand of you on this site voting. Okay, so great, thanks, thanks. All of you voting. It's like an 8020 split. So the interesting thing to me is that maybe instinctually many of us think we can work on the data to improve the performance, although there's value to improving the code as well. But a lot of the history and culture and DNA and instincts of ait AIT is to talk about the code of the neural network architecture rather than the data. Let me share with you what actually happened when Ivan was working on this project. This is what actually happened. 76.2% accuracy was the baseline performance using a model centric approach. Meaning take the baseline, which is a pretty decent model with reasonable hyperparameter parameter tuning, decent learning rate, decent regulation parameter. Different teams actually went in search for neural network architectures. Go to GitHub, download the latest state of the art algorithm, try that out. Different neural network architectures, different models. After a lot of work by multiple teams, this resulted in 0% improvement. There's a bunch of models that wound up between 71 and 75% accuracy. It was very difficult to make improvements this way, but using a data centric approach in about two weeks wound up getting 93.1% accuracy. A much bigger difference in this project by improving the quality of the data. Now let Me just run through a couple other examples. Solar panel defect detection. Look at the solar panel, see if there's a problem with it. Baseline was 75.68% accuracy. Searching for new models, trying different neural network architectures. Slight improvements, but improving the data quality. Much bigger improvement. And one last example, another surface inspection problem, 85% accuracy. Almost no improvement with multicentric approach and a bigger improvement with a data centric approach, by which I mean trying to systematically improve the data quality. When I speak with non tech, when I speak of different people, including especially not less technical teams. I found this analogy to be sometimes useful. So maybe it'll be useful for you too, which is I tell people that data is food for AI. And in the AI world today, the way we use our data, it feels like we're like the chef that wanders in the back of someone's kitchen and figures out what's been at the back of your refrigerator for three months. Maybe slightly leaky. And we just cook that, right? Maybe, maybe the food is not terrible that way. But we know that master chef preparing a good meal views it as a critical part of their job to source vegetables, clean the vegetables, prepare the ingredients, and then at the end there is the application of heat to prepare the meal. But that's a small part, important part of their responsibilities. So too is this in AI. For many years, maybe 10, 20 years, we've been telling each other as if as a joke that hey, did you know 80% of the work in machine learning is data cleaning? 80% of data science is data cleaning. We say it like as a joke, but to me if 80% of our work is preparing high quality data, then I think preparing that data is the core part, a core part of the work of a machine learning engineer. Now the importance of data has been widely discussed, but what I would love to see is a shift toward making the way we approach data more systematic. And that's the importance of mlops, which I'll talk a little bit more about later. A little bit inspired by this 8020 rule. 8020 heuristic. Few weekends ago I went into arXiv and in a completely unscientific survey I went and searched for the hundred most recent machine learning papers and skimmed 100 abstracts of the hundred papers I find. So in this non scientific, I don't want to say it's representative. I just skimmed 100 abstracts. I found that 99% of the abstracts I skimmed seem to be talking about different ways of improving models. And only one of the hundred abstracts 100 papers, research abstracts I skimmed was on data augmentation in this case, and seemed to be about the data. So if 99% or some large percentage of the research work in AI is on 20% of our actual work, then I think it'd be worthwhile shifting quite a lot of our attention to the other 80% and making that more systematic. One theme you hear me repeat multiple times is how to do this systematically in terms of making data preparation systematic. A framework I use is to think about the life cycle of a machine learning project. So here's what I mean. When you think about building a machine learning project, there's usually a lot of emphasis on training the model, and that's important. You've got to understand the neural network architectures, have to understand how to build deep learning algorithms. That's important. But I think training the model, training, error analysis, iterative improvement, that's just one small part of the entire lifecycle of a machine learning project. I think of the full lifecycle as this. And if you are building a machine learning project, I hope this framework will be useful for you too. First you have to scope the project, decide what you want to work on. Once I decided to work on speech recognition for voice search, just decide what to work on. Then you have to define and collect the data. Then you have to train the model. Carrier analysis. Sometimes error analysis would drive you to go back to improve your data set. And then after doing this a few times, you may then deploy the model in production. And often based on the data you get back from your production deployment, you might decide to go back to update the model or go back to collect more data. This diagram. My thinking on this diagram was heavily shaped by two of my friends, Dylan Laird and Daniel Biberieta. But I hope that this diagram, this framework will be useful for you to think about your machine learning projects as well. Let me step through these four steps with an example. Scoping, say you decide to work on speech recognition for voice search, which I once did early in my life. So decide how much data you need, how much resources, how much engineering, how much compute. So decide to do that. Next step is to collect data. So in what I'm calling a data centric approach to AI, the one of the most important questions you have to ask is, is the data labeled consistently? So if it's not just do you have enough data? But also is the data high quality? Let me illustrate that with an example. Here's an audio clip of something. Very typical audio clip you might get in voice search, right? So Let me play this audio. Today's weather, right? So the question is, given this audio clip that you just heard today's weather, would you want to transcribe it like this, which is perfectly fine, today's weather, or would you want to transcribe it like this, which also seems perfectly fine, using dot, dot dot or the ellipsis punctuation mark instead of comma or would you want to say that, well, there's a lot of noise in audio. Sometimes something may fall down and there's a clunk sound or other noise and it's just noise, so let's not transmit it like this. It turns out any of these labeling conventions are perfectly fine. I would probably use the first or the second one, but really any of these will work. But what is not fine is if one third of the labelers uses the first and one third the second and one third the third transcription convention because then your data is inconsistent and is confusing for the learning algorithm and this will hurt your learning algorithms performance. How is the learning algorithm supposed to guess which of these transcription conventions one of the labelers happen to use? Let me illustrate this with a second example. One of our friends, Kian Katunfourouche, really likes iguanas. Iguana detection example. But this example actually maps well to specific computer vision applications like defect inspection, but are more technical, so just hard to explain. But let's say you want to build a computer vision system to detect iguanas. So you go out into the field and maybe go into the forest or wherever the iguanas are and take 500 pictures of iguanas and you send them to a labeling team and say hey labelers, please use bounding boxes to indicate the positions of the iguanas. So what happens? Labeler1 says one iguana, two iguanas. Good job labeler. It was great dataset, really helpful for your learning algorithm. Second labor says, well the iguana on the left, the tail goes all the way to the right. So I should draw a boundary box around the entire iguana. That's one iguana, two iguanas, this is the second. Equally hard working, equally diligent labeler. Can't fault them either. A third label says, well, I have 500 images to label. I'm going to use a boundary box to indicate where the iguana is. Okay. You know, hard to fault this labeler either. Again, it turns out any of these labeling conventions is fine. You can make it work. I would prefer the first or the second to the third, but any of these can be made to work. But what is not okay is if the three different labelers, if different labelers use different labeling conventions, because then your data is inconsistent and this hurts the machine learning algorithms performance. Now what I see is that experienced machine learning engineers, or inexperienced one will sometimes kind of hack around in the Jupyter notebook and spot this problem and fix it. And this will result in an improvement in the learning algorithms performance. But the shift I think we should make as a community is instead of counting on the skill and sometimes luck of an individual machine learning engineer to spot problems like this in images, in audio, or in other problems, I think we should shift as a community toward making the spotting and correcting of problems like these much more systematic. So this is what some of my teams do. Hopefully these practices will be useful to you, which is when there's label. When concerns about a data set, we would ask two independent labelers to label a sample of images and then quantitatively measure consistency between labelers to discover where they agree or disagree. And it turns out in problems with a lot of classes, it's actually not easy to find the one class out of the 39 classes where they disagree to fix that. Which is why having tools and metrics to focus your attention on the right policy and consistency turns out to accelerate machine learning development. And then for classes or the categories of data where the label disagree, revise the label instruction until they become consistent. There is a nascent view called MLOps Machine learning operations that is trying to make the development and deployment of machine learning systems more systematic and repeatable and reliable is a nascent view. But I think these are some of the tools that will make a big difference to mol. So I'll say more about this in a second. But to give an illustration of what a tool like this, what a process like this can do, this is an example from the steel defect detection that I mentioned at the start of this presentation of the 39 classes. One of the classes was foreign particle defects. So sometimes on the sheet of steel there could be a speckle of lubricants or a speckle of dirt on the sheet of steel. And so we found that one labeler will label it like this. Defect, defect, defect, defect, defect, defect, defect. Different labeler will say, well, given these specks of lubricants, specks of dirt, looks like there's a bunch of specs there and there's a bunch of specs there, and it's actually difficult to spot, it's difficult to fault these labelers. They're actually well meaning hard working trying to show you where the defects are. And we found that if you can go discover this problem and ask Labeler 2 to label in a way more consistent with Labeler 1 in this case, then the labels look like this. And this has a big impact of the algorithms performance. I think in terms of how we can make this focus on data more systematic. For a long time a lot of AI, a lot of machine learning has had a model centric view. And I think we're all kind of trained to do this because of the way machine learning research made progress. You download the benchmark data set and if you can improve the performance on that, you publish a paper. So a lot of machine learning progress was made this way, which is good, it's not a lot of bad things. But in the model centric view, which is the way a lot of people approach problems today, the view is collect what data you can develop a model good enough to deal with the noise in the data. In this view, you hold the data fix and iteratively improve the code or the model. In the data centric view, which I urge you to consider taking for some of the problems you might be working on as well, you can view the consistency of the data as paramount and to use tools and processes to systematically improve the data quality. In fact, what I find is that if you have a good enough data set, sometimes there are multiple models that will do well. Go ahead, download the latest state of the art thing that was just published on GitHub two weeks ago if you want. But I found for a lot of practical applications. So unless you have good enough data, there are plenty of models. You do need to understand neural networks and you need to understand models. But when you have that foundation, you understand the models. Then when you get your data set clean, there are multiple models that can work well. But in this view you can hold the code fix and iteratively improve the data. Now I'm about to ask you another question on AHA slides, but I found that especially with small data sets, modern neural networks are low bias machines. So if you have less than, I don't know, 10,000 examples, model size data sets, Modern neural networks are low bias machines, meaning that you can almost you very likely fit your training set perfectly. And so performance problems, the need to improve the performance of your neural network is usually a variance problem. And improving the data set is just a good way to reduce the variance. So I've alluded to maybe the importance of these in smaller data sets. Let me ask you Another question, go to this URL and I'm actually curious. I have no idea what your answer. Think about the last device learning model to train. How many training examples did you have? Please enter an integer. All right, cool. And then, you know what, I will. Yeah, let me take another, I don't know, few seconds and then I'll show what people are answering. Now I'm using a word cloud to show everyone's answers. Not sure that's the best visualization, but okay, show people's answers in another five seconds or something. All right, feel free to keep on answering answers. Cool. Okay. Wow. Right? Maybe a word cloud wasn't the best visualization I chose. So I think the 10,000 is in the middle because that's the single most frequent answer. But there are a lot of different answers. I see on the low end, someone's doing something with 346175 training samples. There are a few in the hundreds. I see 250, 50,000. Some large numbers as well. All right, there's a one with a lot of zeros there flying around someone's 10 million examples. A few people have 5,000 examples. The larger fonts indicates the more frequent answers. 6,000. Right. So several thousand. 120,000. So there's a huge variance with a lot of answers that are 10,000 or fewer, but also some in like 10 million examples or even larger data sets. Cool. All right, wow. Okay. About a couple thousand of you have been submitting answers, so maybe I need to find a better visualization. Didn't work out. So I think it's difficult to know. Sorry, you see this. So it's difficult to know exactly what is the most representative sample of training set sizes across all of AI. I don't actually know how to get that answer, but talking to friends, my gut feeling is that the majority of machine learning problems actually have relatively smaller data sets, although there are a few with very large data sets. So in another, In a I have to admit, not terribly scientific survey, one of my team members went on to Kaggle and again, I don't know if this is representative of the AI world broadly, but went on to Kaggle and counted up the number of training examples in the most recently posted trending data sets. And this was the histogram of data set sizes. And there is a large range with the majority of problems with under 10,000 examples, but then also some very large ones with over 100,000 examples. What I see chatting with many friends across the industry is that the problems with the very large data sets tend to get a lot of the pr, especially because AI had grown up in consumer software Internet. If you're a large consumer software Internet company with a billion users, over hundreds of millions of users, then you have a data set of a billion trillion examples, or hundreds of millions of examples, and should use that. And that's great that you have so much data. But especially as AI makes its way into other industries, everything from healthcare, where the number of electronic health records is just much smaller per healthcare system, or agriculture, where you don't have a billion tractors, you may have dozens of tractors or dozens of fields, or manufacturing or other industries like that, There are a lot of problems where the training set size is less than 10,000 and there, I think a different set of ideas is helpful for building learning algorithms that can help us collectively break into these other industries. So it turns out that smaller data sets, when you have a smaller data set, label consistency is even more important. So let me illustrate an example. So one of the things I used to do some of, you know, many years ago, I used to fly helicopters using machine learning. One of the things you might want to do is input a voltage and try to predict the speed of a motor, the speed of the rotor or speed of a motor in a helicopter, in a machine or something. Here's a small data set with five training examples. And this is a noisy data set. And it turns out when you have a noisy data set like this, it's pretty difficult when the data set is small to know what's the right function to fit this data. It could be, maybe it could be straight line like that, Maybe it's a curve that looks like this flat. And then I don't know, maybe it looks like this, maybe it looks like that. I don't know. There are lots of lines that could fit this data. So when you have small data set, five examples in this case, and noisy labels, it's difficult to fit a good model. It turns out noisy labels is just fine if you have a lot of data. So if you're a consumer Internet company with a billion examples, or maybe a million examples, well, learning algorithm can average out over the noise. So if this data set is equally noisy as that on the left, but because you have lot more data, you can more confidently fill the curve like that. You'll just average over the noise. So that actually works? It works. Okay, maybe when you've log data. To me, an interesting case is if you have a small data set, but a clean, consistent set of labels, Even though this data set has only five examples Same as the leftmost one. Because the data set here is clean, you could very confidently fit this curve. Even though a lot of the PR and hype in the media is about the giant data sets. And if you have a giant data set, good, use it. It's great that you can use it. I have been repeatedly surprised at how often I can get a learning algorithm to work with 40 images or 100 images or just a very small data sets. And doing that, I found is critical to have processes and tools to make sure you have a clean and consistent data set. Let me share with you what the theory says. Let's say you have 500 examples, say your iguana detector, right? So you found 500 pictures of iguanas. And if 12% of the labels are noisy, so incorrectly labeled, then from a theory of information, kind of Shannon information calculation, you can show that the following two techniques are equally effective. Option one is you can go and find the 12% of data that was mislabeled and clean up the noise. And you know 12% is not that high. If you have eight labelers and one of the eight labelers uses an incorrect or different labeling convention, that's 4.5%. So I see a lot of data sets where the label noise is much higher than 12%. But so one thing you could do is clean up the noise. So 12% of 500 is 60. Just find the 600, excuse me, find the 60 images that are mislabeled and clean that up. Option two is you can go out into the field with your camera, find another 500 iguanas wandering around the forest, take another 500 pictures, and then get labelers to label your 500 new images to an equally to the same labeling quality as before, double the training set. Turns out from theory, information calculation, both of these will be about equally effective in terms of improving your learning algorithms performance. I would argue that for a lot of problems, it's much easier to go in and find your 60 mislabeled examples and find a systematic way to clean that up than to go out and double the size of the training set and then label it with a similar level of noise as before. So what I found is that in consumer software Internet, if you're building users, some challenges are cleaning up that data. But in all of the other industries, plenty of problems with 10,000 or fewer training samples, like the numbers that a lot of you entered just now on the interactive website just a couple of minutes ago. There's a lot of room for improvements in problems with fewer than 10,000 examples, which I think is maybe the majority of machine learning problems we as a community face today. I often use the number 10,000 because that's roughly the range below which is feasible for machine learning team to basically look at the entire training set. We can look at all 10,000 examples manually, just barely. And so that gives you more flexibility in how you go about analyzing develop the data. Let me just share with you one more example of this concept. This is a project we're working on with automotive company Automotive Components Classification. So had a noisy data set where the bounding boxes slightly inconsistently labeled. And with the noisy data set, this is the performance accuracy or mean average position as a function of training set size. We found that if you clean up this noise go to clean data set, then this is the performance. Imagine if you had started off with 500 examples and a noisy data set. If you wanted to get up to 0.6 map or 0.6% accuracy, then you could either go in and clean up the data set or you could shift to the right and maybe you need to nearly triple, double, triple go from 500 to over 1000 training samples in order to get that to similar level of accuracy. This is another example where my teams found that it's just easier to clean up the data. I've been talking a lot about the importance of these ideas in smaller data problems. But it turns out that even big data problems but once where there's a long tail of rare events, those are actually small data problems as well. So take web search. Everyone knows that the large web search engines have a lot of data. But it turns out that web search is a long tail problem. Meaning that there are a lot of web search queries that a search engine may see only once today and may never see again or may not see again for a very long time. So for those rare queries in the long tail, there's actually very little data on them. So those are actually small data problems or self driving cars. There are lots of rare events that can happen on the road. Specific configuration of traffic or a different car doing something that your subject car needs to be safe in those situations. And so those rare events are actually small data problems. Also product recommended systems. If you launch e commerce site with millions or tens of millions of more products on your website, then if you have a long tail products, the products that are rarely sold tend to be ones we have relatively small data sets. And so even for those big data problems, when you have a long tail, you actually have small amounts of data. For the tail. And so some of the ideas I'm sharing today, not all, but many of the ideas I'm sharing today will be applicable as well. So when I'm working on smaller data sets, I found that having tools and processes for the data quality turns out to be critical. Again, I know I'm saying data is important. Well, we've known that for decades. But I think the shift is how do you build tools and processes and make this systematic. Come back to that in a second. Let me keep rolling forward with the speech recognition example. Talk about scoping the project, collecting the data, and now training the model. Let's say that you train a speech recognition system and error analysis shows that your algorithm does poorly in speech with car noise in the background. So this is what speech with car noise in the background sounds like. You can hear me talking clearly. I'm not actually in a car, but it was in a car. Maybe this is what you would hear. This happened to me. I built a speech recognition system and found that it doesn't do well when there's this type of con noise like you're hearing in the background. So let me turn that off. So with a model centric view, you would ask, how have I tuned the model architecture to improve performance? And you can do this, Maybe build a filter to filter out the lower frequencies which con noise tends to. You can actually do this and maybe it'll work. But with a data centric view, you would instead ask, how can I modify my data? Maybe new examples, data augmentation, labeling to improve the model performance. What I did was I asked my team to collect more data with con noise in the background and also use data augmentation to synthesize more data with con noise in the background. That's actually what fixes problem relatively efficiently. Today, some of my friends and I, Kai Yang, Daniel Bibirata, we routinely go to our teams and say, hey everyone, the neural network you're using is Good enough. The UNETs, the retina net, the resnet, whatever neural network architecture is good enough. Everyone, let's just hope the code fixed. Let's not mess with the code of network architecture anymore. Let's just keep on working in a systematic way to improve the data quality and let's improve the data to get you the performance you want your system to have. We found for many projects that when we give that direction to our teams, that this allows the team to make faster progress on the machine learning project. Now, one of the themes I keep on coming back to is how can we make this systematic, not just an individual engineer hacking around that spots the problem of car noise and decides to go and collect more data with con noise. So to make it systematic, this is what I tend to do today, which is train a model and then do error analysis to identify the types of data the algorithm does poorly on, such as speech with convoice. And then we would brainstorm or use or sometimes even more systematic ways. Will then ask what type of additional data do you need to improve the performance on that category of data. So you either get more of that data by data augmentation, data generation, data collection, collect more data, or give a more consistent definition for the labels if they were found to be ambiguous. And it turns out that systematic processes like these of error analysis followed by changing the data. This is important because in machine learning we always want more data. But for speech recognition, it's not helpful or it's not nearly as efficient to just say, let's collect a lot more speech data. So having the tools and processes to do the error analysis so that you can, in a much more targeted and efficient way, get more of the data you need, which is more data with speech, with car noise, rather than just more speech data in general. That turns out to be really important for being efficient in how we improve the performance of our systems. All right, let me move on to deploy to production deployment. So today there are some teams that think that when you finally deploy a project in production, that you're at the finish line. I actually tell teams no, sadly, when you deploy in production, well, when you deploy in production, you should celebrate that. Congratulations. And you may be halfway done with your machine learning project, because when you deploy a system in production, the world will change. Concept drift, data drift. You still have to monitor the performance and deployment and build tools to systematically check for concept drift or data drift that can result in performance degradation and then find the right types of data to flow back to the earliest stages of the machine learning project lifecycle to regularly retrain and update the model to engage in continuous learning. And so once again, finding the right data to flow back is, I think, critical to monitor and flow the right data back is critical for monitoring and maintaining your system when it is in production. Now, you've heard me mention many times how to make this systematic. MLOps is a nascent field. And today the websites of large companies have different and slightly inconsistent ways of defining what is mlops. But mlops is, I think, a nascent discipline that is trying to make the development and deployment of machine learning systems systematic. Let Me share with you what I think may be some useful frameworks for you to approach MLOps. In traditional software engineering we needed code to run and so there were software engineers that will write the code and and then DevOps teams that are responsible for quality and infrastructure. After the software engineer maybe not throws the code over the wall, but gradually transitions the code to a DevOps team, the DevOps team is then responsible for managing the data center scaling, making sure that even if there's a spike in traffic that the workload on the system is manageable. There's a network outage, that things stay up. So that's what DevOps teams do. Good DevOps teams do a great job in maintaining this quality and infrastructure keeps things running. Now, with AI we don't just have code, we have code plus data. And so we have these new roles with machine learning engineers building the machine learning models and then also hopefully this nascent discipline to help maintain quality and infrastructure for the data. But there's a big difference, but there's a significant difference between how we deploy machine learning systems or AI systems compared to traditional software systems that implicates how we build MLOps versus DevOps teams. Let me share with you what I mean. Traditional software was relatively more left to right feed forward process. We scope a project, develop a code and deploy in production. And so DevOps teams could be responsible for that. But we've seen that AI software is a more iterative process. We have to develop it. And then there are also these backward pointing arrows. We have to keep on iterating on the data and training model even after production deployment. I found in AI software it works much less well if you train a model and then try to hand it over to a DevOps or MLOps team to just run it for you. Because this stage results in a lot of feedback loops to the earliest stages as well. When I am standing up setting up an mlops team, I think one of the most important things for MLOps team to do is to support ensuring high quality data through all of these stages of the machine learning project life cycle. So the textbook on how to do ML Ops has not yet been written. No one I don't think there's like a set of guiding principles on how to maintain quality and infrastructure for machine learning projects. I think we as a committee are still figuring those things out. In my view, the single most important thing for MLOps team or for MLOps tools to do is to ensure consistently high quality data throughout the machine learning project lifecycle. There are other things the network OPS team can and should do, but it just does this one thing. Then I think the machine learning, development and deployment process can be much more efficient. So during the data collection phase, MLOps teams or tools can help answer how do you define and collect the data so it's consistent during the training phase, MLOps teams can help with well, how do you modify the data systematically and efficiently to improve model performance during deployment? You can ask ML tools or teams can help figure out what data do you need to track concept and data drift and flow the right data back to the earlier parts of the system to improve performance. I have another question for you. Given that given this work of an mlops team to ensure consistently high quality data, I am curious, who do you think is best qualified to take on an MLOps role? And I don't know the right answer to this. I think there's actually no single right answer. I actually honestly don't know the best answer myself. But do you think oops, oh, sorry, I hit the wrong button. Okay, so if you go back to aha slides, that should work. Do you think the MLOps role is best done by a machine learning engineer or data scientist? Or do you think it's best done by a DevOps engineer? Lots of great DevOps engineers, maybe they can expand their role to encompass MLOps work. Or should it be a software engineer? Lots of great software engineers have become machine learning engineers and maybe they could do this too. Or should it be the domain experts? So maybe a manufacturing expert should be responsible for the manufacturing data and a linguist speech expert should be responsible for the speech data. So why don't you go ahead and I'm actually curious what people think because I think that mlops is a nascent and emerging role. So there are probably multiple groups that one could imagine jumping in to this row. Let me share a screen. Let's see what people voted. Cool. All right. Pretty interesting. Okay, cool. I think that a lot of machine learning engineers, data scientists on this call. So I think many of you are eager to have yourself or your team embrace this role. So I think that's great. I think DevOps engineers could certainly take it on. I see less love for the software engineers in this world, but I think software engineers are a little bit further. It's kind of like a two step move rather than a one step move. I think many software engines could do it too. And I do see domain experts playing an important role, a very important role in ML Ops. Because often the domain experts, they understand better than a non domain expert exactly what is the appropriate label and how to debug that process. But I think different people from all of these roles could help with ML Ops. All right, thank you. So I've just two more slides, then I wrap up and take people's questions. Because a lot of AI had grown up in consumer software, Internet companies have a lot of data. There's a lot of hype and excitement about big data. Nothing wrong with that. If you're a lot of data, that can be very helpful. But as AI breaks into other industries, especially once with the data set sizes just aren't as big. I would love to see a shift I think be useful for many of us to shift our mindset from big data to good data and MLOps when we stand at ML Ops. I think most important guiding principle is for the mlops team to help ensure consistently high quality data in all phases. Machine learning, project, lifecycle and so what is good data? Just like food, where it's not about maximizing the number of calories you eat. I think good data is data that is defined consistently. So the definition of labels Y given X is unambiguous covers the important cases. So do you have good coverage? Do you have enough data of speech with car noise in the background or any other important cases has timely feedback from production data. So distribution covers data drift and concept drift and is sized appropriately and other issues that are a little bit more domain specific. In some domains where there are data governance issues, such as in healthcare, where patient privacy, that has to be respected in some problems, in a lot of commercial problems, the issues of bias and fairness like steel defect detection, that's less social impact. But in projects of social impact, good data needs to address those issues obviously and be audited against fairness and bystanders. But I think that if we can in the appropriate domain specific way, which is different for different problems, have MLOps teams help us ensure they have good data, then all of us will really get machine learning projects to work effectively on a lot more problems, including ones where you have fewer than 10,000 which is just smaller data sets but also larger datasets as well. This is my last slide, then we'll go to questions takeaways I hope you will use and this is the first time I'm speaking publicly about these ideas and shaping them for over a year with some of my collaborators. But if you think these ideas are useful, please help me spread the word and help shift people to a data centric AI. But takeaways I think mister's most important task make high quality data Available through all stages of machine learning project lifecycle if we just do this, there's a lot of great models, open source models you can download off GitHub. You can find a code snippet you need on Stack Overflow. You usually can't find a data snippet you need on Stack Overflow or on GitHub because data tends to be much more customized. So if we can ensure consistently high quality flow of data then that together with the code which is also important can help us be more efficient in developing effective machine learning systems and deploying them and maintaining them. AI systems and copeless data For a lot of projects we tend to instinctually take a model centric view. And when you are working on a project, I hope you also consider for your projects a data centric view where you ask how to systematically do error analysis, other types of analysis to change, figure out how to change your data to improve performance. And then finally, I think one of the most important frontiers of AI is going to be MLOps tools to make data centric AI an efficient and systematic process. So it's not counting on the lack of skill of individual engineer to spot the problem and figure out how to fix it, but for us to all have some systematic frameworks of thinking to spot these issues and improve on it. I remember before modern version control my friends and I, we used to write code and then email C files to each other as an email attachment. I will write a piece of code and then take the source file and attach it as an email attachment and send it to my friend and say hey, I'm done editing the code. I won't touch it the next three hours. Why don't you work on it and then email it back to me when you're done. And then they'll do that and then I will manually merge their source code and my source code. Thank goodness we in the software community evolved towards version control where we had a CVS subversion in our git and that made it much more systematic. It feels to me that we are still in the emailing. Code management has become more mature. Data management goes beyond versioning. Versioning is one small piece of it that ensuring good data goes beyond that. And it feels to me like we're still in the emailing stuff, back and forth phase of how we think about data. But so I think this will be an exciting frontier for AI and if we can make MLOps and data centric AI more systematic and efficient, it'll help all of us be more efficient effective in how we build and deploy AI systems. So thank you for sticking with me through this presentation. I'd love to see what questions people have. So if you go to this website, you can enter questions and then vote on the questions, Vote on the questions that you want answered. I'll spend a few minutes to take people's questions before we break at the end of the hour. So let me actually share people's questions. All right, cool. So thank you Vishnu Rajan Teju for your question. How about hybrid approach of model centric? Excuse me? How about a hybrid model of model centric and data centric development? Yes. I find that you don't need to be 100% data centric. Sometimes you do want to improve the model. And in fact there's a lot of cutting edge research where AI work. We do need better models. It's totally great for people to keep on doing research and improving models. What I find is that because we as a community, we've invested so much in improving the models for a lot of problems. I think it's equally useful, maybe even more useful sometimes to work on improving the data. I find that for a lot of applications the model is basically a solved problem. There's a model you can download off open source model you can download off GitHub, that's good enough, but the data needs to be much more custom. But there's really nothing wrong with continuing to improve the model as well. All right, Jack. Data site critical but raw data spin ahead. What's best to generate synthetic relevant synthetic data? You know, I find that there are some really sort of cool, exciting techniques like GAN data augmentation. I've used that and I find that those really exciting techniques is often one of the later techniques I will go to. So I find that there's often simpler ways than, you know, the most exciting cutting edge. Let's use again to generate more data. But instead synthesizing background noise and speech recognition or image warping flips or just going out to collect more data is very problem dependent. I guess maybe I'm stumbling a little bit because I think it's very application dependent. But I found that there are often ways to just collect more data, clean up the data and then eventually you might want to generate synthetic data. But it's usually one of the later techniques I will try. I have done it, but usually not as the first thing I try. They're usually simpler things to get high quality data, improve performance, but it is problem dependent.
