#27 Machine Learning Engineering for Production (MLOps) Specialization [Course 1, Week 3, Lesson 3]

I'd like to share with you a useful framework for thinking about different major types of machine learning projects. It turns out that the best practices for organizing data for one type can be quite different than the best practices for a totally different type. Let's take a look at what are these major types of machine learning projects. Let's fill in this 2x2 grid. One axis will be whether your machine learning problem uses unstructured data or structured data. I found that the best practices for these are very different, mainly because humans are great at processing unstructured data like images and audio and text, and not as good at processing structured data like database records. The second axis is the size of your data set. Do you have a relatively small data set or do you have a very large data set? There's no precise definition of what exactly is small and what is large, but I'm gonna use as a slightly arbitrary threshold whether you have over 10,000 examples or not. And clearly this boundary is a little bit fuzzy and the transitions from small to big data sets is a gradual one. But I found that best practices for if you have, say 100 or a thousand examples, smaller data sets is pretty different than when you have a very large data set. And the reason I chose to number 10,000 is that's roughly the size beyond which it becomes quite painful to examine every single example yourself. You know, if you have a thousand examples, you could probably examine every example yourself. But. But when you have 100,000 examples, 10,000, 100,000 million examples, it becomes very time consuming for you as an individual or maybe a couple of machine learning engineers to manually look at every example. So that affects the best practices as well. Let's look at some examples. If you are training a manufacturing visual inspection from, from just 100 examples of scratch phones, that's unstructured data because this is image data and it's a pretty small data set. If you are trying to predict housing prices based on the size of the house and other features of the house from just 50 training examples, then there's a structured data set with just real value features and a relatively small data set. If you are carrying out speech recognition from 50 million training examples, that's unstructured data. But you have a lot of data. Or if you are trying to recommend products, so online shopping recommendations and you have a million users in your database, then there's a structured problem with relatively large amount of data. For a lot of unstructured data problems, people can help you to label data and data augmentation, such as synthesizing new images or synthesizing new audio. And there's some emerging techniques for synthesizing new text as well. But data augmentation can help. So for manufacturing visual inspection, you can use data augmentation to maybe generate more pictures of smartphones. Or for speech recognition, data augmentation can help you synthesize audio clips with different background noise. In contrast, for structured data problems, it can be harder to obtain more data and also harder to use data augmentation. If only 50 houses have been sold recently in that geography, well, it's hard to synthesize new houses that don't exist. Or if you have a million users in your database, well, again, it's hard to synthesize new users that don't really exist. And it's also harder, not impossible, still worth trying. But it may or may not be possible to get humans to label the data. So I find that the best practices for unstructured versus structured data are quite different. The second axis is the size of your dataset. When you have a relatively small data set, having clean labels is critical. If you have 100 training examples, then if just one of the examples is mislabeled, that's 1% of your data set. And because the data set is small enough for you or a small team to go through it efficiently, it may well be worth your while to go through that hundred examples and make sure that every one of those examples is labeled in a clean and consistent way, meaning according to a consistent labeling standard. In contrast, if you have a million data points, it can be harder, maybe impossible, for a small machine learning team to manually go through every example. Having clean labels is still very helpful. Don't get me wrong, even when you have a lot of data, clean labels is better than non clean ones. But because of the difficulty of having the machine learning engineering team go through every example, the emphasis is on data processes in terms of how you collect and store the data, the labeling instructions you may write for a large team of crowdsourced labelers. And once you have executed some data process, such as asked a large team of labelers to label a large set of audio clips, it can also be much harder to go back and change your mind and get everything relabeled. So let's summarize. For unstructured data problems, you may or may not have a huge collection of unlabeled examples. X maybe in your factory you actually took many thousands of images of smartphones, but you just haven't bothered to label all of them yet. This is also common in the self driving car industry, where many self driving car companies have collected tons of images of cars driving around, but just have not yet gotten that data labeled. For these unstructured data problems, you can sometimes get more data by taking your unlabeled data X and asking humans to just label more of it. This doesn't apply to every problem, but for the problems where you do have tons of unlabeled data, this can be very helpful. And as we've already mentioned, data augmentation can also be helpful. For structured data problems, it's usually harder to obtain more data because you only have so many users or only so many houses or so that you can collect data from. And human labeling on average is also harder, although there are some exceptions, such as in the last video where you saw that we could try to ask people to label examples for the user ID merge problem. But in many cases where we ask humans to label structured data, even when it's a completely fine idea, completely worthwhile to ask people to try to label, if two records are the same person, there's more likely to be a little bit more ambiguity, where even a human laborer sometimes finds it hard to be sure what is the correct label. Lastly, let's look at small versus big data, where I use the slightly arbitrary threshold of whether you have more or less than, say 10,000 labeled training examples. For small data sets, clean labels are critical, and the data set may be small enough for you to manually look through the entire data set and fix any inconsistent labels. Further, the labeling team is probably not that large. It may be one or two or just a handful of people, but that created all the labels. So if you discover an inconsistency in the labels, say one person labeled iguanas one way and a different person labeled iguanas a different way, you can just get the two or three labels together and have them talk to each other and hash out and agree on one labeling convention. For the very large data sets, the emphasis has to be on data process. And if you have hundred labelers or even more, it's just harder to get a hundred people into a room to all talk to each other and hash out a process. And so you might have to rely on a smaller team to establish a consistent label definition and then share that definition with all, say hundred or more labelers and ask them to all implement the same process. I want to leave you with one last thought, which is that I found this categorization of problems into unstructured versus structured small versus big data. I found this to be helpful for predicting not just whether data processes generalize from one to another problem, but also whether other machine learning ideas generalize from one to another. So one tip, if you are working on a problem from one of these four quadrants, then on average, advice from someone that has worked on problems in the same quadrants will probably be more useful than advice from someone that's worked in a different quadrant. I found also in hiring machine learning engineers, someone that's worked in the same quadrant as the problem I'm trying to solve will usually be able to adapt more quickly to working on other problems in that quadrant because the instincts and decisions are more similar within one quadrant than if you shift to a totally different quadrant in this chart. I've sometimes heard people give advice like oh, if you're building a computer vision system, always get at least 1,000 labeled examples. And I think people that give advice like that are well meaning and I appreciate that they're trying to give good advice. But I found that advice to not really be useful for all problems. Machine learning is very diverse and it's hard to find one size fits all advice like that. I've seen computer vision problems built with 100 examples or 100 examples per class. I've seen systems built with 100 million examples. And so if you are looking for advice on a machine learning project, try to find someone that's worked in the same quadrant and as the problem you're trying to solve. Now, we talked about one formulation of different types of machine learning problems. There's one aspect I would like to dive into with you in the next video, which is how for small data problems, having clean data is especially important. Let's take a look at the next video of why this is true.