Lecture excerpt explaining key challenges in training ML models, emphasizing the dual components of code (algorithms/models) and data, the importance of data-centric improvements, and the iterative model-development loop (train → error analysis → modify model/data/hyperparameters). It outlines three milestones—doing well on training, dev/validation, and test sets—and warns that high average test-set accuracy can be insufficient for meeting business goals.
Chapters
What is hard about training a machine learning model that does well? Let's look at some key challenges. One framework that I hope you keep in mind when developing machine learning systems is that AI systems and machine learning systems comprise both code, meaning the algorithm or the model, as well as data. There's been a lot of emphasis in the last several decades on how to improve the code. In fact, a lot of AI research had grown up by researchers downloading data sets and trying to find an algorithm or model that does well on the data set. But for many applications, you have the flexibility to change the data if you don't like the data. And so there are many projects where the algorithm or model is basically a solved problem. Some model you download off GitHub would do well enough. So, and it'd be more efficient to spend a lot of your time improving the data because the data usually has to be much more customized to your problem. This is a view that will carry throughout this week and next week's materials, Diving into more detail when building a machine learning system, you may have an algorithm or a model. This would be your code and some data. And it's by training your algorithm on the data that you then have your machine learning model. They can make predictions. And of course hyperparameters are an additional input to this process. It is important for many applications to make sure you have a well tuned learning rate and regularization parameter, and maybe a few other things. The hyperparameters are important, but because the space of hyperparameters is usually relatively limited, I'm going to spend more of our time focusing on the code and on the data. So model development is a highly iterative process. You usually start off with some model and hyperparameters and data. Train a model and then take the model to carry out error analysis and use that to help you decide how to improve the model or the hyperparameters or the data. Because machine learning is such an empirical process, being able to go through this loop many times very quickly is key to improving performance. But one of the things that will help you improve performance too is each time through the loop being able to make good choices about how to modify the data or how to modify the model, or how to modify the hyperparameters. After you've done this enough times and achieve a good model, one last step that's often useful is to carry out a richer error analysis and have your system go through a final audit to make sure that it is working before you push it to a production deployment. So why is model development hard when building a model, I think there are three key milestones that most projects should aspire to accomplish. First is you probably want to make sure you do well, at least on the training set. So you know, if you're predicting housing prices as a function of the size of a house, are you at least able to fit a line that fits your training set quite well? After you've done well on the training set, you then have to ask if your algorithm does well on the development set or the holdout cross validation set, and then also the test set. If your algorithm isn't even doing well on the training set, then it's very unlikely to do well on the dev set or the test set. So I think of step one as something you have to do first as a milestone on your way toward achieving step two, and then after you do well on the dev set or test set, you also have to make sure that your learning algorithm does well according to the business metrics or according to the project's goals. Over the last several decades, a lot of machine learning developments was driven by the goal of doing well on the dev set or test set. Unfortunately for many problems, having a high test set accuracy is not sufficient for achieving the goals of the project, and this has led to a lot of frustration and disagreements between the machine learning team, which is very good at doing this, and business teams which care more about the business metrics or some other goals of a project. So you may be wondering, hey Andrew, how is it possibly true that achieving low average test set error isn't good enough for a project? There are a few common patterns that I've seen across many projects where you need something beyond low average test set error, and be able to spot these issues will help you be more efficient in addressing them. Let's dive more into this topic in the next video.
