Train/Dev/Test Set Distributions (C3W1L05)

The way you set up your training, dev or development sets and test sets can have a huge impact on how rapidly you or your team can make progress on building a machine learning application. But I've seen teams, even teams in very large companies, set up these data sets in ways that really slows down rather than speeds up the progress of the team. Let's take a look at how you can set up these data sets to maximize your team's efficiency. In this video, I want to focus on how you set up your dev and test sets. So the dev set is also called the development set, or sometimes called the holdout cross validation set. And the workflow in machine learning is that you try a lot of ideas, train up different models on the training set, and then use the dev set to evaluate the different ideas and pick one and keep iterating to improve dev set performance until finally you have one classifier you're happy with, that you then evaluate on your test set. Now, let's say by way of example that you're building a CAD classifier and you are operating in these regions in the us, uk, other European countries, South America, India, China, other Asian countries, and Australia. So how do you set up your dev set or and your test set? Well, one way you could do so is to pick, pick four of these regions. I'm going to use these four, but it could be four randomly chosen regions and say that data from these four regions will go into the dev set and the other four regions, I'm just going to use these four because we've randomly chosen four as well, that those will go into the test set. It turns out this is a very bad idea because in this example, your depth and test sets come from different distributions. I would instead recommend that you find a way to make your def and test sets come from the same distribution. So here's what I mean. One picture to keep in mind is that I think setting up your dev set plus your single row number evaluation metric, that's like placing a target and telling your team where you think is the bullseye you want to aim at. Because what will happen once you've established the dev set in the metric is that the team can iterate very quickly, try out different ideas, run experiments, and very quickly use the dev set and the metric to evaluate classifiers and try to pick the best ones. So machine learning teams are often very good at shooting different arrows at the target and iterating to get closer and closer to hitting the bullseye, to doing well on your metric, on your dev set, and the problem with how we've set up the dev and test set in the example on the left is that your team might spend months iterating to do well on the dev set, only to realize that when you finally go to test set that that data from these four countries or these four regions at the bottom might be very different than the regions in your dev set. So you might have a nasty surprise and realize that all the months of work you spend optimizing to the dev set is not giving you good performance on the test set. So having depth and test sets from different distributions is like setting a target. Having your team spend months trying to aim closer and closer to bullseye, only to realize after months of work that you say, oh wait, to test it, I'm going to move the target over here. And the team might say, well, why don't you make us spend months optimizing for a different bullseye, when suddenly you're going to move the bullseye to a different location somewhere else. So to avoid this, what I recommend instead is that you take all this data, but randomly shuffle the data into the dev and test set so that both the dev and test sets have data from all eight regions and that the dev and test sets really come from the same distribution, which is a distribution of all of your data mixed together. Here's another example. This is actually a true story, but with some details changed. So I know a machine learning team that actually spent several months optimizing on a dev set which was comprised of loan approvals for medium income zip codes. So the specific machine learning problem was given an input X about a loan application, given can you predict Y which is whether or not they'll repay the loan. So this helps you decide whether or not to approve a loan. And so the dev set came from loan applications, they came from medium income zip codes. Zip codes is what we call postal codes in the United States. But after working on this for a few months, the team then suddenly decided to test this on data from low income zip codes or low income postal codes. And of course the distribution of data from medium income and low income zip codes is very different. And the classifier that they spent so much time optimizing in the former case just didn't work well at all on the latter case. And so this particular team actually wasted about three months of time and had to go back and really redo a lot of work. And what happened here was the team spent three months aiming for one target. And then after three months, the manager asked, oh, how are you doing on hitting this other target, this is a totally different location and it just was a very frustrating experience for the team. So what I recommend for set up a dev set and test set is choose a dev set and test set to reflect data you expect to get in the future and consider important to do well on. And in particular, the dev set and the test set here should come from the same distribution. So whatever type of data you expect to get in the future and want to do well on, try to get data that looks like that and whatever that data is, put it into both your dev set and your test set. Because that way you're putting the target where you actually want to hit and having the team iterate very efficiently to hitting that same target. Hopefully the same target. Well, you notice we haven't talked yet about how to set up a training set. We'll talk about the training set in a later video. But the important takeaway from this video is that setting up the dev set, as well as the evaluation metric is really defining what target you want to aim at. And hopefully by setting the dev set and the test set to the same distribution, you're really aiming at whatever target you hope your machine learning team will hit. The way you choose your training set will affect how well you can actually hit that target. But we can talk about that separately in a later video. So I know some machine learning teams that could literally have saved themselves months of work if they follow the guidelines in this video. So I hope these guidelines will help you too. Next, it turns out that the size of your dev and test sets, how to choose the size of them, is also changing in the era of deep learning. Let's talk about that in the next video.