How to split data between training, dev, and test sets. With a million examples, put 98% in training and 1% each in dev and test. The old 70/30 rule doesn't work anymore; size your dev and test sets by how much confidence you need.
Approximately 98% training, 1% dev, 1% test (i.e., ~10,000 examples for dev and ~10,000 for test).
10,000 examples.
The 70/30 (training/test) split rule-of-thumb.
Like a thousand examples maybe, you know, if you had 10,000 examples. These heuristics are not unreasonable. But in the modern machine learning era, we're now used to working with much larger data set sizes. So let's say you have a million training examples. It might be quite reasonable to set up your data so that you have 98% in the training set and 1% DEF and 1% test. I'm going to use D and T to abbreviate the DEF and test sets, because if you have a million examples, then 1% of that is 10,000 examples. And that might be plenty enough for a dev set or for a test set. So in the modern deep learning era, where sometimes we have much larger data sets, it's quite reasonable to use a much smaller than 20 or 30% of your data for a dev set or a test set. And because deep learning algorithms have such a huge hunger for data, I'm seeing that for problems we have large data sets that a much larger fraction of it goes into the training set. So how about the test set? Remember, the purpose of your test set is that after you finish developing a system, the test set helps you evaluate how good your final system is. So the guideline is to set your test set to be big enough to give high confidence in the overall performance of your system. So unless you need to have a very accurate measure of how well your final system is performing, maybe you don't need millions and millions of examples in your test set. Um, and maybe for your application, if you think that having 10,000 examples gives you enough confidence in the final performance, or maybe 100,000 or whatever it is, that might be enough. And this could be much less than, say, 30% of your overall data set, depending on how much data you have. For some applications, maybe you don't need a high confidence in the overall performance of your final system. Maybe all you need is a train and dev set. And I think not having a test set might be okay. In fact, what sometimes happened was people were talking about using train test splits, but what they were actually doing was iterating on the test set. So rather than a test set, what they had was a train dev split and no test set. If you're actually tuning to this set, to this dev set or this test set, it's better to call the dev set. Although I think in the history of machine learning, not everyone has been completely clean and completely rigorous about calling it a dev set when it really should be treated as a dev set. But if all you care about is having some data to train on and having some data to tune to, and you're just going to ship the final system regardless and not worry too much about how it's actually doing. I think it'd be healthier to just call the trained dev set and acknowledge that you have no test set. This is a bit unusual. I'm definitely not recommending not having a test set when I'm building a system. I do find it reassuring to have a separate test set you can use to get an unbiased estimate of how well it's doing before you ship it. But maybe in the but if you have a very large dev set so that you think you won't overfit the dev set too badly, maybe it's not totally unreasonable to just have a trained dev set. Although it's not what I usually recommend. So to summarize, in the era Big data, I think the old rule of thumb of a 7030 split, that no longer applies. And the trend has been to use more data for training and less for depth and test, especially when you have very large data sets. And the rule of thumb is really to try to set the dev set big enough for its purpose, which is help you evaluate different ideas and pick is algorithm A or B better? And the test set. The purpose of test set is to help you evaluate your final classifiers. You just have to set your test set big enough for that purpose and that could be much less than 30% of your data. So I hope that gives some guidance and some suggestions on how to set up your dev and test sets in the deep learning era. Next, it turns out that sometimes partway through a machine learning problem, you might want to change your evaluation metric or change your dev and test sets. Let's talk about when you might want to do that.
