Data augmentation shifts your training distribution, but for large models on unstructured data with clear input-to-label mappings, adding more labeled data rarely hurts. Small models can overfit to augmented variants, and ambiguous cases like distinguishing the digit 1 from the letter I can skew class priors and tank accuracy.
For a lot of machine learning problems, the training set and def and test set distributions start off being reasonably similar. But if you're using data augmentation, you're adding to specific parts of the training set, such as adding lots of data with cafe noise. So now your training set may come from a very different distribution than the dev set and the test set. Is this going to hurt your learning algorithms performance? Usually the answer is no, with some caveats when you're working on unstructured data problems. But let's take a deeper look at what that really means. If you are working on an unstructured data problem and if your model is large, such as a neural network that is quite large and has large capacity and does large low bias, and if the mapping from X to Y is clear, and by that I mean given only the input X, humans can make accurate predictions, then it turns out adding accurately labeled data rarely hurts accuracy. This is an important observation because adding data through data augmentation or collecting more of one type of data is can really change your input data distribution. The probability of X. Let's say at the start of your problem 20% of your data had caffe noise. But using augmentation you added a lot of caffe noise. So now this is 50% of your data is data with caffeine noise in the background. It turns out that so long as your model is sufficiently large the then it won't stop it from doing a good job on the caffeine noise data as well as doing a good job on non caffeine noise data. In contrast, if your model were small then changing your input data distribution this way may cause it to spend too much of its resources modeling caffeine noise settings and this could hurt its performance on non caffeine noise data. But if your model is large enough then then this isn't really an issue. The second problem that could arise is if the mapping from X to Y is not clear meaning given X, the true label of Y is very ambiguous. This doesn't really happen much in speech recognition, but let me illustrate this with an example from Computer vision. This is very rare, so it's not something I would worry about for most practical problems. But let's see why this is important. One of the systems I had worked on many years ago use Google street view images to read host numbers in order to more accurately geolocate buildings and houses in Google Maps. So one of the things that system did was take us input pictures like this and figure out what is this digit. So clearly this is A1 and this is a Alphabet I. You don't see a lot of eyes in street view images, but there are some buildings. You know, you may see a sign that says navigate to house number 42i. But house numbers really rarely have an Alphabet I in it. Now, if you find that your algorithm has very high accuracy on recognizing 1s, but low accuracy on recognizing I's, one thing you might do is add a lot more examples of I's into your training set. And the problem, and this is a rare problem, is there are some images that are truly ambiguous. Is this A1 or is this an I? And if you were to add a lot of new I's to your training set, especially ambiguous examples like this, then that may skew the data set to have a lot more I's and hurt performance, because we know that there are a lot more ones than I's on Hulst numbers. If the learning algorithm sees a picture like this, it would be safer to guess that this is a one rather than that this is an I. But if data augmentation skews the data set in the direction of having a lot more I's rather than a lot of ones, then it may cause the algorithm to guess poorly on an ambiguous example like this. So this is one rare example where adding more data could hurt performance. And this example of 1 versus I is 1 that contradicts the second bullet, because for some images, the mapping from X to Y is not clear. In particular, given only an image like this on the right, even a human can't really tell what this is. Just to be clear, the example that we just went through together is a pretty rare, almost corner case. And, and it's quite unusual for data augmentation or adding more data to hurt the performance of your learning algorithm. So long as your model is big enough, maybe your neural network is big enough to learn from a diverse set of data sources. But I hope that understanding this rare case where it could hypothetically hurt, gives you more comfort with using data augmentation or collecting more data to improve the performance of your algorithm. And even if it causes your training set distribution to become different from your dev set and test set distribution. So far, our discussion has focused on unstructured data problems. How about structured data problems? It turns out there's a different set of techniques that's useful for structured data. Let's take a look at that in the next video.
