A short lecture (YouTube) on data augmentation for computer vision. The speaker emphasizes that most CV tasks benefit from more data and describes common augmentation techniques: horizontal mirroring, random cropping, geometric distortions (rotation, scaling, local warping), and color shifts. The talk mentions PCA color augmentation (as in the AlexNet paper), practical implementation patterns (on-the-fly distortions using CPU threads loading images from disk and feeding training threads/GPU), and that augmentation has hyperparameters which can be initialized from open-source implementations.
Chapters
Most computer vision tasks could use more data. And so data augmentation is one of the techniques that is often used to improve the performance of computer vision systems. I think that computer vision is a pretty complicated task. You have to input this image, all these pixels, and then figure out what is in this picture. And it seems like you need to learn a decently complicated function to do that. And in practice, for almost all computer vision tasks, having more data will help. This is unlike some other domains where sometimes you can get enough data that you don't feel as much pressure to get even more data. But I think today the state of computer vision is that for the majority of computer vision problems, we feel like we just can't get enough data. And this is not true for all applications of machine learning, but it does feel like it's true for computer vision. So what that means is that when you're training a computer vision model, often data augmentation will help. And this is true whether you're using transfer learning and using someone else's pre trained weights to start, or whether you're trying to train something yourself from scratch. Let's take a look at the common data augmentation methods in computer vision. Perhaps the simplest data augmentation method is mirroring on the vertical axis, where if you have this example in your training set, you flip it horizontally to get that image on the right. And for most computer vision tasks, if the left picture is a cat, then mirroring it is still a cat. And if the mirroring operation preserves whatever you're trying to recognize in the picture, this would be a good data augmentation technique to use. Another commonly used technique is random cropping. So given this data set, let's pick a few random crops. So you might pick that and take that crop, uh, you might take that, take that crop, take this, take that crop. And so this gives you different examples to feed into your training set with sort of different random crops of your data set. So random cropping isn't a perfect data augmentation method. You know what if you randomly end up taking that crop, which won't look much like a cat, but in practice it works well so long as your random crops are reasonably large subsets of the original image. So mirroring and random cropping are frequently used. And in theory you could also use things like rotation, sharing of the image. So that's if you do this to the image, distort it that way, introduce various forms of local warping and so on, and there's really no harm with trying all of These things as well, although in practice they seem to be used a bit less, perhaps because of the complexity. A second type of data augmentation that is commonly used is color shifting. So given a picture like this, let's say you add to the R, G and B channels different distortions. In this example, we are adding to the red and blue channels and subtracting from, from the green channel. So red and blue make purple. So this makes the whole image a bit more purpley and that creates a distorted image. For your training set, um, for illustration purposes, I'm making somewhat dramatic changes to the colors. In practice you draw R, G and B from some distribution, uh, that could be quite small as well. But what you do is take different values of R, G and B and use them to distort the color channels. So, so in the second example we are making it less red and more green and more blue. So that turns the whole image a bit more yellowish. And here we are making it much more blue, just a tiny little bit more red. But in practice, the values R, G and B are drawn from some probability distribution. And the motivation for this is that if maybe the sunlight was a bit yellower or maybe the indoor illumination was a bit more yellow, that could easily change the color of an image. But the identity of the cat or the identity of the content, the label Y should still stay the same. And so introducing these color distortions or by doing color shifting, this makes your learning algorithm more robust to changes in the colors of your images. Um, just a comment for the advanced learners in this course. Um, and it's okay if you don't understand what I'm about to say when I use it in red. There are different ways to sample R, G and B. One of the ways to implement color distortion uses an algorithm called pca. This is called Principals Components Analysis, which, um, I taught about in the Machine Learning MOOC in the mlclass.org machine learning course on Coursera. But the details of this are actually given in the Alexnet paper. And it's called sometimes called PCA color augmentation. But the rough idea between PCA color augmentation is, for example, if your image is mainly purple, if it mainly has red and blue tints and very little green, then PCA color augmentation will add and subtract a lot to red and blue, but relatively little to green. So it kind of keeps the overall color or the tint the same. If you didn't understand any of this, don't worry about it. But if you can search online for but you can, if you want, read about the details of it in the Alexnet paper. And you can also find some open source implementations of PCA color augmentation and just use that. So you might have your data stored, your training data stored on a hard disk, when they use this symbol, this round bucket symbol to represent your hard disk. And if you have a small training set, you could do almost anything and it'll be okay. But if you have a very large training set, then this is how people will often implement it, which is you might have a CPU thread that is constantly loading images off your hard disk. So you have this stream of images coming in from your hard disk. And what you can do is use maybe a CPU thread to implement the distortions, be it the random cropping or the color shifting or the mirroring. But for each image you might then end up with some distorted version of it. So let's see this image, I'm going to mirror it and maybe also implement color distortion and so on. And maybe this image ends up being color shifted, so you end up with, you know, some different color cat. And so your CPU thread is constantly loading data, as well as implementing whatever distortions are needed to form a batch or really mini batches of data. And this data is then constantly passed to some other thread or some other process for implementing training. And this could be done on the CPU or really increasingly on the GPU if you have a large neural network to train. And so a pretty common way of implementing data augmentation is to really have, you know, one thread or multiple threads that is responsible for loading the data and implementing distortions and then passing that to some other thread or some other process that then does the training. And often this and this can run in parallel. So that's it for data augmentation. And similar to other parts of training a deep neural network, the data augmentation process also has a few hyper parameters, such as how much color shifting do you implement and exactly what parameters you use for random cropping. So similar to elsewhere in computer vision, a good place to get started might be to use someone else's open source implementation for how they use data augmentation. But of course, if you want to capture more invariances that you think someone else's open source implementation isn't, it might be reasonable to also tune these hyperparameters yourself. So with that, I hope that you'll be able to use data augmentation to get your computer vision applications to work better.
