#20 Machine Learning Engineering for Production (MLOps) Specialization [Course 1, Week 2, Lesson 12]
Andrew Ng explains data augmentation for images, audio, and text. Use three principles: keep examples realistic, preserve the correct label, and target data your model struggles with. Audio mixing and image transforms like flips and contrast changes work well, and iterating on data often beats retraining the model.
Chapters
Data augmentation can be a very efficient way to get more data, especially for unstructured data problems such as images, audio, maybe text. But when carrying out data augmentation, there are a lot of choices you have to make. What are the parameters? How do you design the data augmentation setup? Let's dive into this to look at some best practices. Take speech recognition. Given an audio clip like this, AI.
Is the new electricity.
If you take background cafe noise that sounds like this. And add these two audio clips together, literally take the two waveforms and sum them up, then you can create a synthetic example that sounds like this.
AI is the new electricity.
So it sounds like someone saying, AI is the new electricity in a noisy cafe. This is one form of data augmentation that lets you efficiently create a lot of data that sounds like data collected in a cafe. Or if you take the same audio.
AI is the new electricity and add.
It to background music, then it sounds like someone saying it with maybe the radio on in the background.
AI is the new electricity.
Now, when carrying out data augmentation, there are a few decisions you need to make. What types of background noise should you use? And how loud should the background noise be relative to the speech? Let's take a look at some ways of making these decisions systematically. The goal of data augmentation is to create examples that your learning algorithm can learn from. As a framework for doing that, I encourage you to think of how you can create realistic examples that the algorithm does poorly on. Because if the algorithm already does well on those examples, then there's less for it to learn from. But you want the examples to still be ones that a human will, or maybe some other baseline can do well on. Because otherwise, one way to generate examples that the algorithm does poorly on would be to just create examples that are so noisy that no one can hear what anyone said. But that's not helpful. You want examples that are hard enough to challenge the algorithm, but not so hard that they're impossible for any human or any algorithm to ever do well on. And that's why when I am generating new examples using data augmentation, I try to generate examples that meets both of these criteria. Now, one way that some people do data augmentation is to generate an augmented dataset and then train the learning algorithm and see if the algorithm does better on the dev set and then fiddle around with the parameters for data augmentation and train the learning algorithm again and so on. This turns out to be quite inefficient because every time you change your Data augmentation parameters. You need to train your neural network or train your learning algorithm all over. And this can take a long time. Instead, I found that using these principles allows you to sanity check that your new data generated using data augmentation is, is useful without actually having to spend maybe hours or sometimes days of training a learning algorithm on that data to verify that it will result in the performance improvement. So specifically, here's a checklist you might go through when you are generating new data. One, does it sound realistic? You want your audio to actually sound like realistic audio of the sort that you want your algorithm to perform on. Two, is the X to Y mapping clear? In other words, can humans still recognize what was said? This is to verify point two here. And three, is the algorithm currently doing poorly on this new data and that helps you verify 0.1. If you can generate data that meets all of these criteria, then that will give you a higher chance that that when you put this data into your training set and retrain the algorithm, that that will result in you successfully pulling up part of this rubber sheet. Let's look at one more example using images. This time, let's say that you have a very small set of images of smartphones with scratches. Here's how you may be able to use data augmentation. You can take the image and flip it horizontally. This results in a pretty realistic image. The foam buttons are now on the other side, but this could be a useful example to add to your training set. Or you could implement contrast changes. I've actually brightened up the image here, so the scratch is a little bit more visible. Or you could try darkening the image. But in this example, the image is now so dark that even I as a person can't really tell if there's a scratch there or not. And so whereas these two examples on top would pass the checklist we had earlier, that the human can still detect the scratch. Well, this example is too dark. It would fail that checklist. And so I would try to choose a data augmentation scheme that generates more examples that look like the ones on top and fewer of the ones that look like the ones here at the bottom. And in fact, going off the principle that we want images that look realistic, that humans can do well on, and hopefully the algorithm does poorly on, you can also use more sophisticated techniques, such as take a picture of a phone with no scratches and use Photoshop in order to artificially draw a scratch. And this technique, literally using Photoshop, can also be an effective way to generate more examples because this Example, with a scratch here, you may or may not be able to see it, depending on the video compression and image contrast when you're watching this video. But with a scratch here, this looks like a pretty realistic scratch. This is actually generated Photoshop and I as a person can recognize this scratch. And so if the learning algorithm isn't detecting this right now, this would be a great example to add. I've also used more advanced techniques like GANs Generative adversarial networks to synthesize scratches like these automatically. Although I found that techniques like that can also be overkill, meaning that there's simpler techniques that are much faster to implement that work just fine without the complexity of building a GAN to synthesize scratches. You may have heard of the term model iteration, which refers to iteratively training a model using error analysis and then trying to decide how to improve the model. Taking a data centric approach to AI development, sometimes it's useful to instead use a data iteration loop where you repeatedly take the data and the model, train your learning algorithm, do error analysis, and as you go through this loop, focus on how to add data or improve the quality of the data. And for many practical applications, taking this data iteration loop approach with a robust hyperparameter search, that's important too. But taking a data iteration loop approach results in faster improvement to your learning algorithms performance, depending on your problem. So when you're working on an unstructured data problem, data augmentation, if you can create new data that seems realistic that that humans can do quite well on, but that the algorithm struggles on, that can be an efficient way to improve your learning algorithm's performance. And so if you found through error analysis that your learning algorithm does poorly on speech with CAFE noise, data augmentation to generate more data with caffeine noise could be an efficient way to improve your learning algorithms performance. Now, when you add data to your system, question I've often often been asked is, can adding data hurt your learning algorithms performance? Usually for unstructured data performance, the answer is no, with some caveats. But let's dive more deeply into this in the next video.
