Hyperparameter Tuning in Practice (C2W3L03)

You've now heard a lot about how to search for good hyperparameters before wrapping up our discussion on hyperparameter search, I want to share with you just a couple final tips and tricks for how to organize your hyperparameter search process. Deep learning today is applied to many different application areas and that intuitions about hyperparameter settings from one application area may or may not transfer to a different one. There is a lot of cross fertilization among different applications domains. So for example, I've seen ideas developed in the computer vision community, such as confnets or resnets, which we'll talk about in a later course, successfully applied to speech, and I've seen ideas that were first developed in speech, successfully applied in NLP and so on. So one nice development in deep learning is that people from different application domains do read increasingly research papers from other application domains to look for inspiration for cross fertilization. In terms of your setting for the hyper parameters, though, I've seen that intuitions do get stale. So even if you work on just one problem, say logistics, you might have found a good setting for the hyperparameters and kept on developing your algorithm, or maybe seeing your data gradually change over a course of several months, or maybe just upgraded the servers in your data center. And because of those changes, the best setting of your hyperparameters can get stale. So I recommend maybe just retesting or re evaluating your hyperparameters at least once every several months to make sure that you're still happy with the values you have. Finally, in terms of how people go about searching for hyperparameters, I see maybe two major schools of thought or maybe two major different ways in which people go about it. One way is if you babysit one model. And usually you do this if you have maybe a huge dataset, but not a lot of computational resources, not a lot of CPUs and GPUs. So you can basically afford to train only one model or a very small number of models at a time. In that case, you might gradually babysit that model even as it's training. So for example, on day zero you might initialize your parameters random and then start training. And then and you gradually watch, you know, your learning curve, maybe the cost function J or your dev set error or something else gradually decrease over the first day. Then at the end of day one you might say, gee, it looks like it's learning quite well. I'm gonna try increasing the learning rate a little bit and see how it does. And then maybe does better and that's your day two performance. And after two days you say, okay, it's still doing quite well. Maybe I'll further the momentum term a bit or decrease the learning rate a little bit now. And then you're now into day three, and every day you kind of look at it and you'll try nudging up and down your parameters. And maybe on one day you found your learning rate was too big, so you might go back to the previous day's model and so on. We're kind of babysitting the model one day at a time, even as a training over a course of many days or over the course of several different weeks. So that's one approach. And people that babysit one model that is watching a performance and, you know, patiently nudging the learning rate up or down. That's usually what happens if you don't have enough computational capacity to train a lot of models at the same time. The other approach would be if you train many models in parallel. So you might have some setting of the hyperparameters and just let it run by itself either for a day or even for multiple days. You may get some learning curve like that. And this could be a plot of the cost function J, or cost of your training error, or cost of your dev set error, but some metric that you're tracking. And then at the same time, you might start up a different model with a different setting of the hyperparameters. And so your second model might generate a different learning curve, maybe one that looks like that or looks like that one looks better. And at the same time, you might train a third model, which might generate a learning curve that looks like that and another one that maybe this one diverges. So it looks like that, and so on. But you might train many different models in parallel, where these orange lines are different models. Right? And so this way you can try a lot of different hyperparameter settings, and then just maybe quickly at the end, pick the one that works best. Looks like in this example, it was maybe this curve that looked best. So to make an analogy, I'm gonna call the approach on the left the panda approach. You know, when pandas have children, they have very few children, usually one child at a time. And then they really put a lot of effort into making sure that the baby panda survives. So that's really babysitting, you know, one model or one baby panda. Whereas the approach on the right is more like what fish do. I'm going to call this a caveat strategy. There's some fish that lay over 100 million eggs in one season, in one mating season. But the way fish reproduce is they lay a lot of eggs and don't pay too much attention to any one of them. But, you know, just see that hopefully one of them or maybe a bunch of them will do well. So I guess this is really the difference between how mammals reproduce versus how fish and a lot of reptiles reproduce. But I'm going to call it the panda approach versus the caviar approach, since that's more fun and memorable. So the way to choose between these two approaches is really a function of how much computational resources you have. If you have enough computers to train a lot of models in parallel, then by all means take the caviar approach and try a lot of different hyperparameters and see what works. But in some application domains, I see this in some online advertising settings as well as in some computer vision applications where there's just so much data and the models you want to train are so big that it's difficult to train a lot of models at the same time. It's really application dependent, of course, but I've seen those communities use the panda approach a little bit more, where you are kind of babying a single model along and nudging the parameters up and down and trying to make this one model, you know, work. Although of course, you know, the panda approach, having trained one model and seen it work or not work, maybe in the second week or the third week, maybe you actually initialize a different model and then, you know, baby that one along. Just like even pandas, I guess, can have multiple children in their lifetime, even if they have only one or a very small number of children, and at any one time. So hopefully this gives you a good sense of how to go about the hyperparameter search process. Now, it turns out that there's one other technique that can make your neural network much more robust to the choice of hyperparameters. Doesn't work for all neural networks, but when it does, it can make the hyperparameter search much easier and also make training go much faster. Let's talk about this technique in the next video.