Andrew Ng ranks hyperparameters by importance: learning rate matters most, then momentum, minibatch size, and hidden units. Use random sampling instead of grid search to explore the space more effectively, then zoom in on promising regions.
The learning rate (alpha).
beta1 = 0.9; beta2 = 0.999; epsilon = 1e-8.
Hi, and welcome back. You've seen by now that training a neural net can involve setting a lot of different hyperparameters. Now, how do you go about finding a good setting for these hyperparameters? In this video, I want to share with you some guidelines, some tips for how to systematically organize your hyperparameter tuning process, which hopefully will make it more efficient for you to converge on a good setting of the hyperparameters. One of the painful things about training deep nets is the sheer number of hyperparameters you have to deal with, ranging from the learning rate alpha to to the momentum term beta. They're using momentum or the hyper parameters for the ADAM optimization algorithm, which were beta one, beta two and epsilon. Um, maybe you have to pick the number of layers, maybe you have to pick the number of hidden units for the different layers. And maybe you want to use learning rate decay so you don't just use a single, um, learning rate alpha. And then of course, you might need to choose the mini batch size. So it turns out some of these hyperparameters are more important than others. For most learning applications, I would say alpha, the learning rate is the most important hyperparameter to tune other than alpha. A few other hyperparameters I tend to would maybe tune. Next would be maybe the momentum term. I say 0.9 is a good default. I'd also tune the mini batch size to make sure that the optimization algorithm is running efficiently. Often I'll also fiddle around with the hidden units of the ones I've circled in orange. These are really the three that we consider second in importance to the learning rate, alpha, and then third in importance. You know, after fiddling around with the others, the number of layers can sometimes make a huge difference and, and so can learning rate decay. And then when using the Adam algorithm, I actually pretty much never tune beta 1, beta 2 and epsilon. Pretty much always use 0.9, 0.999 and 10 to the minus 8. Although you can try tuning those as well if you wish. But hopefully this gives you some rough sense of what hyperparameters might be more important than others. Um, alpha most important for sure, followed maybe by the ones I've circled in orange, followed maybe by the ones I circled in purple. But this isn't a hard and fast rule, and I think other deep learning practitioners may well disagree with me or have different intuitions on these. Now, if you're trying to tune some set of hyperparameters, how do you select a set of values to explore in earlier generations of machine Learning algorithms. If you had two hyperparameters, which I'm calling hyperparameter one and hyperparameter two here, it was common practice to sample the points, you know, in a grid like so, and systematically explore, um, these values. Here I'm placing down a five by five grid. Um, in practice it could be more or less than a five by five grid. But you try out in this example, all 25 points, and then, you know, pick whichever hyperparameter works best. And this practice worked okay when the number of hyperparameters was relatively small. In deep learning, what we tend to do, and what I recommend you do instead is choose the points at random. So go ahead and, you know, choose maybe the same number of points, right, 25 points, and then try out the hyperparameters on this randomly chosen set of points. And the reason you do that is that it's difficult to know in advance which hyperparameters are going to be the most important for your problem. And as you saw in the previous slide, some hyperparameters are actually much more important than others. So to take an example, let's say hyperparameter one turns out to be alpha, the learning rate. And to take an extreme example, let's say that hyperparameter 2 was that value epsilon that you have in the denominator of the ADAM algorithm. So your choice of alpha matters a lot, and your choice of epsilon hardly matters. So if you sample on a grid, then you've really tried out five values of alpha, and you might find that all of the different values of epsilon give you essentially the same answer. So you've now trained 25 models and only gotten to try out five values for the learning rate alpha, which is the thing that's really important. Whereas in contrast, if you were to sample at random, then you, you know, you will have tried out 25 distinct values of the learning rate alpha, and therefore you'd be more likely to find a value that works really well. I've explained this example using just two hyperparameters. In practice, you might be searching over many more hyperparameters than this. So if you have, say, three hyperparameters, I guess instead of searching over a square, you're searching over a cube, where this third dimension is hyperparameter three. And then by sampling within this, you know, three dimensional cube, you get to try out a lot more values of each of your three hyperparameters. And in practice, you might be searching over even more hyperparameters than three. And sometimes it's just hard to know in advance which ones turn out to be the really important hyperparameters for your application. And sampling at random rather than in a grid ensures that you're more richly exploring the set of possible values for the most important hyperparameters, whatever they turn out to be. When you sample hyperparameters, another common practice is to use a course to find sampling scheme. So let's say in this two dimensional example that you sample these points and maybe you found that this point worked the best, and maybe a few other points around it tended to work really well. Then in the course defined scheme, what you might do is zoom in to a smaller region of the hyperparameters and then sample more densely within this space. Um, maybe again at random, but to then focus more resources on searching within this, um, blue square if you're suspecting that the best setting of the hyperparameters may be in this region. So after doing a coarse sample of this entire square, if that tells you to then focus on on a smaller square, you can then sample more densely in this smaller square. So this type of a course to find search is also frequently used. And by trying out these different values of the hyperparameters, you can then pick whatever value allows you to do best on your training set objective, or does best on your development sets, or whatever you're trying to optimize in your hyperparameter search process. So I hope this gives you a way to more systematically organize your hyperparameter search process. The two key takeaways are use random sampling, not a grid search. And consider optionally, but consider implementing a course defined search process. But there's even more to hyperparameter search than this. Let's talk more in the next video about how to choose the right scale on which to sample your hyperparameters.
