Parameters vs Hyperparameters (C1W4L07)

Being effective in developing your deep neural nets requires that you not only organize your parameters well, but also your hyperparameters. So what are hyperparameters? Let's take a look. So the parameters of your model are W and B. And there are other things you need to tell your learning algorithm, such as the learning rate alpha, because you need to set alpha, and that in turn will determine how your parameters evolve. Or maybe the number of iterations of gradient descent you carry out. Your learning algorithm has other, you know, numbers that you need to set, such as the number of hidden layers. So we call that capital L or the number of hidden units, right? Such as, you know, N1, N2, and so on. Um, and then you also have the choice of activation function. Do you want to use a relu or a tanh or a sigmoid or something? Especially in the hidden layers. And so all of these things are things that you need to tell your learning algorithm. And so these are parameters that control the ultimate parameters, W and B. And so we call all of these things below hyperparameters, because these things like alpha, the learning rate, the number of iterations, number of hidden layers, and so on, these are all parameters that control W and B. So we call these things hyperparameters because it is the hyperparameters that, you know, somehow determine the final value of the parameters W and B that you end up with. In fact, deep learning has a lot of different hyperparameters. Um, later in a later course, we'll see other hyperparameters as well, such as the momentum term, uh, the mini batch size, um, various forms of regularization parameters, and so on. And if none of these terms at the bottom make sense yet, don't worry about it. We'll talk about them in the second course. Because deep learning has so many hyperparameters, in contrast to earlier eras of machine learning, I'm going to try to be very consistent in calling the learning rate alpha a hyperparameter, rather than calling it a parameter. I think in earlier eras of machine learning, when we didn't have so many hyperparameters, most of us used to be a bit sloppier and just call alpha a parameter. And technically alpha is a parameter, but it's a parameter that determines the, the real parameters. I've tried to be consistent in calling these things like alpha, the number of iterations, and so on, hyperparameters. So when you're training a deep net for your own application, you find that there may be a lot of possible settings for the hyper Parameters that you need to just try out. So applied deep learning today is a very empirical process where often you might have an idea. For example, you might have an idea for the best value for the learning rate. You might say, well, Maybe Alpha equals 0.01, I wanna try that. Then you implement it, try it out, and then see how that works. And then based on that outcome, you might say, you know what, I've changed my mind. I wanna increase the learning rate to 0.05. And so if you're not sure what's the best value for the learning rate to use, you might try one value of the learning rate alpha and see the cost function j go down like this. Then you might try a larger value for the learning rate alpha and see the cost function blow up and diverge. Then you might try another ver and see it go down really fast but converge to higher value. You might try another version and see it, you know, see the cost function J do that. Then after trying to set the values, you might say, okay, it looks like this. The value of alpha gives me pretty fast learning and allows me to converge to a lower cost function j. So I'm going to use this value of alpha. You saw in the previous slide that there are a lot of different hyperparameters. And it turns out that when you're starting on a new application, I actually find it very difficult to know in advance exactly what's the best value of the hyperparameters. So what will often happen is you just have to try out many different values and go around this cycle, you know, try out some value, maybe you try five hidden layers with this many number of hidden units, implement that, see if it works, and then iterate. So the title of the slide is that applied deep learning is a very empirical process. An empirical process is maybe a fancy way of saying you just have to try out a lot of things and see what works. Another effect I've seen is that deep learning today is applied to so many problems ranging from computer vision to speech recognition to natural language processing, to a lot of structured data applications such as maybe online advertising or web search or product recommendations and so on. And what I've seen is that first I've seen researchers from one discipline, any one of these, try to go to a different one. And sometimes the intuitions about hyperparameters carries over and sometimes it doesn't. So I often advise people, especially when starting on a new problem, to just try out a range of values and see what works. And the next course we'll See a systematic way. We'll see some systematic ways for trying out a range of values, right? And second, even if you're working on one application for a long time, you know, maybe you're working on online advertising as you make progress on the problem, it's quite possible that the best value for the learning rate or the number of hidden units and so on might change. So even if you've given tune your system to the best value of hyperparameters today, it's possible you find that the best value might change a year from now, maybe because the compute infrastructure, be it CPUs or the type of GPU you're running on, or something has changed. But, so maybe one rule of thumb is, you know, every now and then, maybe every few months, if you're working on a problem for an extended period of time, for many years, just try a few values for the hyperparameters and double check if there's a better value for the hyperparameters. And as you do so, you slowly gain intuition as well about the hyperparameters that work best for your problem. And I know that this might seem like an unsatisfying part of deep learning, that you just have to try a lot of values for these hyperparameters. Uh, but maybe this is one area where deep learning research is still advancing and maybe over time we'll be able to give better guidance for the best hyperparameters to use. But it's also possible that because CPUs and GPUs and networks and datasets are all changing, and it is possible that the guidance won't converge for some time, and you just need to keep trying out different values and evaluate them on a hodakkar validation set or something and pick the value that works for your problem. So that was a brief discussion of hyperparameters. In the second course, we'll also give some suggestions for how to systematically explore the space of hyperparameters. Um, but by now you actually have pretty much all the tools you need to do the programming exercise. Before you do that, just to share with you one more set of ideas, which is I often am ask what does deep learning have to do with the human brain?