Learning Rate Decay (C2W2L09)

One of the things that might help speed up your learning algorithm is to slowly reduce your learning rate over time. We call this learning rate decay. Let's see how you can implement this. Let's start with an example of why you might want to implement learning rate decay. Suppose you're implementing mini bash gradient descent with a reasonably small mini bash. Maybe your mini bash has just 64, 128 examples. Then as you iterate, your steps will be a little bit noisy and it will tend toward this minimum over here, but it won't exactly converge. But your algorithm might just end up wandering around, um, and never really converge because you're using some fixed value for alpha and there's just some noise in your different mini batches. But if you were to slowly reduce your learning rate alpha, then during the initial phases, while your learning rate alpha is still large, you can still have relatively fast learning. But then as alpha gets smaller, your steps you take will be slower and smaller. And so you end up oscillating in a tighter region around this minimum rather than wandering far away, even as training goes on and on. So the intuition behind slowly reducing alpha is that maybe during the initial steps of learning, you could afford to take your much bigger steps. But then as learning approaches convergence, then having a slower learning rate allows you to take smaller steps. So here's how you can implement learning rate decay. Recall that one epoch is one pass through the data, right? So if you have a training set as follows, maybe you break it up into different mini batches. Then one, the first pass through the training set is called the first epoch, and then the second pass is the second epoch, and so on. So one thing you could do is set your learning rate alpha to be equal to one over one plus a parameter which I'm gonna call the decay rate times the epoch num. And this is gonna be times some initial learning rate Alpha 0. Note that the decay rate here becomes another hyperparameter which you might need to tune. So here's a concrete example. Um, if you take several epochs, so several passes through your data, if alpha 0 is equal to 0.2 and the decay rate is equal to 1, then during your first epoch alpha will be 1 over 1 plus 1 times alpha 0. So your learning rate will be 0.1. That's just, you know, evaluating this formula. When the decay rate is equal to 1 and the epoch norm is 1 on the second epoch, your learning rate decays to 0.67 on the third, 0.5 on the fourth, 0.4, and so on. And feel free to evaluate more of these values yourself and get a sense that you know as a function of your epoch number, your learning rate gradually decreases. Where is this? According to this formula, up on top. So if you wish to use learning rate decay, what you can do is try a variety of values of both hyperparameter and alpha zero as well as of this decay rate hyperparameter, and then try to find a value that works well, other than this formula for learning rate decay, there are a few other ways that people use. For example, this is called exponential decay, where alpha is equal to some number less than 1, such as 0.95 times epoch num times alpha 0. So this will exponentially quickly decay your learning rate. Other formulas that people use are things like alpha equals some constant over epoch num square root times alpha 0 or some constant k. Another hyperparameter over the mini bash number t Square Rooted Times Alpha 0. And sometimes you also see people use a learning rate that decreases in discrete steps, where for some number of steps you have some learning rate and then after a while you decrease it by 1/2, after a while, by 1/2, after A while, by 1/2. And so this is a discrete staircase. So so far we've talked about some using some, you know, formula to govern how alpha, the learning rate changes over time. One other thing that people sometimes do is manual decay. And so if you're training just one model at a time, and if your model takes many hours or even many days to train, what some people would do is just watch your model as it's training over, you know, a large number of days, and then manually say, oh, it looks like the learning rate slowed down. I'm gonna decrease alpha a little bit. Of course this works, this manually controlling alpha, really tuning alpha by hand, R by R, or day by day. This works only if you're training only a small number of models. But sometimes people do that as well. So now you have a few more options for how to control the learning rate alpha. Now, in case you're thinking, wow, this is a lot of hyperparameters, how do I select amongst all these different options? I would say don't worry about it for now. And next week we'll talk more about how to systematically choose hyperparameters. Um, for me, I would say that learning rate decay is usually lower down on the list of things I try setting alpha just a fixed value of alpha and getting that to be well tuned has a huge impact. Learning rate decay does help. Um, sometimes it can really help speed up training. But it is a little bit lower down my list when when in terms of the things I would try. But next week when we talk about hyperparameter tuning, you see more systematic ways to organize all of these hyperparameters and how to efficiently search amongst them. So that's it for learning rate decay. Um, finally, I want to also want to talk a little bit about local optima and saddle points in neural networks so you can have a little bit better intuition about the types of optimization problems your optimization algorithm is trying to solve when you're trying to train these neural networks. Let's go on to the next video to see that.