RMSProp (C2W2L07)

You've seen how using momentum can speed up gradient descent. There's Another algorithm called RMSprop, which stands for root mean square prop, that can also speed up gradient descent. Let's see how it works. Recall our example from before, that if you implement gradient descent, you can end up with huge oscillations in the vertical direction, even whilst trying to make progress in the horizontal direction. In order to provide intuition for this example, let's say that the vertical axis is the parameter B and the horizontal axis is the parameter W. Right. Really could be W1 and W2 or some of the center parameters, but this is named as B and W for the sake of intuition. And so you want to slow down the learning in the B direction or in the vertical direction and speed up learning, or at least not slow it down in the horizontal direction. So this is what the RMS propeller algorithm does to accomplish this. On iteration T, it will compute as usual the derivatives DW DB on the current mini batch. So I was going to keep this exponentially weighted average. Instead of v dw, I'm going to use new notation SDW. So SDW is equal to beta times their previous value plus 1 minus beta times dw squared. I sometimes write this dw star star 2 to denote the exponentiation, I just write this as dw squared. So for clarity, this squaring operation is an element wise squaring operation. So what this is doing is really keeping an exponentially weighted average of the squares of the derivatives. And similarly we also have sdb equals beta s db plus 1 minus beta db squared. And again the squaring is an element wise operation. Next, RMS PROP then updates the parameters as follows. W gets updated as W minus the learning rate. And whereas previously we had um, alpha times dw, now is DW divided by square root of S d W and B gets updated as B minus the learning rate times. Instead of just a gradient, this is also divided by now divided by S db. So let's gain some intuition about how this works. Recall that in the horizontal direction, or in this example in the W direction, we want learning to go pretty fast, whereas in the vertical direction or in this example in the B direction we want to slow down or to damp out the oscillations in the vertical direction. So with these terms SDW and sdb, what we're hoping is that SDW will be relatively small, so that here we're dividing by a relatively small number, whereas SDB would be relatively large, so that here we're dividing by a relatively large number in order to slow down the updates in the vertical direction. And indeed, if you look at the derivatives, these derivatives are much larger in the vertical direction than in the horizontal direction. So you know the slope is very large in the B direction, right? So with derivatives like this, this is a very large DB and a relatively small DW because the function is sloped much more steeply in the vertical direction, that is in the B direction, than in the W direction than in the horizontal direction. And so DB squared will be relatively large, so SDB will be relatively large, whereas compared to that, DW will be smaller, or DW squared will be smaller, and so S DW will be smaller. So the net effect of this is that your updates in the vertical direction are divided by a much larger number, and so that helps damp out the oscillations, whereas the updates in the horizontal direction are divided by a smaller number. So the net impact of using RMS prop is that your updates will end up, you're looking more like this, right? That your updates in the vertical direction get damped out, but then horizontal direction, it can keep going. And one effect of this is also that you could therefore use a larger learning rate alpha and get faster learning without it diverging in the vertical direction. Now, just for the sake of clarity, I've been calling the vertical and horizontal directions B and W just to illustrate this. In practice, you're in a very high dimensional space of parameters. So maybe the vertical dimensions where you're trying to dampen oscillations is some set of parameters, w1, w2, w17, and the horizontal dimensions might be w3, w4, you know, and so on, right? And so the separation is a W and B is just for illustration. In practice, DW is a very high dimensional parameter vector, DB is also a very high dimensional parameter vector. But the intuition is that in dimensions, when you're getting these oscillations, you end up computing a larger sum or weighted average for these squares of derivatives. And so you end up damping out the directions in which there are these oscillations. So that's RMS prop and it stands for root mean squared, because root mean squared prop, because here you're squaring the derivatives and then you take the square root here at the end. So finally, just a couple last details on this algorithm before we move on. In the next video, we're actually going to combine RMS prop together with momentum. So rather than using the hyperparameter beta, which we had used for momentum, I'm going to call this hyperparameter beta 2, just to not clash the same hyper parameter for both momentum and for RMSprom. And also to make sure that your algorithm, uh, doesn't divide by zero. You know what, if square root of S d w right is very close to zero, then this thing could blow up. Just to ensure numerical stability. When you implement this in practice, you add a very, very small epsilon to the denominator. Doesn't really matter what epsilon is. Use 10 to the negative 8 would be a reasonable default. But this just ensures slightly greater numerical stability that you know, for numerical round off or whatever reason that you don't end up dividing by a very small number. So that's RMSprop and similar to momentum, has the effect of damping out the oscillations in gradient descent in mini bash gradient descent and allowing you to maybe use a larger learning rate alpha and certainly speeding up the learning speed of your algorithm. So now you know how to implement RMSprop and this will be another way for you to speed up your learning algorithm. Um, one fun fact about RMSprop, it was actually first proposed not in a academic research paper, but in a Coursera course that Geoff Hinton had taught on Coursera many years ago. Uh, I guess Coursera wasn't intended to be a platform for dissemination of novel academic research, but it worked out pretty well. In that case, it was really from the Coursera course that RMS props start to become widely known and it really took off. We've talked about momentum, we've talked about RMSprop. It turns out that you put them together, you can get an even better optimization algorithm. Let's talk about that in the next video.