Adam Optimization Algorithm (C2W2L08)

During the history of deep learning, many researchers, including some very well known researchers, sometimes proposed optimization algorithms and showed they work well in a few problems. But those optimization algorithms subsequently were shown not to really generalize that well to the wide range of neural networks you might want to train. So over time I think the deep learning community actually developed some amount of skepticism about new optimization algorithms and lot of people felt that, you know, gradient descent with momentum really works well. It was difficult to propose things that work much better. So RMS PROP and the ADAM optimization algorithm, which we'll talk about in this video, is one of those rare algorithms that has really stood up and has been shown to work well across a wide range of deep learning architectures. So this is one of the algorithms that I wouldn't hesitate to recommend you try because many people have tried it and seen it work well on many problems. And, and the ADAM optimization algorithm is basically taking momentum and RMS prop and putting them together. So let's see how that works. To implement Adam, you would initialize v.

Dw equals 0, s dw equals 0 and similarly vdbs db equals 0.

And then on iteration.

T you would.

Compute derivatives, compute DW DB using current mini batch. So usually you do this with mini batch gradient descent and then you do the momentum exponentially weighted average.

So V DW equals beta.

But now I'm going to call this beta 1 to distinguish it from the hyperparameter beta 2 were used for the.

RMS portion of this.

So this is exactly what we had when we're implementing momentum, except that I've now called the hyperparameter beta one instead of beta.

And similarly you have VDB as follows. So one minus beta one times db. And then you do the RMS prop like update as well.

So now you have a different hyperparameter.

Beta 2 plus 1 minus beta 2 DW squared. And again the squaring there is element Y squaring of your derivatives DW and then SDB is equal to this plus 1 minus beta 2 times DB.

So this is the momentum like update with hyperparameter beta 1 and this is.

The RMS prop like Update with hyperparameter beta 2.

In the typical implementation of Adam, you do implement bias correction.

So you're gonna have V corrected, corrected means after bias correction, dw equals v dw divided by 1 minus beta 1 to the power of T. If you've done T iterations and similarly v db corrected equals vdb divided by 1 minus beta 1 to the power of t. And then similarly you implement this bias correction on S as well. So that's s dw divided by 1 minus beta 2 to the t and s db corrected equals s db divided by 1 minus beta 2 to THE T. Finally you perform the update.

So W gets updated as W minus alpha times. So if we're just implementing momentum, you'd.

Use V DW or maybe VDW corrected.

But now we add in the RMS PROP portion of this. So we're also going to divide by.

Square root of S DW corrected plus epsilon. And similarly B gets updated as a similar formula V DB corrected divided by square root S corrected DB plus epsilon.

And so this algorithm combines the effect of gradient descent with momentum together with gradient descent with RMS prop. And this is commonly used learning algorithm that's proven to be very effective for many different neural networks of a very wide variety of architectures. So this algorithm has a number of hyperparameters. The learning rate hyperparameter alpha is still important and usually needs to be tuned. So you just have to try a range of values and see what works. A common choice really the default choice for beta 1 is 0.9. So this is the moving average weighted.

Average of DW, right? This is the momentum like term the.

Um, hyper parameter for beta 2. The authors of the Adam paper, inventors.

The Adam algorithm recommend 0.999.

Again, this is computing the moving weighted average of DW squared as well as.

DP squared and then epsilon.

The choice of epsilon doesn't matter very much, but the authors of the Adam.

Paper recommended 10 to the minus 8.

But this parameter, you really don't need to set it and it doesn't affect performance much at all. But when implementing adam, what people usually do is just use the default values of beta 1 and beta 2 as well as epsilon. I don't think anyone ever really tunes epsilon and then try a range of values of alpha to see what works best. You could also tune beta one and beta two, but it's not done that.

Often, um, among the practitioners I know.

So where does the term Adam come from? Adam stands for adaptive.

Moment estimation.

So beta 1 is computing the mean of the derivatives, this is called the first moment. And beta 2 is used to compute exponentially weight average of the squares. And that's called the second moment. So that gives rise to the name adaptive moment estimation. But everyone just calls it the ADAM optimization algorithm. And by the way, one of my long term friends and collaborators is called Adam Coates. Far as I know, this algorithm doesn't have anything to do with him, except for the fact that I think he uses it sometimes. But sometimes I get asked that question. So just in case you're wondering. So that's it for the ADAM optimization algorithm. With it, I think you'll be able to train your neural networks much more quickly. But before we wrap up for this week, let's keep talking about hyperparameter tuning as well as gain some more intuitions about what the optimization problem for neural networks looks like. In the next video, we'll talk about learning rate decay.