Bias Correction of Exponentially Weighted Averages (C2W2L05)

You've learned how to implement exponentially weighted averages. There's one technical detail called bias correction that can make your computation of these averages more accurately. Let's see how that works. In the previous video you saw this figure for beta equals 0.9. This figure for beta equals 0.98. But it turns out that if you implement the formula as written here, you won't actually get the green curve when, say, beta equals 0.98. You actually get the purple curve here, and you notice that the purple curve starts off really low. So let's see how to fix that. When you're implementing a moving average, you initialize it with v0 equals 0, and then v1 is equal to 0.98, v0 plus 0.02 theta1, but v0 is equal to 0, so that term just goes away. So v1 is just 0.02 times theta1. So that's why if the first day's temperature is, say, 40 degrees Fahrenheit, then V1 will be 0.02 times 40, which is 8. So you get a, you know, much lower value down here. So it's not a very good estimate of the first day's temperature. V2 will be 0.98 times V1 plus 0.02 times THETA2. And if you plug in, you know, V1, which is this down here, and multiply it out, then you find that V2 is actually equal to 0.98 times 0.02 times theta1 plus 0.02 times theta2. And that's 0.0196 theta1 plus 0.02 theta2. So again, you know, assuming theta1 and theta2 are positive numbers, when you compute this, V2 be much less than theta1 or theta2. So V2 isn't a very good estimate of the first two days temperature of the year. So it turns out that there's a way to modify this estimate that makes it much better, that makes it more accurate, especially during this initial phase of your estimate, which is that Instead of taking VT, take VT divided by 1 minus beta to the power of T, where T is the current day that you're on. So let's take a concrete example. When T is equal to 21 minus beta, the power of T is 1 minus 0.98 squared. And it turns out that this is 0.0396. And so your estimate of the temperature on day two becomes V2 divided by 0.0. And this is going to be 0.0196 times theta1 plus 0.02 theta2. You notice that these two things adds up to the denominator 0.0396. And so this becomes a weighted average of theta1 and theta2. And this removes this bias. So you notice that as T becomes large, beta to the T will become will approach zero. Which is why when T is large enough, the bias correction makes almost no difference. This is why when T is large, the purple line and the green line, you know, pretty much overlap. But during this initial phase of learning, when, um, you're still warming up your estimates, when bias correction can help you to obtain a better estimate of the temperature, and there's this bias correction that helps you go from the purple line to the green line. So in machine learning, for most implementations of the exponentially weighted average, um, people don't often bother to implement bias correction because most people would rather just wait that initial period and have a slightly more biased estimate and go from there. But if you are concerned about the bias during this initial phase, while your exponentially weighted moving average is still warming up, then bias correction can help you get a better estimate early on so that you now know how to implement exponentially weighted moving averages. Let's go on and use this to build some better optimization algorithms.