Learn exponentially weighted averages, a technique for smoothing noisy time series data. Using London temperatures, the lecture shows how the formula Vt = beta V{t-1} + (1-beta) thetat creates different smoothing effects: beta=0.9 gives roughly 10-day averaging, beta=0.98 gives 50-day averaging, and beta=0.5 gives 2-day averaging, trading smoothness against how quickly the average adapts to new data.
I want to show you a few optimization algorithms that are faster than gradient descent. In order to understand those algorithms, you need to be able to use something called exponentially weighted averages, also called exponentially weighted moving averages in statistics. Let's first talk about that. And then we use this to build up to more sophisticated optimization algorithms. So even though I now live in the United States, I was born in London. So for this example, I got the daily temperature from London from last year. So on January 1st, temperature was 40 degrees Fahrenheit. And I know most of the world uses a Celsius system, but I guess I live in the United States, which uses Fahrenheit. Um, so that's 4 degrees Celsius. And on January 2nd, it was 9 degrees Celsius, and so on. And then about halfway through the year, a year has 365 days. So this would be sometime day number 180 would be sometimes sometime in late May, I guess it was 60 degrees Fahrenheit, which is 15 degrees Celsius, and so on. So it starts to get, you know, warmer towards summer, and it was colder in January. So you plot the data. You end up with this with day one being sometime in January, that, you know, being the beginning of summer, and that's the end of the year, kind of late December. So this would be January, uh, January 1st. This would be middle of the year, approaching summer. And this would be the data from the end of the year. So this data looks a little bit noisy. And if you want to compute the trends, kind of the local average or moving average of the temperature, here's what you can do. Let's initialize. V0 equals 0. And then on every day, we're gonna average it with a weight of 0.9 times whatever was the previous value, plus 0.1 times the that day's temperature. So theta1 here would be the temperature from this first day. And on the second day, we're again gonna take a weighted average. So 0.9 times the V previous value, plus 0.1 times today's temperature, and so on, V2 plus 0.1 times theta3 and so on. And the more general formula is V on a given day is 0.9 times V from the previous day, plus 0.1 times the temperature of that day. So if you compute this and plot it in red, this is what you get. You get a moving average, or what's called an exponentially weighted average of the daily temperature. So let's look at the equation we had from the previous slide. It was VT equals previously we had 0.9. We're going to turn that into parameter beta, beta times VT minus 1 plus. And then previously with 0.1, boom. Turn that into 1 minus beta times THETA T. So previously we had beta equals 0.9. It turns out that for reasons we'll go into later, when you compute this, you can think of VT as approximately averaging over something like 1 over 1 minus beta days temperature. So for example, when beta equals 0.9, you could think of this as averaging over the last 10 days temperature. And that was the red line. Now let's try something else. Let's set beta to be very close to one. Let's say it's 0.98. Then if you look at one over one minus 0.98, this is equal to 50. So this is, you know, think of this as averaging over roughly the last 50 days temperature. And if you plot that, you get this green line. So notice a couple things. With this very high value of Beta, the plot you get is much smoother because you're now averaging over more days of temperature temperature. So the curve is just, you know, less wiggly. It's now smoother. But on the flip side, the curve is now shifted further to the right because you're now averaging over a much larger window of temperatures. And by averaging over a larger window, this formula, this exponentially weighted average formula, it adapts more slowly when the temperature changes, right? So there's just a bit more latency. And the reason for that is when beta is equal to 0.98, that then it's giving a lot of weight to the previous value and a much smaller weight, just 0.02 to whatever you're seeing right now. So when the temperature changes, when temperature goes up or down, this exponentially weighted average just adapts more slowly when beta is so large. Now let's try another value. If you set beta to another extreme, let's say it is 0.5, then this, by the formula we have on the right, this is something like averaging over just two days temperature. And you plot that, you get this yellow line. And by averaging only over two days temperature, you have a much. It's as if you're averaging over a much shorter window. So you're much more noisy, much more susceptible to outliers. But this adapts much more quickly to if the temperature changes. So this formula is how you implement an exponentially weighted average. Again, it's called an exponentially weighted average moving average in the statistics literature, and I'm gonna call it exponentially weighted average for short, and by varying this parameter or later we'll see it's actually a hyperparameter of your learning algorithm. You can get slightly different effects and there will usually be some value in between that works best. That gives you the red curve, which, you know, maybe looks like a better average of the temperature than either the green or the yellow curve. You now know the basics of how to compute exponentially weighted averages. In the next video, let's gain a bit more intuition about what it's.
