Derivatives With Computation Graphs (C1W2L08)

In the last video, we worked through an example of using a computation graph to compute a function J. Now let's take a cleaned up version of that computation graph and show how you can use it to figure out derivative calculations for that function J. So here's our computation graph. Let's say you want to compute the derivative of J with respect to V. So what is that? Well, this says if we were to take this value of V and change it a little bit, how would the value of J change? Well, J is defined as V times V. And right now V is equal to 11. So if we were to bump up V by a little bit to 11.001, then J, which is 3 volts, so currently 33 will get bumped up to 33.0. So here we've increased V by 0.001, and the net result of that is that J goes up three times as much. So the derivative of J with respect to V is equal to 3 because the increase in j is 3 times the increase in V. And in fact, this is very analogous to the example we had in the previous video, where we had F of a equals 3a. And so we then derived that DF DA, which with slightly simplified or slightly sloppy notation, you can write as df da was equal to 3, right? So instead here we have j equals 3v. And so dj dv is equal to 3, with here j playing the row of F and V playing the row of A in this previous example that we had right from from an earlier video. So in the terminology of back propagation, what we're seeing is that if you want to compute the derivative of this final output variable, which usually is a variable you care most about with respect to V, then we've done sort of one step of back propagation. So we've gone one step backwards in this graph. Now let's look at another example. What is DJ da? In other words, if we bump up the value of A, how does that affect the value of J? Well, let's go through the example. Right now A is equal to 5, so let's bump it up to 5.001. The net impact of that is at v, which was a plus u. So that was previous to 11. This would get increased to 11.001. And then we've already seen as above, that J now gets bumped up to 33.003. So what we've seen is that if you Increase A by 0.001, J increases by 0.003. And by increase A, I Mean, if you were to take this value 5 and just plug in a new value, then the change to A will propagate to the right of the computation graph. So that J ends up being 33.003, and so the increase to J is three times the increase to A. So that means this derivative is equal to three. And one way to break this down is to say that if you, if you change A, then that would change V, and through changing V, that would change J. And so the net change to the value of J when you bump up the value when, when you nudge the value of A up a little bit is that first by changing A, you end up increasing V. Well, how much does V increase? Right? It is it will increase by an amount that's determined by DV da, and then the change in V will cause the value of J to also increase. So in calculus, this is actually called the chain rule, that if A affects V affects J, then the amount that J changes when you nudge A is the product of how much V changes when you nudge A times how much J changes when you nudge V. So in calculus, again, this is called the chain rule. And what we saw from this calculation is that if you Increase A by 0.001, V changes by the same amount. So DVDA is equal to 1. So in fact, if you plug in what we have worked out previously, dv dj is equal to 3 and dvda is equal to 1. So the product of these three times one that actually gives you the correct value that dj da is equal to 3. So this little illustration shows how by having computed dj dv has its derivative with respect to this variable. It can then help you to compute DJ da. And so that's another step of this backward calculation. I just want to introduce one more new notational convention, which is that when you're writing codes to implement backpropagation, there'll usually be some final output variable that you really care about, right? So final output variable that you really care about or that you want to optimize. And in this case, this final output variable is J is really the last node in your computation graph. And so a lot of computations will be trying to compute the derivative of that, you know, final output variable. So D of this final output variable with respect to some other variable, let me just call that D var, right? So a lot of the computations you have would be to compute the derivative of the finite output variable, really J, in this case with various intermediate variables such as a, b, c, u or V. And when you implement this in software, you know, what do you call this variable name, right? One thing you could do is in, in Python, you could write, you know, give us a very long variable name like d final output var over dvar. But that's a very long variable name. Could call this, I guess DJ dvar. But because you're always taking derivatives with respect to DJ with respect to this finite output variable, I'm gonna introduce a new notation where in code, when you are computing this thing in the code you write, we're just going to use the variable name DVAR in order to represent that quantity. Okay? So dvar in the code you write will represent the derivative of the final output variable you care about, such as J or sometimes the loss L with respect to the various intermediate quantities you're computing in your code. So this thing here, in your code, you know, you use DV to denote this value. So DV would be equal to 3. And your code, you represent this as a DA, right? Which is we also figured out to be equal to three. Okay, so we've done backpropagation partially through this computation graph. Let's go through the rest of this example on the next slide. So let's go to a cleaned up copy of the computation graph. And just to recap what we've done so far is go back with here and figured out that DV is equal to 3. And again the definition of DV, that's just the variable name of the code is really dj DV and we figured out that da is equal to 3. And again, da is the variable name in your code, and that's really the value of DJ da. And we'll sort of hand wave how we've gone backwards on these two edges like so. Now let's keep computing derivatives. Let's look at the value U. So what is DJ du? Well, through a similar calculation as what we did before, you know, we start off with u equals 6. Um, if you bump up u to 6.001, then v, which is previous 11 goes up to 11.001. And so j goes from 33 to 33.003. And so the increase in j is 3x, so this is equal. And the analysis for U is very similar to the analysis we did for A. This is actually computed as DJ DV times DV du, where this we had already figured out was three. And this turns out to be equal to one. So with kind of one more step of back propagation, we end up computing that DU is also equal to three. And DU is of course, just this DJ du. Now we just step through one last example in detail. So what is DJ db? Right? So, you know, imagine if you are allowed to change the value of B and you want to tweak B a little bit in order to minimize or maximize the value of J, right? So what is the derivative or what's the slope of this function J when you change the value of B a little bit? It turns out that using the chain rule for calculus, this can be written as the product of two things. It's DGJ DU times DU db. And the reasoning is if you change B a little bit, so b goes from 3 to, say, 3.001. Um, the way it'll affect J is it will first affect U. So how much does it affect U? Well, U is defined as B times C, right? So this will go from 6 when B is equal to 3, to now 6.002, right? Because C is equal to 2 in our example here. And so this tells us that dudb is equal to 2, because when you bump up b by 0.001, u increases twice as much. So dudb this is equal to 2. And now we know that U has gone up twice as much as B has gone up. Well, what is DJ Du? We've already figured out that this is equal to 3. And so by multiplying these two out, we find that DJ DB is equal to 6. And again, here's the reasoning for the second part of the argument, which is we want to know when U goes up by 0.002, how does that affect j? The fact that dj du is equal to 3, that tells us that when u goes up by 0.002, j goes up three times as much. So j should go up by 0.006, right? So that comes from the fact that dj du is equal to three. And if you check the math in detail, you will find that if B becomes 3.001, then U becomes 6.002, V becomes 11.002, right? So that's A plus U. So that's 5 plus U. And then J, which is equal to three times V, that ends up being equal to 33.006, right? And so that's how you get that DJ DB is equal to 6. And to fill that in, this is if we go backwards. So this is DB is equal to 6, and DB really is the Python code variable name for, you know, DJ db. And I won't, I won't go through the last example in great detail, but it turns out that if you also compute out dj, this turns out to be dj du times d u and this turns out to be 9. This turns out to be 3 times 3. I won't go through that example in detail, but so through this last step, you know, it is possible to derive that DC is equal to. So the key takeaway from this video, from this example is that when computing derivatives, and computing all of these derivatives, the most efficient way to do so is through a right to left computation following the direction of the red arrows. And in particular we'll first compute the derivative with respect to V, and then that becomes useful for computing the derivative with respect to A and the derivative with respect to U, and then the derivative with respect to U. For example, this term over here and this term over here. Those in turn become useful for computing the derivative with respect to B and the derivative with respect to C. So that was a computation graph and how there's a forward or left to right calculation to compute the cost function such as J that you might want to optimize, and a backwards or a right to left calculation to compute derivatives. If you're not familiar with calculus or the chain rule. I know some of those details would have gone by really quickly, but you didn't follow all the details. Don't worry about it. In the next video, we'll go over this again in the context of logistic regression and show you exactly what you need to do in order to implement the computations. You need to compute the derivatives for the logistic regression model.