Forward and Backward Propagation (C1W4L06)

In the previous video you saw the basic blocks of implementing a deep neural network. A forward propagation step for each layer, and a corresponding backward propagation step. Let's see how you can actually implement these steps. We'll start with forward propagation. Recall that what this will do is input al minus 1 and output al and the cache zl. And we just said that from implementational point of view, maybe we'll cache WL and BL as well, just to make the functions call a bit easier in the programming exercise. And so the equations for this should already look familiar. The way to implement a forward function is just this Bengals wl times al -1 b l. And then al equals deactivation function applied to Z. And if you want a vectorized implementation, then it's just that times Al minus 1 plus B, with the B adding B being a Python broadcasting and al equals G applied element wise to Z. And you remember on the diagram on for the fourth step where we had this chain of boxes going forward, so you initialize that with feeding in a zero which is equal to X. So you know, you initialize this really what is the input to the first one, right? It's really um, a zero, which is the input features to either for one training example, if you're doing one example at a time, or um, a capital zero, the entire training set if you are processing the entire training set. So that's the input to the first forward function in the chain. And then just repeating this allows you to compute forward propagation from left to right. Next, let's talk about the backward propagation step Here your goal is to input DAL and output DAL minus 1 and DWL and DB. Let me just write out the steps you need to compute these things. DZL is equal to DAL element wise product with G of L prime Z of L, um, and then Compute the derivatives DWL equals DZL times A of L minus 1. Um, I didn't explicitly put that in the cache, but it turns out you need this as well. And then DBL is equal to DZL. And finally DA of L minus 1 is equal to WL transpose times DZL. Okay, and I don't want to go through the detailed derivation for this, but it turns out that if you take this definition for DA and plug it in here, then you get the same formula as we had in the previous week for how you compute DZL as a function of the previous dzl. In fact, well, if I just plug that in here, you end up that DZL is equal to WL plus 1 transpose DZ L plus 1 times GL prime ZFL I know this is a looks like a lot of algebra. You could actually double check for yourself that this is the equation we had written down for backpropagation last week when we were doing a neural network with just a single hidden layer. And as a reminder, this times is element wise product. But so all you need is those four equations to implement your backward function. And then finally I'll just write out the vectorized version. So the first line becomes dzl equals dal element wise product with gl prime of zl maybe. No surprise there, dwl becomes 1 over m dzl times a l minus 1 transpose and then dbl becomes 1 over m np.sum dzl then x is equals 1. Keep dims equals true. We talked about the use of NP sum in the previous week to compute db. And then finally da l minus 1 is wl transpose times dz of l. So this allows you to input this quantity DA over here and output DWL DBL the derivatives you need as well as DAL minus one, right? As follows. So that's how you implement the backward function. So just to summarize, take the input X, you might have the first layer, maybe has a relu activation function, then go to the second layer, maybe uses another relu activation function, goes to the third layer, maybe has a sigmoid activation function. If you're doing binary classification and this outputs Y hat and then using Y hat you can compute the loss. And this allows you to start your backward iteration. I'll draw the arrows first. I guess I don't have to change pens too much. Where you would then have backprop compute the derivatives, compute, you know, DW3, DB3, DW2 DB2, DW1, DB1. And along the way you would be computing, I guess the cache would transfer z1, z2, z3. And here you pass back with da2 and da1. This could compute da0, but we won't use that. So you can just discard that. Right? And so this is how you implement forward prop and backprop for a three layer neural network. Now, there's just one last detail that I didn't talk about, which is for the forward recursion, we would initialize it with the input data X. How about the backward recursion? Well, it turns out that the A of L when you're using logistic regression, when you're doing binary classification, is equal to y over a plus one minus Y over one minus a. So it turns out that the derivative of the loss function with respect to the output with respect to y hat can be shown to be equal to this. If you're familiar with calculus, if you look up the loss function L and take derivatives with respect to y hat or respect to a, you could show that you get that formula. So this is the formula you should use for DA for the final layer, capital L. Um, and of course, if you were to have a vectorized implementation, then you initialize the backward recursion not with this, but with DA capital A for the layer L, which is going to be, you know, the same thing for the different examples, right Over a for the first training example, plus 1 minus Y for the first training example, over 1 minus A for the first training example, down to the nth training example, it's 1 minus A of M. So that's how you implement the vectorized version. That's how you initialize the vectorized version of backpropagation. So you've now seen the basic building blocks of both forward propagation as well as backpropagation. Now, if you implement these equations, you will get a correct implementation of forward prop and backprop to get you the derivatives you need. You might be thinking, wow, there was a lot equations. I'm slightly confused. I'm not quite sure I see how this works. And if you're feeling that way, my advice is, when you get to this week's programming assignment, you will be able to implement these for yourself and they'll be much more concrete. And I know there was a lot of equations, and maybe some of the equations didn't make complete sense, but if you work through the calculus and the linear algebra, which is not easy, so, you know, feel free to try. But that's actually one of the more difficult derivations in machine learning. It turns out the equations wrote down are just the calculus equations for computing the derivatives, especially in backprop. Uh, but once again, if this feels a little bit abstract, a little bit mysterious to you, my advice is, when you've done the programming exercise, it will feel a bit more concrete to you. Um, although I have to say, you know, even today, when I implement a learning algorithm, sometimes even I'm surprised when my learning algorithm implementation works. And it's because a lot of the complexity of machine learning comes from the data rather than from the lines of code. So sometimes you feel like you implement a few lines of code not quite sure what it did, but this almost magically works. And because a lot of the magic is actually not in the piece of code you write, which is often, you know, not too long. It's not. It's not exactly simple, but it's not, you know, 10,000 or 100,000 lines of code. But you feed it so much data that sometimes, even though I've worked in machine learning for a long time, sometimes it still, you know, surprises me a bit when my learning algorithm works. Because a lot of the complexity of your learning algorithm comes from the, the data rather than necessarily from your writing, you know, thousands and thousands of lines of code. All right, so that's how you implement deep neural networks. And again, this will become more concrete when you've done the programming exercise. Um, before moving on, I want to discuss, in the next video, want to discuss hyperparameters and parameters. It turns out that when you're training deep nets, being able to organize your hyperparameters well will help you be more efficient in developing neural networks. And in the next video, let's talk about exactly what that means.