Learn how to track matrix dimensions in neural networks. For a layer l, W has shape (nl, n_l-1), b is (nl, 1), and Z and A are (nl, m) where m is the number of examples. dW and db match their corresponding weight shapes.
n_l by n_{l-1} ("the dimensions of wl must be nl by nl minus 1.")
n_l by 1 ("blue should be NL by one dimensional.")
Z^{[l]} and A^{[l]} are n_l by m; X (A^{[0]}) is n_0 by m; b^{[l]} is n_l by 1 and is broadcasted to n_l by m.
Chapters
When implementing a deep neural network, one of the debugging tools I often use to check the correctness of my code is to pull a piece of paper and just work through the dimensions and matrix I'm working with. So let me show you how to do that, since I hope this will make it easier for you to implement your deep nets as well. So capital L is equal to five. Count them quickly, not counting the input layer. There are five layers here, so four hidden layers and one output layer. And so if you implement forward propagation, the first step will be z1 equals w1 times the input features x plus b1. So let's ignore the bias terms B for now and focus on the parameters W. Now, this first hidden layer has three hidden units. So this is layer 0, layer 1, layer 2, layer 3, layer 4, and layer 5. So using the notation we had from the previous video, we have that n1, which is the number of hidden units in layer 1 is equal to 3. And here we would have that n2 is equal to 5, n3 is equal to 4, n4 is equal to 2, and n5 is equal to 1. And so far we've only seen neural networks with a single output unit. But later, in later courses, we'll talk about neural networks with multiple output units as well. And finally, for the input layer, we also have n0 equals nx is equal to 2. So now let's think about the dimensions of Z W and X Z is the vector of activations for this first hidden layer. So Z is going to be three by one. It's going to be a three dimensional vector. So I'm going to write it as a n1 by one dimensional vector or n1 by one dimensional matrix, three by one in this case. Now, how about the input features X? X? We have two input features. So X is in this example, two by one, but more generally it will be N. So what we need is for the matrix W1 to be something that when we multiply an N0 by one vector to it, we get an N1 by one vector, right? So you have sort of a three dimensional vector equals something times a two dimensional vector. And so by the rules of matrix multiplication, this has got to be a 3 by 2 matrix, right? Because a 3 by 2 matrix times a 2 by 1 matrix or times a 2 by 1 vector, that gives you a 3 by 1 vector. And more generally, this is going to be an N1 by N0 dimensional matrix. So what we figured out here is that the dimensions of w1 has to be n1 by n0. And more generally, the dimensions of wl must be nl by nl minus 1. So for example, the dimensions of w2 for this it will have to be 5 by 3 or it will be n2 by n1, because we're going to compute z2 as w2 times a1. And again, let's ignore the bias for now. But so this is going to be three by one and we need this to be five by one. And so this had better be five by three. And similarly, W3 will be is really the dimension of the next layer comma, the dimension of the previous layer. So this is going to be four by five, W4 is going to be two by four and W5 is going to be one by two. Okay, so the general formula to check is that when you're implementing the matrix for a layer L that the dimension of that matrix be nL by n L1. Now let's think about the dimension of this vector B. This is going to be a three by one vector. So you have to add that to another three by one vector in order to get a three by one vector as the output. Or in this example you need to add this, which is going to be five by one. So it's going to be another five by one vector in order for the sum of these two things that I have in the boxes, to be itself a five by one vector. So the more general rule is that in the example on the left B1 is N1 by one, right, that's three by one. And in the second example it is this is N2 by one. And so the more general case is that blue should be NL by one dimensional. So hopefully these two equations help you to double check that the dimensions of your matrices W as well as of your vectors B are the correct dimensions. And of course, if you're implementing back propagation, then the dimensions of DW should be the same as dimension of W. So DW should be the same dimension as W and DB should be the same dimension as B. Now the other key set of quantities whose dimensions to check are these Z x as well as A of L, which we didn't talk too much about here, but because Z of L is equal to G of A of L applied element wise, then Z and A should have the same dimension in these types of networks. Now let's see what happens when you have a vectorized implementation that looks at multiple examples at a time. Even for a vectorized implementation, of course, the dimensions of W, B, DW and DB will stay the same, but the Dimensions of ZA as well as X will change a bit in your vectorized implementation. So previously we had z1 equals w1 times x plus b1, where this was n1 by 1, this was n1 by n0, x was n0 by 1 and b was n1 by 1. Now in a vectorized implementation, you would have z1 equals w1 times X plus b1, where now z1 is obtained by taking the z ones for the individual example. So there's z11, z12 up to z1 m and stacking them as follows. And this gives you z1. So the dimension of z1 is that instead of being n1 by 1, it ends up being n1 by m. If m is the size of your training set, the dimensions of w1 stays the same. So it's still n1 by n0. And x instead of being n0 by 1 is now all your training examples stacked horizontally. So it's now N0 by M. Um, and so you notice that when you take a n1 by n0 matrix and multiply that by an n0 by m matrix, that together they actually give you an n1 by m dimensional matrix, as expected. Now the final detail is that B1 is still N1 1, but when you take this and add it to B, then through Python broadcasting, this would get duplicated into an N1 by M matrix and then add it element wise. So on the previous slide we talked about the dimensions of W, B, DW and DB. Here what we see is that whereas ZL as well as AL are of dimension NL by 1, we have now instead that capital ZL as well as capital AL are NL by M. And the special case of this is when L is equal to 0, in which case a 0, which is equal to just your training set input features X is going to be equal to n 0 by m, as expected. And of course, when you're implementing this in back propagation, we'll see later, you end up computing DZ as well as da. And so these will of course have the same dimension as Z and A. So I hope the little exercise went through helps clarify the dimensions of the various matrices you'll be working with when you implement backpropagation for a deep neural network. So long as you work through your code and make sure that all the matrices dimensions are consistent, that will usually help you go some ways toward eliminating some class of possible bugs. So I hope that exercise for figuring out the dimensions of the various matrices you'll be working with is helpful. When you implement a deep neural network if you keep straight the dimensions of these various matrices and vectors you're working with, hopefully that will help you eliminate some class of possible bugs. It certainly helps me get my code right. So next, we've now seen some of the mechanics of how to do certain forward propagation in a neural network. But why are deep neural networks so effective? And why do they do better than shallow representations? Let's spend a few minutes in the next video to discuss the.
