Andrew Ng walks through building a TensorFlow program from scratch using a quadratic cost function. You'll see how to declare variables, build costs with TensorFlow operations, run gradient descent and Adam optimizers, and use placeholders to feed in training data. The lecture explains computation graphs and automatic differentiation.
Chapters
Welcome to the last video for this week. There are many great deep learning programming frameworks. One of them is TensorFlow. I'm excited to help you start to learn to use TensorFlow. What I want to do in this video is show you the basic structure of a TensorFlow program and then leave you to practice and learn more details and practice them yourself in this week's program exercise. This week's programming exercise will take some time to do, so please be sure to leave some extra time to do it as a motivating problem. Let's say that you have some cost function J that you want to minimize. And for this example, I'm gonna use this highly simple cost function J of W Bengals w squared minus 10w plus 25. So that's the cost function. You might notice that this function is actually W minus 5 squared. If you expand out this quadratic, you get the expression above. And so the value of W that minimizes this is W equals 5. But let's say we didn't know that and you just have this function. Let's see how you can implement something in TensorFlow to minimize this. Because a very similar structure of program can be used to train neural networks where you can have some complicated cost function j of WB depending on all the parameters of your neural network. And then similarly, you'll be able to use TensorFlow to automatically try to find values of W and B that minimize this cost function. But let's start with the simpler example on the left. So I'm running Python in my Jupyter notebook, and to start up TensorFlow, you import numpy as np and it's idiomatic to use import tensorflow and as tf. Next, let me define the parameter w. So in TensorFlow you're going to use tf dot variable to define a parameter type equals tf dot float32. And then let's define the cost function. So remember, the cost function was W squared minus 10 W plus 25. So I'm just TF dot add. Um, so I'm gonna have W squared plus TF dot multiply. So the second term was minus 10 times W. And then I'm gonna add that to 25. So let me put another TF dot add over there. So that defines the cost j that we had. And then I'm gonna write train equals tf.train. gradient descent optimizer. Let's use a learning rate of 0.01 and the goal is to minimize the cost. And finally, the following few lines are quite idiomatic. Init equals tf.global variables initializer and then session equals tf.session. so it starts a TensorFlow session session.run init to initialize the global variables. And then for TensorFlow to evaluate a variable, we're gonna use cess run w. We haven't done anything yet. So with in this line above, initialize W to 0 and define a cost function. We'll define train to be our learning algorithm, which uses a gradient descent optimizer to minimize the cost function. But we haven't actually run the learning algorithm yet. So session.run we evaluate W and let me print session.run so if you run that evaluates W to be equal to zero because we haven't run anything yet. Now let's do session dot run um train. So what this will do is run one step of gradient descent and then let's evaluate the value of W after one step of gradient descent and print that. So we do that after one step of gradient descent, W is now 0.1. Let's now run a thousand iterations of gradient descent. So dot run train and let's then print session dot run W. So this would run a thousand iterations of gradient descent, and at the end W ends up being 4.9999. Remember, we said that we're minimizing W minus 5 squared, so the optimal value of W is 5, and it got very close to this. So hope this gives you a sense of the broad structure of a TensorFlow program. And as you do the programming exercise and play with more TensorFlow code yourself, some of these functions that I'm using here will become more familiar. Some things to notice about this, W is the parameter we're trying to optimize, so we're going to declare that as a variable. And notice that all we had to do was define a cost function using these add and multiply and so on functions. And TensorFlow knows automatically how to take derivatives with respect to the add and multiply as well as other functions, which is why you only had to implement basically forward prop and it could figure out how to do the back prop or the gradient computation, because that's already built in to the add and multiply as well as the squaring functions. By the way, in case this notation seems really ugly, TensorFlow actually has overloaded the computation for the usual plus, minus and so on. So you could also just write this nicer format for the cost, comment that out and rerun this and get the same result. So once W is declared to be a TensorFlow variable. The squaring, multiplication, adding and subtraction operations are overloaded. So you don't need to use this ugliest syntax that I had above. Now there's just one more feature of TensorFlow that I want to show you, which is this example, minimize a fixed function of W. What if the function you want to minimize is a function of your training set? So what if you have some training data X, um, and when you're training a neural network, the training data X can change. So how do you get training Data into a TensorFlow program? So I'm gonna define T and X, which is think of this as playing a role of a training data or really the training data with both X and Y. But we only have X in this example. So this gonna define X replaceholder and it's going to be of type float 32 and let's make this a three by one array. And what I'm going to do is whereas the cost here had fixed coefficients in front of the three terms in this quadratic, it was 1 times w squared minus 10 times w plus 25. We could turn these numbers 1 minus 10 and 25 into data. So what I'm going to do is replace the cost with cos equals x 00 times w squared plus x 10 times w plus x 20 times 1. So now x becomes sort of like data that controls the coefficients of this quadratic function. And this placeholder function tells TensorFlow that x is something that you provide the values for later. So let's define another Array. Coefficients equals NP dot array 1 minus 10. And yes, the last value was 25. So that's going to be the data that we're gonna plug into X. So finally we need a way to get this array coefficients into the variable X. And the syntax to do that is it is doing the training step that the values for will need to be provided for X. So I'm going to set here feed dict equals X maps to coefficients and I'm gonna change this, I'm gonna copy and paste put that here as well. All right. Hopefully I didn't have any syntax errors. Let's try rerunning this and we get the same results hopefully as before. And now if you want to change the coefficients of this quadratic function, let's say you take this T10 and change it to 20 minus 20 and let's change this to 100. So this is now the function X minus 10 squared. And if I rerun this, hopefully I find that the value that minimizes X minus 10 squared is W equals 10. Let's see. Cool. Great. And we got W very close to 10 after running 1000 iterations of gradient descent. So what you see more of when you do the programming exercise is that a placeholder in TensorFlow is a variable whose value you assign later. And this is a convenient way to get your training data into the cost function. And the way you get your data into the cost function is with this syntax, when you're running a training iteration to use the feed dict to set x to be equal to the coefficients here. And if you're doing mini batch gradient descent, where on each iteration you need to plug in a different mini batch, then on different iterations you use the feed dict to feed in different subsets of your training set different mini batches into where your cost function is expecting to see data. So hopefully this gives you a sense of what TensorFlow can do. And the thing that makes it so powerful is all you need to do is specify how to compute TensorFlow, the cost function, and then it takes derivatives, and it can apply a gradient optimizer or an Atom optimizer or some other optimizer with just, you know, pretty much one or two lines of code. So here's the code again. I've cleaned this up just a little bit. And in case some of these functions or variables seem a little bit mysterious to you still, they will become more familiar after you practice with it a couple times, but by working through their programming exercise. Just one last thing I want to mention. These three lines of code are quite idiomatic in TensorFlow. And what some programmers will do is use this alternative format, which basically does the same thing. Set session to TF session to start the session, and then use the session to run init and then use the session to evaluate, say, W and then print a result. But this with construction is used in a number of TensorFlow programs as well. It more or less means the same thing as the thing on the left, but the with command in Python is a little bit better at cleaning up in case there's an error on exception while executing this inner loop. So you see this in the programming exercise as well. So what is this code really doing? Let's focus on this equation. The heart of a TensorFlow program is something to compute a cost, and then TensorFlow automatically figures out the derivatives and how to minimize that cost. So what this equation or what this line of code is doing is allowing TensorFlow to construct a computation graph, and a computation graph does the following. It takes x00, um, it takes w and then I guess W gets squared and then x00 gets multiplied with w squared. So you have x00 times w squared and so on, right? And eventually, you know, this gets built up to compute this x w x 00 times w squared plus x 10 times w plus and so on. And so eventually you get, you know, the cost function, right? And I guess the last term to be added would be x2 0, right? Gets added to be the cost. I won't write out the formula for the cost. And, and the nice thing about TensorFlow is that by implementing basically forward propagation through this computation graph to compute the cost, TensorFlow has already bat built in all the necessary backward functions. So remember how training a deep neural network had a set of forward functions instead of backward functions? And programming frameworks like TensorFlow have already built in the necessary backward functions, which is why by using the built in functions to compute the forward function, it can automatically do the backward functions as well to implement back propagation through, through even very complicated functions and compute derivatives for you. So that's why you don't need to explicitly implement backprop. And this is one of the things that makes the programming frameworks help you become really efficient. If you look at the TensorFlow documentation, I just want to point out that the TensorFlow documentation uses a slightly different notation than I did for drawing the computation graph. So it uses x00, W. And then rather than writing the value like w squared, the TensorFlow documentation tends to just write the operation. So this would be a square operation, and then these two get combined in a multiplication operation and so on. And then the final node, I guess that'd be a addition operation where you Add X to 0 to find the final value. So for the purposes of this class, I thought that this notation for the computation graph will be easier for you to understand. But if you look at the TensorFlow documentation, if you look at the computation graphs in the documentation, you see this alternative convention where the nodes are labeled with the operations rather than with the value. But both of these representations, you know, represent basically the same computation graph. And there are a lot of things that you can do with just one line of code and programming for frameworks. For example, if you don't want to use gradient descent, but instead you want to use the ADAM optimizer. By changing this line of code, you could very quickly swap it, swap in a better optimization algorithm. So all the modern deep learning programming frameworks support things like this and makes it really easy for you to code up even pretty complex neural networks. So I hope this is helpful for giving you a sense of the typical structure of of a TensorFlow program. To recap the material from this week, you saw how to systematically organize the hyperparameter search process. We also talked about batch normalization and how you can use that to speed up training of your neural networks. And finally, we talked about programming frameworks for deep learning. There are many great programming frameworks. Um, and we had this last video focusing on TensorFlow. With that, I hope you enjoyed this week's programming exercise and that helps you gain even more familiarity with these ideas.
