Learn TensorFlow and Deep Learning fundamentals with Python (code-first introduction) Part 2/2

Hello and welcome to part 2 of this fairly long, or lengthy, it was originally going to be 14, 14, not 40, 14 hours long video, but YouTube's video limit is 12 hours, so I've separated it. Part 1, if you're watching this, you should have already watched part 1, so if you're hearing me say you should have already watched part 1, click on the link to part 1 and make sure you go and watch that. But this is part 2, which basically continues off right where part 1 of this learning TensorFlow and deep learning code-first video series starts off. So I won't hold you up. All the links you need, such as the course GitHub repo, all the materials will be down below. And if you want to sign up to the full version of the course, which is about 20+ hours more of the materials than what you've already covered, there'll be a link to that in the description below as well. Without any further ado, happy coding. Now, we've seen that our neural network that we've built can model straight lines with somewhat better than guessing accuracy. However, when it comes to non-straight lines, it's not doing too well. And so we also mentioned the missing piece. We're like treasure hunters here. The missing piece is non-linearity. So I cannot emphasize enough how important this concept is for neural networks. Now, before we write any code, I want to introduce to you a little question here. I want something to think about just over the next few videos while we, we start to learn about non-linearity. What could you draw if you had an unlimited amount of straight, in other words linear, and non-straight or non-linear lines? So just think about that. What kind of patterns could you draw? You have an unlimited amount of straight and non-straight lines. Let's look at our data again. We have some linear data, so you might think, is this possible to model with straight lines? I think so. And then we have some nonlinear data, not possible to model with straight lines. Now, you could post an argument to say, oh, well, Daniel, what if you just draw really small straight lines? And go in between the gaps here. Well, that is one option, but let's just pretend that we need straight and non-straight lines to draw the patterns that we need. So keep thinking about this question, and again, before we write any code, we're going to have a little bit of a play around. I introduced the TensorFlow Playground in a previous video. However, we haven't looked at it together. I'm just going to playground.tensorflow.org, And we can set up our data here. What I might do is zoom in nice and far. Okay, beautiful. So we have a few things going on here. We have data. Now this data looks very similar to our circulator that we're working with, except they're using orange and blue dots instead of red and blue dots. And we have some features, x1 and x2. We could pretend that this is our x0 and x1. Now, right now this neural network has 2 hidden layers. What we might do is reduce this down to 1 hidden layer and 1 hidden neuron. So a very simple neural network. And let's have a play around. What are these? We've seen the learning rate. All right, let's switch this to— what have we seen before? I believe Adam's is the default, is 0.001. And activation, hmm, we haven't really played around with these. What we might do is switch this to linear. Now we've seen what linear data is, what linear data looks like, and we've got regularization rate, problem type, classification. All right, so let's just leave it at these settings for now, and I'm going to press play and see what happens. So we get the test loss that starts to— oh, it's just evening out at about 0.5. What does that remind us of? Do you remember our model that we built previously that only ended up with 50% accuracy? I think this is what's going on here. I know, to test this out, how about we recreate this neural network of what's going on? We've already got the data, we've already got— we can create the hidden layer, and we can set the learning rate, we can— we might see how we can set the activation too. All right, let's do that. Let's go back and let's write some code. So come in here. Now, how might we do this? Just as before, we're going to set the random seed and tf.random.setseed 42. And now let's go number 1, create the model, model for tf.keras.Sequential. If we come down here, And then we want to go tf.keras.layers.Dense, and the activation can be— what did we set the activation to in here? Linear. How might we get access to that? Well, one shortcut is to just go linear as a string, or we could do tf.keras.activations.linear. Ah! All right, now we've got a parameter that we haven't seen before, activation, but we actually have seen this before. If we go back to our slides, and I think we might have to go a few back here. Yeah, there we go. We've got an activation parameter. Now we're going to start to look into, we've seen how we can add layers to try and improve our deep model. We've seen how we can increase the number of hidden units, but right now, these aren't really working for us. So we're up to changing the activation functions. All right, so we're working with this hyperparameter here. Now let's go back, and what we're going to do is compile our model, because we want to recreate what we've done in TensorFlow Playground, and see if we can replicate those results. This is a very, a very good exercise to go through for yourself if you're playing around with the TensorFlow Playground. To just recreate what you can see in there. Binary cross entropy. This is the— because we're working with a binary problem. Now, this is another way you can write the losses. You can actually write the losses in many different ways. We've seen this one, binary cross entropy, and we could even do it— let me just introduce you to writing things as strings. Cross entropy, just in case you see it like that somewhere else. And optimizer, we can also set this to Adam. That would be the same as writing it out like this, tf.keras.optimizers.Adam. The benefit of being able to write it out like this is that we can set the learning rate. Remember, LR is short for learning rate, and the metrics can be accuracy. So we got the same thing so far. Come back to our neural network playground, features X. We haven't passed our model data yet, but we have these. These are our blue and red dots. We've got one hidden layer with one neuron, and the activation is linear. The learning rate is 0.001. Okay, now let's see if we can— number 3 is fit the model, and we're going to start to add the history variable here whenever we start to fit models now, because this will come up later on. Just want to start building that habit. All right, let's do it, 100 epochs. I mean, this one's run for over 1,000, but we're just going to stick with 100 so we don't have an incredibly large output in our notebook, and Shift and Enter. All right, what kind of results are we getting? Wow, looks like our model is performing worse than guessing right now. Is that aligned with the— okay. So the TensorFlow Playground is getting quite similar results. They're basically still guessing as well. If you were trying to separate these two, the blue and the orange dots, if you were just tossing a coin for 1,000 different times and you got heads and tails 500 different times, well, you're going to get these— about these results. Hmm. Now let's remind ourselves of what our data looks like, because whenever our models predictions aren't working very well, we can evaluate our model with— by looking at the evaluation metrics, or we can adhere to our model of visualize, visualize, visualize. So our data is 0, get the 0th axis of x and the 1st axis of x, and we'll color it with y, and we'll choose a color set up as plot.color_map red, yellow, blue. So we're just reminding ourselves of what our data looks like. All right, wonderful. And now we've got model 4, which is a trained model, albeit looks like it's not performing very well. Let's check out its predictions anyway. Check the decision boundary for our latest model. So this is why we created a handy function before, our beautiful plot decision boundary function, where we pass it our trained model. This can be model 4. We'll pass it our features, X and Y. What does it look like? Where's our model's decision boundary? Oh my goodness, it's all over the shop. So our model, if Blue is the blue class and red is the red class. I mean, red's right up here. Yellow is the crossover because we've set our color map up as so. Our model is basically going, you know what, anything in this yellow could be blue or red, which is basically why our model's accuracy is below guessing. Hmm, now what can we do? It looks like our model's still predicting straight lines. So what we might try is in the next video, or actually, if you want to jump ahead, reset the TensorFlow Playground and try playing around with activation here. What happens if you set it to a different value? See what happens, and we'll go through that in the next video. In the last video, we had to play around with the TensorFlow Playground, and we saw that setting the activation function with one hidden neuron, one hidden layer, learning rate of 0.001, didn't result in a very good separation of the orange and blue dots, similar to, to the neural network that we built, for our latest neural network that we built. So there's Model 4, we just copied what we've got here. Now, I hope you've thought about this question here. What could you draw if you had an unlimited amount of straight linear and non-straight nonlinear lines? So remember our data is, that we're trying to model for classification is nonlinear. So it's not possible to model with just straight lines alone. If we come back here, we've spoken about nonlinear. Right now our activation is linear. How about if we change this activation function to something that is not linear? So it could be any of these three values, even though we haven't seen what these are. Let's just change it to— let's try the top one, ReLU. Okay, and let's see what happens if we just run this. Hmm, still not getting a very good value. I wonder if we just keep it training for longer. Well, we're almost at 1,000 epochs now, and it's still not improving. But before we explore different ones here, let's see how we might use the ReLU activation in our own neural network. So let's come back to our notebook. Let's just see if we can replicate. We'll take another practice and replicate this neural network here with TensorFlow code. So first of all, actually, we'll put in a note here. Let's try build our first neural network with a nonlinear activation function. And now when I say nonlinear, I really mean just anything that is not linear. So we could choose linear, it didn't perform very well, but if we wanted to choose any of these which are not linear, we're going to start with ReLU, which is very common, as you'll see in your deep learning journey. So set random seed, tf.random.set_seed. Now we're going to create a model, or we'll put number 1 here, so just remind ourselves, create a model, with a nonlinear, so just anything but linear, activation. So we go model5 equals tf.keras.Sequential, and then we go back here, tf.keras.layers.Dense, and we'll just keep it the same as what we did before. We'll set activation equal to— you see here by default it's None. So this is where we're going to set it to nonlinear, and we could do ReLU, like this, or we could do tf.keras.activations.relu. Wonderful. Now, 2 is we're going to compile the model just exactly as we did before. model5.compile, loss equals— what problem are we working on? tf.keras.losses. We're going to use binary cross entropy. And what optimizer have we been using? The Adam optimizer, optimizers.Adam. We're just setting the lr to what we have in the TensorFlow Playground,.001, just the same there. And then the metrics is going to be accuracy. Now, number 3, let's fit the model. Model 5, this is exciting, our first model with a nonlinear activation. Oh, we said we were going to start Developing the habit of saving the history. x, y, epochs equals 100. Let's see what happens. Oh, what did we mess up here? tf.keras.optimizers, a typo, as usual. All right, wonderful. So we keep going. We're using a nonlinear activation function. Remember, in this case, nonlinear is just anything but linear. And again, our model ends up performing basically worse than guessing. Hmm, so it's still not learning. What if we come back to our keynote? Now, we go back to the slide of where we looked at improving a model. So, so far, this is what we've tried. We've tried adding layers, we've tried increasing the number of hidden units, and we've tried changing the activation function as well as the optimization function. Far out, we've tried a fair few things. But we've only just tried changing the activation function. We haven't done that in conjunction with these two. So how about we try that? What if we increase the number of neurons and layers and change the activation function? I've got an idea. We can try that in TensorFlow Playground first. So before the next video, go back to the TensorFlow Playground, and try increasing the number of hidden layers, the number of neurons, keep the activation and the learning rate the same, and everything else the same, but just increase the number of hidden layers to whatever you want. I think there's a maximum of 6. That's all right, whatever you want, and increase the number of hidden neurons to whatever you want, and then run it and see what happens. And if you really wanted to, if your neural network works or if it doesn't, reproduce it in TensorFlow code, and I'll see you in the next video where we're going to do everything we just talked about. Welcome back! How'd you go? Did you manage to, to get the TensorFlow Playground to find the patterns or distinguish orange from blue dots? Did you get the test loss to decrease, the training loss to decrease? Did you increase the number of hidden layers or the number of hidden units? I hope you did, but if not, that's what we're going to go through in this video. Now we've discussed that we've tried to improve our model, and we've tried adding layers, we've tried to increase the number of hidden units, we've even tried to add a nonlinear activation function, 'cause the data we're working with is nonlinear. We've even tried changing the optimization function, but we haven't done all of these in conjunction with each other. So let's have a play around with the TensorFlow Playground and see If we can adjust those parameters, or hyperparameters, see if we can get this to distinguish patterns between orange and blue dots. So I'm going to add another hidden layer, and I'm going to increase this to, let's say, 4 neurons. Yeah, that's a good number, and I'll, I'll do the same for this layer. All right, I'm keeping everything the same, so I've just increased the hidden layer, increased the number of neurons. Learning rate the same, activation is the same, it's nonlinear, so at the moment it's ReLU. And let's press play and see what happens. Oh, whoa, what do we got here? That's a good sign. The test loss is going down, and it's continuing to go down. Haha. Wow, all right, we're nearly at 1,000 epochs. It's still going down. Let's just wait here for a second. Is it gonna keep going down? Oh, look at this! So it's starting to be able to distinguish orange dots from blue dots. Now, once it hits 2,000, I think we'll stop there. You could leave yours running for longer and see what happens, but how cool is that? So just by increasing the number of hidden layers, adding a few more hidden neurons, and changing it to a nonlinear activation function, we get much better results. Now, again, some great practice. Let's replicate this neural network in TensorFlow code. So we'll come back to our notebook, and time to replicate the multi-layer neural network from TensorFlow Playground in code. And we'll set the random seed. It's going to be tf.random.set_seed, 42. And what do we have to do for our model? Well, we had 2 hidden layers with 4 hidden neurons each, and then we had an output, and we have a ReLU activation function, a learning rate of 0.001. All right, that seems pretty straightforward. So create the model. Uh, what number are we up to? I believe we're up to model 6 equals tf.keras.Sequential Beautiful. Now we need 2 hidden layers here. tf.keras.layers.Dense with how many hidden units? 4 or 6? 4. And then activation equals ReLU. Wonderful. I'll just come to the end of this, and then we'll create another one. tf.keras.layers.Dense— forgot the S there— 4 hidden units and activation equals ReLU. Wonderful. Now we'll go step 2, compile the model. model6.compile, just exactly the same as we've done before. Again, getting lots of practice here. Loss can be— we'll use the string notation for, for this model. Cross entropy, binary cross entropy, that is. Optimizer equals tf.keras.optimizers.Adam, and the lr is 0.001, which is Adam's default learning rate, so we don't actually have to put that one there. And then finally, we're going to set the metrics, metrics equals accuracy, and let's fit the model. So we're saving our results to the history variable, model6.fit on x, y, just the same number of epochs that we fit before. All right, let's see what happens here. All right, do we get any improvements? Oh, looks like our model is performing even worse than guessing. So below, what if we evaluate our model? Far out, what's the difference here? Evaluate the model. model6.evaluate, and we'll go x and y. Hmm, why is our model performing far worse than what we— like this model trained, oh, 2,000 epochs. Maybe we need to train our model for longer. What if we did that? What if we come up here, make it 250 epochs? Let's try that. Oh, whoa, our metrics are jumping all over the place. So it definitely seems like our model is still basically guessing. What if we added— you know what I might think it might be? You can't see it in this Neural Network Playground, but I think it doesn't make sense, the fact that we're dealing with 2 features and we're dealing with a binary classification problem, and the last layer has 4 hidden neurons instead of the same— instead of 1 for binary classification. So let's see what happens if we just tf.keras.layers.Dense, because we want our output to be one or the other, right? Not 4 different options. We want it to be red dot or blue dot, not 4. So let's see what happens here. Accuracy still 50%. Wow, even after 200 or close to 250, it's about to be 250 epochs, we're still getting 50%. Hmm. What if we evaluate that? What's going on? What should we do when we're not sure what our model is doing based on the evaluation metrics? We should visualize, visualize, visualize. So how do our model predictions look? So remember, we've got our handy plot decision boundary function. I'm going to pass in model6, x and y, What type of decision boundary is our model creating? Oh my goodness. All right, so it's starting to realize that red might be towards the outside, but it's still operating with, it looks like straight lines. Hmm, what gives? Our model looks like it's the exact one in the TensorFlow Playground too. I mean, ideally, our yellow line would go inside or in between the red and blue circle. All right, let's model this circle once and for all. We're going to build one more model, I promise, but actually we're going to build plenty more models throughout the course, but for this circle, I mean, we've done enough times. It's time to reveal the missing piece here. It is, if we come back to our keynote, we've looked at improving our model, we've added, we've altered the activation functions in the hidden layers, but we haven't changed the activation function in the output layer. So we've set up ReLU, ReLU here. Now what if we came back to our— we looked at this right at the start. I mean, we could have come back to this to begin with, but I wanted to go through the concept of figuring things out. Let's go back to our architecture of a classification model, the typical architecture, that is. So if we have a look, what are we dealing with? We're dealing with binary classification. We've got hidden layers, well, we've got 2 at the moment. We've got neurons per hidden layer, generally 10 to 100, but we've seen on TensorFlow Playground that 4 is enough for this type of dataset, so we'll stick with that. Output layer shape is 1, we, we've set that up. Hidden activation is usually ReLU, okay, we've set that one up. Output activation, Sigmoid. Ah, so on our demo model here, we also have an activation on our output layer, but on the current neural network that we're working with, come back, Model 6, we don't have any activation here. And remember, for a dense layer, the activation is by default none. Hmm, so what should we do? If in doubt, we could refer to our little table that we have here, sigmoid, or we could search something like this. We could go, what activation function to use for binary classification? Now, maybe we go here. What activation function for the output layer? So regression, linear, softmax, Simple sigmoid works too, but softmax works better. Okay, we could dig through this information here. We could go back for another one. Here we go. The output layer contains a single neuron in order to make predictions. It uses— this is binary classification, that's what we're after— the sigmoid activation function. All right, that must be the missing piece that this TensorFlow playground doesn't show us. It doesn't show us the output layer. But that's all right, we've got TensorFlow code. So I want to issue you another challenge. If you want to model this circle once and for all, create a model like model6, but for the output layer, add in a sigmoid activation function. I'll let you do your research and give that a go, but otherwise, we're going to model this circle once and for all by introducing the output layer activation function, in the next video. Welcome back. Last video, we discussed that we're probably missing an activation function for our output layer. However, I hope you didn't take my word for it. I hope you tried to write the code yourself, but if not, let's do it. Let's model this circle once and for all. I'm getting sick of seeing these, uh, these straight yellow lines. So let's, um, set the random seed as usual. You can probably tell that my, uh, coding hands are eager to write this neural network and get this circle modeled. So let's do step 1, create a model just as we have before. I believe we're up to model 7. Fingers crossed this is lucky model 7, you know. And we're going to go sequential, go there, tf.keras.layers, just going to be the exact same hidden layers as what we've been using before. The activation equals ReLU. Wonderful. Come back. tf.keras.layers.Dense, 4 as well. Activation equals ReLU. tf.keras.layers.Dense. We want 1 for the output layer because we're dealing with binary classification, one thing or another. And here is where we're going to introduce the magical. We could do tf.keras activations.sigmoid, or if we wanted to keep in line with the string notation that we've been using, we could just turn this into sigmoid. Now again, where does this come from? We could ask Google, or we could check out the typical architecture of a classification model. So binary classification, hidden activation usually ReLU, or the output activation is sigmoid for binary. When we deal with multiclass, we'll have to deal with softmax, so keep that in mind going forward. Now we'll come back. All right, our model architecture is looking great. Let's compile it. Compile the model, model7.compile. Now we want to go loss tf.keras.losses. No, we're using string notation. Come on, Daniel. Let's go binary_cross_entropy. Wonderful. And then optimizer can be tf.keras.optimizers.Adam. We'll set the learning rate to be 0.01. Again, that's the default. We don't necessarily need to do that, but just keeping in line with staying true to our TensorFlow Playground setup. And we will go metrics equals accuracy. Wonderful. Now let's fit the model. I'm eagerly awaiting to see if the results for this one are better than our previous model. This is the fun part of neural networks and deep learning is running so many different modeling experiments. epochs equals 1. Oh, let's watch the training. I was about to set verbose equal to 0, but let's watch it. If this neural network's gonna work, I wanna see those metrics going where they should go. Accuracy, it starts, are we going up? Yes, oh my gosh, look at that. Look at that. We finished with an accuracy. Before we added that output activation function, we were getting about a 50% accuracy, and now we're borderline 99%. But again, let's not trust just the metrics. Let's try— number 4 is evaluate our model. So model lucky number 7, that's where we're at. Evaluate X and Y. Are we getting the same? Wow, we are. Loss below 0.3 and accuracy is 99%. How does that line up with our TensorFlow Playground? So that loss is getting almost 10 times lower, but this is fit for 2,000 epochs. Maybe ours would somewhat approach that if we kept training. But again, let's not just trust the metrics, let's visualize. Let's visualize our incredible metrics. So again, we created this beautiful function a few videos ago because we want to use it multiple times. Plot decision boundary, model number 7, X, Y. Are you ready? 3, 2, 1. Oh my goodness, how much better is that? So it looks like our model has basically perfectly found the decision boundary between the red and blue dots, except for maybe a couple of points. Oh, that one, that one got caught, and that one got caught there. So that's why we're not getting a perfect result, but we're very close. That's 99% accuracy between two evenly spread classes or two evenly spread labels. That's pretty darn good. But I have a question for you. I wanna put this down here. We've discussed this previously, but I'm gonna put the question emoji. It's a little challenge. So question equals, or equals, I'm too used to saying equals. What's wrong with the predictions we've made? Are we really evaluating? So if we're looking at this, our evaluation metric and our plotting of predictions, are we really evaluating our model correctly? Here's a little hint: what data did the model learn on, and what data did we predict on? Now, have a think about that. You probably know the answer already. If not, perfectly fine. But before we, we answer that, I want to emphasize what we've just covered, which is back to the question we had before. If we go to our non-linearity slide, I posed this question a couple of videos ago: what could you draw if you had an unlimited amount of straight linear and non-straight nonlinear lines? And then we looked at our linear data and we looked at our nonlinear data. So the combination of linear straight lines and non-straight lines If we go back here, so nonlinear functions, remember we haven't even discussed what ReLU is or what sigmoid is. We just know that it's not linear. So we come back, linear, and these three are not linear. That's all we've discussed for now. But this is a— I'm gonna even write this down. So let me put a key here. You wanna take away, anything from the few videos we've just gone through, is that the combination of linear, which is straight lines, and nonlinear, which is non-straight lines— that's all you need to know for now— functions is one of the key fundamentals of neural networks. And notice key is on purpose because I've got key there. Now, back to our question. Just think of it like this: if I gave you an unlimited amount of straight lines and non-straight or non-linear lines, you could essentially draw any pattern that you wanted to. Now, that is essentially what our neural networks are doing. This dataset is relatively easy, but if you imagine if we're working on a whole bunch of different other datasets such as Building a neural network to understand what's in a picture. There's almost an unlimited amount of things. Look up, if we go food images, if we want to build a neural network to identify different patterns in pictures of food, look how many different patterns there are here. We need a whole bunch of non-straight and straight lines. So that's the essence of what a neural network is doing when it looks at different examples of data. It's drawing patterns with straight and non-straight lines through it. So if this doesn't really make sense for now, you might even be thinking, hey Daniel, I've never actually seen a linear function or a nonlinear function before. Well, you kind of have. We've been using the whole time. They're what power the layers we've just built. So with that being said, In the next video, let's take a look at applying these linear and nonlinear functions that we've been using in our neural networks, but just applying them on their own. So I'll see you there. In the previous video, we modeled our nonlinear classification data once and for all. I mean, look at that, that's a beautiful sight, isn't it? Our decision boundary is basically splitting them perfectly. Now, We've also discussed the concept of linear, straight lines, and nonlinear, non-straight lines, or functions, but we haven't really built up an intuition about them. We've just, we've just got familiar with the names of some of them. We've looked at ReLU, and we've looked at sigmoid, but let's start to build up an intuition of them. So right here, now we've discussed the concept of linear and nonlinear functions, lines, let's see them in action. So to do so, how about we create a toy tensor, which is very similar to the data we pass into our models? Because we're dealing with TensorFlow, all of the data we use gets encoded into a tensor. We pass that tensor to a neural network, it figures out patterns and outputs another tensor. So let's go a equals tf.cast, I'm gonna go tf.range. I want just a tensor, a nice one from -10 to +10, and I want it to be of float32. Let's have a look. Okay, nice and simple, just -10 all the way up to 9 because it includes 0. But it's 20, 20 long. Now let's have a look at what it looks like. Visualize our toy tensor, plt.plot. Wonderful, nice and straight line. What would that be called? What's a straight line? It's a linear line, right? Now, what activation function did we just try? In our output layer? The sigmoid. Okay, so how might we apply this sigmoid activation function directly to tensor A? I've got an idea. Why don't we look that up? TensorFlow sigmoid activation function. TF Keras activation sigmoid, okay. Wonderful. Oh, the sigmoid activation function. Sigmoid x equals 1 divided by 1 plus exponent to negative x. Okay, what if we just did sigmoid activation function? What does that look like? We're going to get some fancy math notation, I'm guessing. Yeah, there we go. Okay, so sigmoid z equals 1 on 1 plus e to the power of negative z. We get a line like that. So see how that's nonlinear? All right, enough looking at pictures. Let's try to replicate this. We could do it, TF Keras activation sigmoid, but how about we start off by just replicating this? So what I would do in a case scenario like this where I'm trying to replicate something, I would go, let's start by replicating sigmoid. And I put this here. So I'm trying to build a function that replicates this here. I was just trying to point at the screen, but I realized you can't see that. So let's go def sigmoid, and it's gonna take an input x. And then how about we just, it's relatively, it's only this, 1 divided by 1 plus. Now to get the exponent, We— let's look that up. Can we do it in TensorFlow? TensorFlow exponent. tf.math.exp. Wonderful. Does it have an alias? Oh, tf.exp. All right, exp. So if we go tf.exp, pass in negative x, is that our sigmoid function? Really? That's all it is? So let's try it out. Use the sigmoid function on our toy tensor. So we created A up here before. Remember, this is what A looks like right now. It's linear. Now if we use our sigmoid, sigmoid on A, all right, we get a whole bunch of what looks like gibberish numbers, but let's plot them. That's a much better way to look at them. How do we do that? Plot our A, our toy tensor transformed by sigmoid. So plt.plot, we want to go sigmoid. What do you think this will look like? Give you a second to think about it. Let's check. Ho ho, that's what's up. Now notice here that the values are between 0 and 1. Hmm. That might be something we look at later on, but the most important point here is that this line was originally straight and has now been modified to be non-straight. Now, where is the intuition there? If we come back, remember how we couldn't draw a curved line around our data, but as soon as we added this sigmoid activation function to our neural network, it was like, Now that we've seen what it does to a straight line, it was like we gave our neural network a tool to go, "Hey, you've been trying to draw patterns with just straight lines before, but now you've got this non-straight line, so use that as best you can to find better patterns." And as it turns out, it was able to find better patterns. So let's not stop there, let's keep going. What was the other activation function that we've used so far? We come up here, we've used ReLU activation. All right, so let's see, how can we replicate this? 'Cause that looks pretty cool. I wonder if we can do the same for ReLU. Let's go another tab, TensorFlow ReLU function. And what do we get? tf.keras.activations.relu, all right. Well, it looks like we can do very similar notation to our sigmoid function. But does it have a definition of what ReLU is? Oh, it's just the max between x and 0. Oh, so does that mean it's going to make all negative numbers 0? Hmm, or we could even just go, what does ReLU do? In a neural network, the activation function is responsible for transforming the summed weighted input from the node into the activation of the node or the output for that input. Hmm, we could dive into that. ReLU is the max function, x0. What if we go images? Is there a function of it? Equals 0 for x less than 0, or x for x equal or greater than 0. Okay, why don't we just replicate that? Max, 0, or zed. Okay, let's give that a go. Let's recreate the ReLU function and see what happens. Oh, one thing before we do, notice what this line looks like. How's it look compared to our original tensor, our toy tensor that we created above? So maybe we go def ReLU, and we want to return How do we get the maximum in TensorFlow? We can just do max, is it maximum or just max? Maximum, wonderful. We want to return the maximum between 0 and x. That's all we've done for ReLU. If we come here, we want to go all. Is there a Wikipedia page? Wikipedia is usually pretty good. There we go, rectifier neural network. This is all we've written. Relu equals max. That's it. That's all we've done. Okay, now let's pass our toy tensor to our custom relu function. relu. What's going to happen? What do you think is going to happen? Well, let's check it out. All right, how is this different to our original tensor? Just by looking at it, this one's a bit easier to look at than the sigmoid. But see how these are all 0? It seems like ReLU, all it's done is it's turned the negative numbers to 0. And that we've done that with the maximum function by just going, hey, look at the input you're getting and give me the maximum of 0 or the number itself. So is 0 larger than -1? Yes, it is. So we set it to 0. All right. Now, better still, let's see how this looks. So plot_relu_modified_tensor. I'm gonna go here, plot, plot_relu_a. Let's see. Haha. Now, what is that? We come back. Well, let's just look at, bring our straight line tensor down here. So this, it started out straight, and now this is, you could still argue that this is straight, but it's got a kink in it. So now we have a bunch of tools. We've given our neural network with, hey, here's this curvy line, and here's this bendy line. Now you've got these two tools. Let's start drawing patterns in our data. I mean, I could draw some pretty cool shapes if I had this curvy line and this bendy line. Now let's not stop there. We've got one more activation function that we've tried. Remember, we're now in our neural network playground. The first thing we tried was linear, and then we tried nonlinear. So let's, let's do that. Let's, for completeness, let's see how— let's try the linear activation function. So might just do what we did before, TensorFlow linear activation function. TF Keras activations linear, linear activation function, pass-through. What does it do? Arguments: x, the input tensor, returns the input unmodified. Are you serious? We just put a tensor into the linear activation function and it just returns the same tensor. Did that even need a function? Well, we said we're working for completeness. We might as well try it out. We go tf.keras.activation.linear. It's not even fun to replicate this one. Has no, has no model activation or activations. That's what we need. We need the S. Come in here, Shift and Enter. Wow, are you serious? That's all that does, just our exact same tensor. Well, for completeness, let's, uh, does the linear activation function change anything? So plot, plot tf.keras.activation.linear, a. We'll see. Oh, again I'm making the same error. Hold on, this can't be real life. Let's go, does a, does a even change? a tf.keras.activations.linear. All of the elements are still the same. I guess, well, the linear activation function lives up to its documentation, returns the input unmodified. But this is a fundamental concept of what we've, what we've just covered. Now, let's come back to our slide. I've just prettified all the code we've just written, but the important thing is that we've written this code and we've seen it happen. But it makes sense that the model didn't really learn anything using only linear activation functions, because the linear activation function doesn't even change our input data in any way. So it's just basically passing the same input data through the entire neural network, and the outputs, no wonder they're basically as good as guessing, because it hasn't changed a single thing. Whereas with our nonlinear functions, such as sigmoid for the output layer, we give our neural network this, this tool here of using this curved line, and the same thing here for the ReLU function. When we give our model nonlinear functions, it's able to deduce patterns in nonlinear data. Whoo! That's a fairly important concept. Now, we've only covered two activation functions here, but the premise of what we've just seen is the concept of nonlinearity is— we've covered that. So that's the main takeaway you need to take away from this series of videos. Is that neural networks use a combination of linear activations and nonlinear activations to find patterns in data. Now, if you want a resource for learning more about activation functions, there's the Machine Learning Cheat Sheet Activation Functions. I'll leave this in the resources section, but mlcheatsheet.readthedocs.io actually has a whole bunch of different stuff. but here are some of the most popular and useful activation functions. We've seen linear, ReLU, leaky ReLU is a, is a different form of ReLU, and sigmoid. I'll let you go through here. Maybe some of your extracurriculum could be to reproduce these in TensorFlow code. However, we're going to push on, and in the next video, we're going to see how we can evaluate and improve our classification model. This one that we've built here. So with that being said, I'll see you in the next video. In the last few videos, we tackled the important concept of non-linearity, and we learned that the combination of linear straight lines and non-linear non-straight lines functions is one of the key fundamentals of neural networks. Or other words, how they find patterns in data, and we even saw a few different examples of linear functions. We even rebuilt our own nonlinear functions in the sigmoid activation function and the ReLU activation function. But now it's time to evaluating and improving our classification model. Alrighty, so do you recall that in a previous video I posed the question of What's wrong with the predictions that we've made so far? We scroll back up again, jumping all over the place here, but I did pose a question before. What's wrong with the predictions we've made? Are we really evaluating our model correctly? Hint: What data did the model learn on and what data did we predict on? So, so far in our toy example, we've been training and evaluating or training and making predictions on the same dataset, but why is that wrong? We'll come down here. What should we do? What type of data or what dataset should our model learn on and what dataset should we evaluate our model on? Now, if you answered that question with, "We should train our models on the training dataset and test our models on the test dataset," you'd be 100% correct, but at the moment, we don't have a training or a test dataset. So let's remind ourselves of the 3 datasets. Possibly the most important concept in machine learning, and I know I say that about a lot of the concepts we've talked about, but this is probably the number one thing to do with data. So we've got the course materials, as if you were studying at a university course, is the training set. The validation set is a practice exam, and the test set, the test dataset, is the final exam. And the goal here is for our machine learning model or deep learning model to generalize. In other words, the ability for our model to perform well on data it hasn't seen before. So what do you think we should do to properly set up our machine learning model training and testing? Well, if you guessed we should create— actually, let's write ourselves a little note here. So far we've been training and testing on the same dataset. However, in machine learning, this is basically a sin. So let's create a training and test set. All right, so how many examples do we have? Let's, uh, check how many examples we have. We can get length of x, that'll tell us Beautiful. It's 1,000 because we use the make circles function all those videos ago to create our data. Now we could, because our data is in random order, order, right? If we look at that and we look at y, we could randomly split this using scikit-learn's train test split, this one here. or we could create a train and test dataset by indexing. I'll let you choose how you do yours, but I'm going to create mine using indexing. So split into train and test sets, and then I'm going to go here. I'm going to set X_train and Y_train equal to— we're going to do an 80/20 split. So 80% of this, of our samples is going to be training data, and 20% is going to be testing data. So the first 100 samples will be— or 800, sorry— of X and Y will be training samples. And then we'll do the same for X_test and Y_test. However, these will be the last 200 samples, so from index 800 onwards. Wonderful. Now let's check the shape of X_train and X_test, and then Y_train shape And y_test shape. What have we got at the output? Okay, X_train and X_test. So there's 800 examples in X_train, and it's of a shape 2, that's excellent. And 200 examples in X_test, and then Y_train has 800 labels, and then Y_test has 200 labels. Beautiful. So now we've got a training and test set. How about we recreate a model fit on the training data, and then we can evaluate it on the, the testing data. This one here. So let's write that there. Let's recreate a model to fit on the training data and evaluate on the testing data. How we should have been doing things right from the start. So first of all, we'll set the random seed, tf.random.seed. Set_seed 42, and then we'll go number 1 is create the model. Now again, we're retyping the same code because we're making the same model as lucky_model_7, but we're getting ourselves— or we're getting a lot of practice writing TensorFlow model code, which is very important because we are doing a TensorFlow deep learning course. So if we go over here And what was it? tf.keras— well, we could find model7.summary. What's this going to tell us? Okay, so we had 2 hidden layers with 4 hidden neurons each. So let's recreate that. layers.Dense. And what was our activation function? It was nonlinear, ReLU. Come back here. Then we'll create another layer, tf.keras.layers.Dense, activation equals ReLU. And then what was our output activation? We recreated it again. If you need to, refer to the slide which, which has the activation for the output layer of a binary classification model, but it was sigmoid. So now let's compile the model. So we want model8.compile. We want to set the loss function to binary cross entropy because we're working with a binary problem. And then we want to set the optimizer equal to tf.keras.optimizers.Adam. However, we're going to make a change here. We're going to change the learning rate. So Adam's default learning rate is 0.001. But I have a feeling that if we increase it to 0.01, our model might be able to discover the patterns it found in our data faster than what it did previously. So with Model 7, we fit for 100 epochs. We come back up here, we fit for 100 epochs, but it looks like our model was a little bit slow out of the gates. So it really didn't start learning much. So it's still in the 50s here, and then it's not until about halfway that it starts to really increase up past the 60s in about 5 epochs, and then the 70s in about another 10 or so epochs, and then it gets really close to 100% accuracy. So the reason why Remember how we discussed what the learning rate is? It's okay if you don't, but we mentioned it briefly in a previous video. I'll set the metrics here, but the learning rate dictates— because let's actually, let's take a step back. The optimizer tells our model how it should improve or how it should update its internal patterns that it's learned. The loss function says how wrong these, those patterns are, and then the optimizer says, hey, you should improve them in this way, and the learning rate is how much our model should improve those patterns. So if we set the learning rate to be a lower value, say Adam's default, such as 0.001, this might be the equivalent of saying, hey, every time you take an epoch, improve your weights by 0.001. So if we were to change it to 0.01, we've increased it by 10%. So basically, every epoch, we're We've given our model the potential to improve its weights by 10 times as much. Now that's not exactly how it works, but that's how I intuitively understand the learning rate. And in practice, you'd be surprised how much that simple definition of going, hey, the higher the learning rate, the more our model will update, works in practice. And by the way, Adam's default learning rate of 0.001, and these other default parameters here, are actually very good for the majority of problems that you'll, you'll work on. Of course, as you start to get more and more into the, the deep learning world, you can start to tune these to your problems. But you'll find that a lot of the values that we use in TensorFlow, all of these parameters that are hard-coded already, have been experimentally found as very, very, very good. So now let's go here. We're going to fit the model, model8. .fit. We're going to fit on the training data. Oh, what have we been getting into the habit of the last few videos? That's right, setting the history variable. y_train epochs 25. So I've increased the learning rate by 10, but decreased the amount of epochs our model is going to look at, or in other words, the amount of times the model is going to go through the training data by 4 times. So let's see if it can still get just as good results with only 25 steps versus 100 steps. What have we done wrong here? Oh, we need to set this as model 8. All right, where do we get to? Oh, do you notice what's happened here straight away? So we have epoch 1 out of 25. We start at 54% and then we're into the 70s after only 8 epochs. And then we're into the 90s after only 15. I mean, if we go back up to our previous model, we didn't reach the 90s until epoch 77. Wow. Okay. So if we come down here, look at that. By the end, we're borderline 98% accuracy after only 25 epochs. Now, the real test here, of course, is for evaluate the model on the test dataset. So let's go model8.evaluate. Now that we have a test dataset, X_test, Y_test, how does it perform? Wow. So on the test dataset, in other words, data our model has never seen before, it gets an accuracy value of 100%. That means every single test value, it's— it got correct. Out of 200, it predicted them all correct. That is amazing. Now we can really evaluate this. How, how can we do so? By plotting the decision boundaries. We've done this for our complete dataset before, but now let's do it both for the training dataset and the testing dataset. Plot the decision boundaries for the training and test sets. plot.figure, figsize, equals 12, 6. Beautiful. Plot.subplot . Plot.title, let's go train data. So here we're just saying, well, hey, we want to create a subplot. We want it to have, I believe it's rows, columns. So we want it to have 1 row, 2 columns, and the first value is going to be the training plot. Plot decision boundary, and then we're going to pass it model8, our trained model. X is going to be X_train, our training data, and then Y is going to be Y_train, our training labels. And then we'll set up the second subplot. This is going to be 1, 2, and the second plot, so 1 row, 2 columns. The first one is training, the second one is going to be test, and then again plot decision boundary, model 8, x equals the test dataset, so x_test, y equals the test labels. Beautiful. And let's show our plot. Doing binary classification. Oh, would you look at that? So our model's decision boundary, the yellow line, goes through the training data. It misses a few examples here. I think it might have missed that one, the red dot, and I think this blue dot. So that's why our model doesn't get 100% results on the training dataset, but that's okay. We want our model to generalize to data it hasn't seen before. So when we look at the test dataset, it gets about 100%, or it does get 100%, not about 100%. It gets 100% accuracy. Excellent, and that was after 25 epochs. So the main takeaway is that all we did was we, we took our Model 7, the model that was working, but we increased the learning rate by 10 times. Now again, this is one of those hyperparameter things that won't always work. Our Adam's default learning rate is actually very good for the majority of problems you're working on. The only reason I set this to be a little bit higher is because I had an inkling that our model might learn a bit faster because it performs so well on our dataset. So again, in practice, you may take a little bit of, uh, may take a little bit of tweaking for the learning rate, but we'll see in an upcoming video how we can design a function to find the ideal learning rate value for us. But in the meantime, let's, uh, Since we've been saving our model's training history to this history variable, in the next video, let's see how we might visualize that training history. You may have noticed in a few of the previous videos, and whenever we called the fit function and created our different models, we've been getting into the habit of setting history equals model8.fit something, or model whatever number.fit. Now I think we have seen this before, plotting the history variable, but we haven't discussed it very recently. So let's go through that now. Let's see how we can plot the loss, or also referred to as training, but probably most often referred to as loss curves. Let's go here. Now the reason why we're doing this, if we come back to our keynote, let's find our Where's our Data Explorer's motto? First of all, what is our Machine Learning Explorer's motto? It is visualize, visualize, visualize. And we've had a little bit of experience visualizing our data and our models and our predictions, but not so much training. So that's what we're going to cover here. And of course, it's a good idea to visualize these as often as possible. So let's see. What the history variable is. But first, to understand it, how about we look up the docstring of TensorFlow fit function? What does this give us? Here we go. tf.keras.Model. So this is the model class which our sequential model is built off, and then when we call fit off this. So if we look for fit, um, Does it have here? Once the model is created, you can config the model with losses and metrics with model.compile. We've seen that in practice. Train the model with model.fit. That's not what we want. We want the fit function. We want the fit function to discuss what it returns. Here we go, fit. So we can actually pass a fair few things to fit. If we look here, if we go X, Y, batch size, a whole bunch of things. We'll see a fair few of these throughout the course, but if you want to skip ahead and read them, You definitely can, but I want to see what it returns. Where do we go here? Returns. Here we go. A history object. history.history attribute is a record of training loss values and metric values at successive epochs, as well as the validation loss values and validation metrics, if applicable. Beautiful. So that's why We've been setting up this history variable. Now, history, the documentation just said that history.history. So what happens if we go history.history? What does that give us? Ah, accuracy. Okay. So it looks like there's about 25 values there. So I think it tracks at every epoch, which is beautiful. So really what history tracks for us is this output here. And it's good to look at these in numerical form, but we can also look at them in visual form. So let's convert the history object into a DataFrame, make it nice and tabular so we can structure it up. pd.DataFrame, and we want to go history.history. Let's see what it looks like. Beautiful. So we can see our model's loss started at around 0.7 and decreased right down to about 0.14. And the accuracy started at about half, 50%, and then increased right up to 97%. And that was on the training dataset. So now, how about we look at the loss curves? We just plot this. So plot the loss curves. Pd.DataFrame history.history, and then we're going to just go simple plot. I believe by default it's going to be a line plot. So this is going to be model8 training curves, or we'll call them loss curves because that's what we've been calling them, loss curves. And let's have a look. Wonderful. Now that is Probably the ideal loss curve scenario we want to see for a binary classification problem, or actually most classification problems, because accuracy metric is going up and loss is going down. So actually, for many problems, I think this is a key we can add in here. Let's write— I'll write in here, note the loss function, because remember, what is the loss function? It's how wrong our model is. So for many problems, the loss function going down means the model is improving. The predictions it's making are getting closer to the ground truth labels. Now you might be wondering, okay, I understand that, it's good to see the loss curve going down, but what other values or what other value can I generate from looking at plots like this? Well, in future videos, we're going to see how we can compare multiple different models and check out their loss curves. So the value in this is if we are running, say we ran 10 experiments at the same time, and we plotted all of our models' loss curves together, and say model 8 had a learning rate of 0.01, model 10 had a learning rate of 0.001, and we noticed model 8's loss decreases far quicker than model 10's, but after about 100 epochs, model 10 starts to catch up. So that's where we'd sort of be able to use this visual knowledge to guide our future experiments. So just keep in mind that whenever we set the history variable, we can inspect our model's training curves by plotting them like this, and we'll see in the future another way to do this with TensorBoard, but for now, let's just leave it there, and in the next video, we're going to check out how we can use loss curves to find the best learning rate. So I'll see you there. In the previous lectures, we've seen how much the learning rate hyperparameter can influence our model's training. So wouldn't it be great if we had a method that we could use to find the ideal learning rate? I mean, a value which, when our model started training, meant that its loss decreased as fast as possible. —because remember, the loss is a measurement of how wrong our model is—so if we want to decrease the loss as much as possible, what value could we set the learning rate to be? Hmm, well, to do this, we're going to have to visualize our loss decreasing and potentially decrease our learning rate during training. Now, have we visualized the loss decreasing? I think we have. There we go. Okay, so we visualize the loss decreasing. Now, how might we decrease the learning rate during training? Because so far we've only hard set the learning rate, so using something like lr 0.01. But I haven't actually introduced you the answer to this question that I just asked, so don't worry if you're not sure. But that's what we're going to do in this video. It's finding the best learning rate. And to do so, I'm going to introduce a new concept which is called— to find the ideal learning rate, in other words, it's, uh, the learning rate where the loss decreases the most during training, because remember, that's the loss metric, we want that to decrease during training. We're going to use the following steps. So the first one is a learning rate callback. Now callback is the new concept that we're going to talk through this lecture, or we're going to code it out actually. Now callback, if you're wondering what it is, is you can think of a callback as an extra piece of functionality you can add to your model while It's training. So when our model trained before, it went through the data and found patterns, but it'd be great if we could execute some other kind of functionality while this training is taking place. That is where callback comes into play. So that's the first one, a learning rate callback. We'll see that in a second. And we're going to need another model. So we could use the same one as above, but We're practicing building models here. And finally, we're going to need a modified loss curves plot. So very similar to what we've got here, this loss curve, but we're going to have to modify it because what we're going to set up is a learning rate callback to start at a certain value of a learning rate and gradually decrease or increase that learning rate during training, and then We'll make another plot to plot the loss versus the learning rate to find out the learning rate value that the loss decreases the most. Now, if that didn't make sense, remember, what's our motto? If in doubt, code it out. So let's get started. We're going to create a new model. So set random seed, tf.random.set_seed. Beautiful. Now we're going to create a model. Now it's just going to be the same as model 8. Remember how I said we're going to have heaps of practice coding our models? Well, we're almost up to 10 models so far in this one little section. So model9 equals tf.keras.Sequential. And that's really the only way to get better at programming or any kind of machine learning or data science is to just to keep writing more code, keep working on different things. There's no secret here. ReLU. We could go up and see what it is, but, or you could just follow along with what we're doing. So looks like it's got, oh, we forgot the layers. It's got 2 dense layers with 4 hidden units with the ReLU nonlinear activation. Beautiful. And then we have an output layer with 1 hidden unit, and we're using the sigmoid activation. Beautiful. So now what's next after we create the model? So we're going to compile the model just the same as before, except now we're using model9. Model9.compile loss, we're working with binary, so we're going to do binary cross entropy. And then we're going to put in here the optimizer. Let's use Adam. We use the text-based version of Adam. Metrics equals accuracy. Beautiful, and now, here's the step that's going to be different. So we're familiar with these steps here, but we're going to introduce, as I said, a learning rate callback. There are many different types of callbacks, but for this one we're using a learning rate callback. And so, a callback works during model training. So to get it to run during model training, it has to exist before model training. So before we call, model.fit, yeah, yeah, yeah, our callback has to exist. So let's create that. Create a learning rate callback. So we're going to call it lr_scheduler because, as you'll see in a second, if we go tf.keras callbacks, there is a callback called learning_rate_scheduler. Beautiful. And now how do we find out the docstring of this? We can press Command+Shift+Enter or Command+Shift+Space, sorry. It came up automatically. So this is learning rate scheduler. At the beginning of every epoch, this callback gets the updated learning rate value from schedule. So schedule is this parameter here, function provided at init, with the current epoch and current learning rate, and applies the updated learning rate on the optimizer. Okay. So this is kind of a way of saying every epoch, if we put in some functionality here to change the learning rate, it's going to— this callback, if it's running during the fit function, it's going to give our optimizer, in our case, Adam, the updated learning rate. So let's see what we might do. What we're going to do is do lambda epoch— oop, lambda needs a b— epoch, and then we're going to go 1e So this is just 10 to the power of negative 4 here, and then times 10 epoch divided by 20. Wonderful. So essentially what this is saying is to, or for the learning rate scheduler, every epoch to traverse a set of learning rate values starting from 1e-4 and increasing by 10 to the power of epoch divided by 20. Every epoch. So let's see what that looks like. Now we go here, fit the model. So this time we're going to pass lrSchedulerCallback, model— oh, we're getting into the habit of saving history— history9 equals model9.fit. Now let's pass it our training data and training labels. Epochs, we'll run it for 100 epochs, and then callbacks. Now callbacks come as a list, so we're going to pass it in lr_scheduler, because you can pass multiple callbacks here. So say you want a callback2, callback3. I'm not sure why I'm putting an underscore between call and back, it's actually just one word, but that's all right. We don't have callbacks2 and 3, We only have one. So let's just see what's going on here. We run this. Beautiful. Now our training appears to be just working as normal. This is— we're very familiar with this. We've seen this before. But I want you to have a think about it. What might be different now? If we passed it our learning rate scheduler here, what's it going to do every epoch? It's okay if you're not sure, because remember our motto: if in doubt, code it out. Now what we can do is check out the history. Remember we saved it to history9, so we'll turn it into a DataFrame so that we can plot it. history9.history, and then we're going to go plot figsize, only my favorite two numbers in poker, 10, 7. Also just a a handy size to put out in this kind of window. xlabel equals epochs. Beautiful. Let's see what this looks like. Oh, OK. So this looks a little bit different to the loss curves that we plotted before. It's the exact same sort of plot that we created up here. So this is our model 8 loss curve. So you see here, history.history. Now this is model 9. History9.history.plot, we've just given the x label epochs. So if we look at this axis, the y-axis, our learning rate starts very low, basically zero, and then as the epochs go on and on and on, it starts to increase. Okay, so that's essentially what this code up here is doing. So it starts at a low number, 1e-4, and every epoch it increases by 10 to the power of epoch divided by 20. So that's why towards the end we get this exponential curve, it starts to increase really fast. But what happens here with our accuracy? Our accuracy seems to go up slightly, but then goes down, and the loss goes down fairly significantly here, and then it stays low, but then goes back up. Hmm, so looking at this, what did we want? What do we want before? We wanted the learning rate where our model's loss decreases the fastest. So potentially, this value here, whatever it is, the learning rate at, say, let's say 45 epochs, maybe. This is where our learning— our loss seems to be decreasing the fastest. I've got an idea. Let's plot the learning rate values during training versus the loss. So how might we do that? Hmm, let's go plot, because we can't really see what's going on here. Let's get accuracy out of there and let's just compare learning rate to loss. So plot the learning rate versus the loss. So our LRs is— what did it start at? It started at 1e-4 times 10 to the power of— now we're going to have to do— we need a range, so tf range 100, because that was how many epochs we did, divided by 20. How does that look? That should give us— there we go, shape 100. Beautiful. So if we go len LR's— LR's short for learning rate— there we go, we have 100 different values of learning rate. So see how it starts here at 1e-4, and then it slowly increases as we go along. All we've done with this line of code here is we've just replicated the same thing we passed to the learning rate scheduler, except that we had to substitute in 100 as an integer for epoch because we set 100 here. That's all we've done. So these are our different learning rates that our model tried out. Now what can we do? How about we create a plot? plt.figure, figsize, equals 10, 7. Wonderful. Now we're going to do a semi-log x plot. So that means— this just means we want log on the x scale. Make a plot with log scaling on the x-axis. You'll see what I mean by that in a second. So log x, lrs. So we're just passing it this that we created, which is just a tensor of this shape. May have to be NumPy, I'm not entirely sure. We'll find out. If in doubt, run the code. Now we're going to go history, we're going to get the loss. So this is what we want. So on the x-axis we want LRs, and on the y-axis we want the loss values from our model history. Beautiful. Now we're just going to decorate our plot with xlabel can be learning rate, and the ylabel can be loss. And finally, let's give it a title. Plot.title, learning rate versus loss. Beautiful. What does this look like? Aha. Okay. Now, before we even discuss this, I want you to maybe pause the video for like 10 seconds and just have a look at it and have a think about where do you think our ideal learning rate would be? Totally okay if you're not sure, but just have a look. Remember what we're trying to do here is we're trying to pick a learning rate value where the loss decreases the fastest or the most. So what, what part of this graph is the loss decreasing the most? So have a think about that and press play when you're ready to go. How'd you go? Well, I'll tell you the methodology. To figure out the ideal value of the learning rate, or at least the ideal value to begin training our model with, The rule of thumb here is to take the learning rate value where the loss is still decreasing, so maybe here, but not quite flattened out like it, like it looks here. And it's usually about 10 times smaller than the bottom of the curve. So in our case, our ideal learning rate would be somewhere in this section here. So it ends up being between 0.01, so 10^-2, so this value, and 0.02, so it'd be about this value here. Now, I actually did prepare something that highlights this a bit better earlier. Finding the ideal learning rate. Here's the code that we ran. There's just our plotting code. Boom! This is what I was talking about. So this is the lowest point on the curve, somewhere down here. But remember, the ideal learning rate is somewhere about 10 times smaller than that, so we have to go back here. So it's somewhere in there. Now if we go back, if we said that the ideal learning rate is 0.01 or 10 to the power of negative 2, what was the learning rate we set above here that our model got really good performance? Where's model 8? Wow! We set our learning rate to be 0.01. How phenomenal is that? You could call it a lucky guess, but there's another little heuristic we can go here. We can either find the best learning rate through this methodology here, or we could use the default value learning rate, or just take a guess, because an example of other typical learning rate values are 10 to the power of 0, 10 to the power of -1, 10 to the power of negative 2, 10 to the power of negative 3, and 1e negative 4. If we have a look at these, so see here, what do these have in common? They're all multiples of 10. Now, you could have a learning rate that's 0.03, or you could have a learning rate that's 0.25. Realistically, there's a whole bunch of different ranges of learning rates that you could use, but why Daniel, you've given me these values, but you've said your learning rate can be almost anything. The reason why I'm highlighting this is because whenever you use an optimizer, say, let's go TensorFlow Adam. If you use a pre-built optimizer, as we've discussed before, their default parameters are generally pretty good, as in, for most of the time, They'll work pretty well, but for when they don't, well, you've got some other learning rates that you can try here. You could just try hard coding these. Typically, you probably won't ever use 1. You'll start to use below 1, so starting from here and keep going down, or you can use this methodology to find the ideal learning rate. So up here, by using a learning rate scheduler, have it decrease during training, and then plot the log learning rate versus your model's loss, and find the value where— or the learning rate where the loss curve decreases the fastest. Or in other words, just like this graphic here. So see here, loss decreasing very fast. The ideal learning rate is going to be somewhere between the lowest point in the curve and about 10 times smaller than that point. So have a practice with that. Potentially you could train another model on some other data that we've worked with and find the ideal learning rate. But, now that we've found the ideal learning rate, let's fit another model. Actually, you can probably try that before we go there. So, pick a learning rate in this little section here, create a new model, and fit it to our data, and I'll meet you in the next video. In the last video, we saw how we might be able to find our model's ideal learning rate by the learning rate scheduler callback, and starting with a learning rate of a fairly low value, and slowly increasing it every epoch, and then plotting the different learning rate values versus our model's loss. We also discussed that the ideal learning rate is somewhere between the lowest point on the curve, and if we jump back to a region where the loss is still decreasing. So how about we try building a model with a learning rate of— so if we look at this point on the curve, it's still— the loss is still decreasing sharply. So that's the value we actually set before. If we come back up to Model 8, Model 8, there we go. So this is the learning rate we used before, 0.01. And then if we come down here, which is the same as 10 to the power of negative 2. So if we go 10 to the power of negative 2, but we can also see that if we go jump up one little notch here, because this is a log scale, remember, that the loss is still fairly sharply decreasing here. So that would be a value of 0.02. So how about we try build a model with that learning rate and see if it can achieve similar results to Model 8 that we use this learning rate with in less epochs. Because remember, what is the learning rate code for? It codes for how fast our model should try and update its patterns. The higher the value, the more our model is going to update its internal patterns every epoch. So let's see it in action. Let's try using a higher ideal learning rate with the same model as before. So we're going to set random seed, tf.random.set_seed 42. Beautiful. And then we're going to create the model. We're up to model 10 now. How good is that? Double digits. Sequential. Wonderful. Now you could probably just jump ahead here if you really wanted to. We're going to have 2 layers that are dense. With 4 hidden units and a nonlinear activation, ReLU. And then we'll do the same for the other layer. tf.keras.layers.Dense, and it's going to have 4. Activation is going to equal ReLU, or ReLU, depending on how you want to pronounce it. And then the final layer, just the same as Model A, Dense, activation equals Sigmoid. And now what do we have to do next? This is where we're going to compile the model with the ideal learning rate. So the value we've just picked off the curve, so this one here, 0.02. And then we'll go and put this here, learning rate we used before, model 8. I'm gonna come here. I'm gonna go model10.compile. Loss is going to be— what loss are we using? Binary cross entropy. And then we're going to go optimizer equals tf.keras— we have to use the, the class version of Adam here. lr equals— what are we going to set it to? 0.02. Beautiful. Because we said we're picking this learning rate value here. Let's come back down. Metrics equals— we'll just keep everything else the same except we're increasing our learning rate by 0.01. And now let's go fit the model for 20 epochs, because Model 8 we fit for 25 epochs, so 5 less than before. Let's see what kind of results it achieves with less epochs. Model 10.fit. Actually, I might save this as history 10. That's probably better. history 10.fit, X_train, Y_train, and epochs equals 20. Let's see what happens. Come down. How did it go? All right, loss equals 0.878 and accuracy is 98.24. So what was our Model 8 results? Let's go, let's go back. So let's save this 0.9824. And if we come back, so 0.9824, you remember that too. Where's Model 8? We've gone too far. Model 8. There we go. Oh, 25. And it's 9,749. Ah, so 9824. Model 10 has performed slightly better than Model 8 on less epochs, so less chances to look at the data, all because we increase the learning rate slightly. So it gave our Model 10 a chance to learn patterns in the data faster because they were updating with larger steps. Now, of course, As with every hyperparameter with our deep learning models, that might not always happen. But that's just another example of how powerful tuning the learning rate for your models can be. So how about we evaluate it? Evaluate model10 on the test dataset. Actually, we could have just done this to begin with. model10.evaluate, um, X_test, Y_test. Wonder how they perform differently on the test data. 99. Beautiful. And then we go evaluate model 8 on the test data. How did this perform? model8.evaluate . Hmm. So our model 10 gets a lower loss value on the test dataset than our Model 8, but Model 8 gets a higher accuracy value. Hmm, now that depends on which one of these metrics you want to optimize for. So again, remember, the metrics you get from the training dataset aren't always important as the metrics you get from the testing dataset. So this is something you'd want to investigate further depending on what your needs are. And depending, so potentially our Model 10 just because it's learned faster doesn't mean that its eventual performance on a test dataset or unseen data will turn out to be better. So this is where it takes a little bit of trial and error to figure out which model is ideal for your use case. Now, how about we finalize this Model 10 with the ideal learning rate and just see how the predictions look. So I'm going to go plot the decision boundaries for the training and test sets. So plt.figure, figsize equals 12, 6. Wonderful. And then we'll do— we'll get a subplot going. That's 1, 2, 1. Then we'll do the training data first. We'll set up the title there, then we'll bring in our trusty function plotdecisionboundary. And that takes in model10. It'll also take in x_train first. We need an underscore there. And then it'll also take in y_train. And then we want to subplot again, 1, 2, 2. So again, this is row 1, column 2, or 2 columns. And this is section 1, so the first one. And this is one row, two columns, and the second section. So we'll see what that looks like in a second. plt.title is going to be test, and we want to bring in our fancy function from above, model10, x equals x_test, and then y equals y_test. Beautiful. And then plt.show. Binary classification Wonderful. So again, our model 10, using an ideal learning rate that we picked off the loss curve, gets basically perfect predictions on the training dataset and the test dataset. And see, this is what I was talking about before with the subplot function. We want 1 row, 2 columns, and this first plot train is the first element here, and this test plot is the second element in the subplot. So, with that being said, we've explored a few ways to evaluate our classification models. We visualized them, but there are a few more classification evaluation methods that we should really be looking at. So, let's have a look at those in the upcoming videos. So far, we've seen a few visually rich ways to evaluate our classification models. Let's have a look at a few more evaluation methods that we can use. Now, these are some of the most common evaluation metrics that you should have in your machine learning and deep learning toolbox to evaluate your classification models. So let's have a look here. They've got a key: TP equals true positive, not toilet paper, TN equals true negative, FP equals false positive, and FN equals false negative. So let's have a look. We've got metric name accuracy. We've seen that one. There's the formula there. True positives plus true negatives, so all of the true predictions, on top of, divided by all of the other predictions. If you wanted to do this in code, the accuracy metric, you can do tf.keras.metrics accuracy, or if you want to use scikit-learn, it has an accuracy score function in there. Now, when should you use accuracy? Well, it's a default metric for many classification problems. However, it's not the best for imbalanced classes. So for example, if you had 10,000 examples of one class and only 10 examples of another class, and you got a classifier to score 99.999% or something like that, it could just predict that everything is all one class and get that sort of result. So in that case, you probably want to look into using metrics like precision. Now there's a formula for precision here, true positives over the total positive predictions, including false positives there. If you wanted to use it in code, you can use precision. Higher precision leads to less false positives. Let's go to Google and go, what is a false positive? Oh, coronavirus testing, what is a false positive? That's a topic of the moment. What is a false positive? There we go. A false positive is when someone who does not have coronavirus tests positive for it. So you can see where a false positive may have altercations. Oh, it's not a good thing, because if someone tested positive for coronavirus, that could go on to have a whole bunch of adverse effects that you didn't want. So if you're training a machine learning model, deep learning model as well, and you wanted it to predict less false positives, in other words, predicting if someone had coronavirus when they actually didn't, you probably want to optimize for the precision metric. Now, there's also recall, which is true positives over true positives plus false negatives. If you wanted to use a code, you could use one of these two functions here. Now, higher recall leads to less false negatives. So if a false positive is someone being predicted as having coronavirus when they don't actually have it, What do you think a false negative is? And what would, what would be the implications there? So the false negative in this case would be if someone was to predict, we did a coronavirus test and I actually had coronavirus, but my test came back and said that I didn't have it. So you could imagine, again, that would have consequences as well. If I had a test that said that I didn't have coronavirus and I go about my life and do whatever I want to do, and then I start giving it to other people, well then that's not an ideal scenario, is it? So for problems where false negatives are not good for your use case, you want to train your deep learning models to have higher recall. However, you might be thinking, why don't we just increase precision and recall? There's often a trade-off between the two. I'll show you here. What is the precision recall trade-off. Precision recall trade-off. Beautiful. Let's understand precision recall. Where is it? Have we got a trade-off curve? There we go. This is probably it. Unfortunately, you can't have both precision and recall high. If you increase precision, it will reduce recall and vice versa. This is called the precision-recall trade-off. So keep that in mind, that in an ideal case, your model would have high precision and high recall, but usually when you try to improve one versus the other, the other one goes down. So say we wanted high precision, you would have lower recall, and the inverse of that is also true. Another option is to try and improve the F1 score, which is like a combination between precision precision and recall. So the F1 score is one of the scores that I like. It's usually a good overall metric for your classification models. But again, me just reading these evaluation metrics to you out loud probably doesn't make as much sense as to when you start to code them up and start exploring them yourself. So just keep that in mind. We're just talking about these here. We're just naming these different metrics. It's not until you get to start to use them in practice that you'll really start to understand when to use which. And then finally, another great one is a confusion matrix. So this is particularly helpful when you're dealing with all the way from binary to multiclass classification. You can create your own custom function, or scikit-learn has a built-in confusion matrix function. So when to use? So when comparing predictions to truth labels to see where the model gets most confused. It can be hard though to use with very large numbers of classes, as we'll see in a future project. So keep these in mind. Again, this is just a slide. In the upcoming video, we're going to have a practice implementing a confusion matrix. But for these ones here, the accuracy, we've already, we've already done this during our model training. So for precision, recall, and F1 score, your homework for this video is to dive into the TensorFlow or scikit-learn documentation for each of these three, and then I'll see you in the next video we'll start applying some of these metrics or evaluation methods to the problems we've been working on. In the last video, we introduced some more classification evaluation methods. Now again, these are some of the most common and the ones you definitely want to keep in your toolbox. So make sure you, you screenshot this or take a note down of these metrics because they're probably going to be the ones you most commonly see with classification deep learning models. We didn't write much code last video, so let's make up for that. We'll go back to our notebook here. And all right, another heading here: More classification evaluation methods. And so alongside visualizing our model's results as much as possible, there are a handful of other classification evaluation methods and metrics you should be familiar with. And because this is not a markdown cell, it's going to keep going. That is all right. We'll just put a little few dot points here. We'll just replicate that slide that we had. We had accuracy, which is probably the most common, and then precision. A higher precision leads to less false positives. And then we had recall A higher recall leads to less false negatives. However, there is the precision-recall trade-off, which is an important concept to be aware of. There's the F1 score, which is a combination of precision and recall and usually a good overall classification metric. And then of course there's the confusion matrix, which is another visual way of looking at things. And then finally, this is not TensorFlow specific, but it is also another way that you can see everything. Classification Report from scikit-learn. Is that on the slide before? I don't think so. We can put this one here. I go here, Classification Report scikit-learn. The good news is a lot of these classification metrics all have a very similar principle. They take in some true values and they compare them to our model's predictions. That is the crust of, of whenever we're evaluating a model. What were the true values the model should have predicted? And what were the values that it did predict? Let's compare the two. So I'll just put this little link in here. For the rest, the previous slide that we looked at shows the code examples that you can use. So let's start off with accuracy. I mean, our model has already used it because we passed in accuracy up here as the metrics. Let's write some code to make that look a little bit better. So we can go here. We've got check the accuracy of our model, and we've got loss accuracy equals model10.evaluate, and we're going to evaluate it on the test dataset. And then we're going to print— we'll make this a little bit prettier. So model has or just model_loss on the test set. That sounds a bit better, doesn't it? And then we're going to do— because it's an f-string, we can pass that. Beautiful. And then model_accuracy on the test set. Wonderful. And then we can just— because we're going to set accuracy up here, let's times that by 100, and then We'll just put— we'll shorten it to 2 decimal places. Will this work? I hope so. There we go. Okay, so see what we did there was we just evaluated it on the test dataset, and then we found out the loss because that's what our model is going to return with the evaluate function. So the loss is there, and then the model accuracy, it comes out in this dot point notation. We've just adjusted it here to be more visually appealing. So what should we work on next? How about a confusion matrix? So what we do is we'll end this video here, and we'll come back to the next video, and we'll see how we can make a confusion matrix with our model. Last video, we left off with the question, how about a confusion matrix? We checked out the accuracy Again, if you want a little bit of homework, you can check out our model's precision, recall, and F1 score. Before we create a confusion matrix, let's see what the anatomy of a confusion matrix is. So this is what we're going to be working towards creating. So it's a matrix, so it's got rows and columns. And on the y-axis of a confusion matrix are usually the truth labels. So this is what ideally our model did predict, but on the x-axis is what our model actually did predict. So inherently, what happens when you lay it out like this is that on the diagonal are the correct predictions. In other words, the true positives and the true negatives. So you might be wondering, well, what's a positive and what's a negative? Well, in the case of binary classification, a true positive is when the model was supposed to predict 1 when the truth is 1. And a true negative is that the model predicts 0 when the truth is 0. So knowing this, you might have guessed that the ones outside of this diagonal are the false positives and the false negatives. In other words, a false positive is when a model predicts 1 when the truth is 0, and a false negative is a model predicts 0 when the truth is 1. So again, this is just seeing the name of things. Let's code this up and see what it looks like. Let's go back to the previous slide and see where the code is at. Confusion matrix, what is the code? Custom function or sklearn.metrics.confusion_matrix. All right, let's not reinvent the wheel to begin with. Let's bring in scikit-learn's confusion matrix. So scikit-learn confusion matrix, metrics.confusion_matrix. Beautiful. Okay. Is there an example? All right, so example, from sklearn.metrics import confusion_matrix. We need true labels, we need some predictions, and then we just pass it to the function. Wow. Okay, let's try it out. So create a confusion matrix. So from sklearn.metrics import confusion_matrix. Wonderful. Then we need to make some predictions. How do we do that with our trained model? So we'll save them to y_pred equals model10 dot— how do we make predictions with a trained model? If you guessed predict, you'd be correct. So we'll predict on the test dataset, and then we can create our confusion matrix. How good is that? I love scikit-learn. I love TensorFlow. Some beautiful, beautiful code libraries out there. So that's all it is. We've got the y_test, which are our test labels, and our model's predictions. Let's see what the confusion matrix looks like. Oh no, what's happening? No module named sklearn. Oh, typo. Standard sklearn.metrics, import confusion_matrix. This should work. Oh no. Classification metrics can't handle a mix of binary and continuous targets. Hmm, what is happening here? Well, what do we do when we face value errors like this? We have to inspect what we're trying to predict on. So what does y_test look like? Maybe we view the first 10 values. Okay, so that's what y_test looks like. What do our preds look like? They look the same. Ah, I see where the trouble is. Look, this is why we're getting a value error. Classification metrics can't handle a mix of binary and continuous targets. So our test values are in binary form, so it's 0 or 1, whereas our predictions are in continuous form, so they're not 0 or 1. So what should we do here? What do we have to do to compare our test array and predictions array? Well, we're going to have to convert our predictions array into zeros and ones. But what even are these values here? Well, these are called— let's write this down. Oops. Looks like our predictions array has come out in prediction probability form. So this is the output. We'll make that in bold. So the standard output from the sigmoid or softmax, as we'll see later on, activation functions. So what we're going to have to do is convert them so they're in prediction probability. So this is a value that the model has output. So the closer the value is to 1, the more the model thinks that it's a 1 label. And the closer the value is to 0, the more the model thinks it's a 0 value. So out of the knowledge that we've learned with TensorFlow so far, is there a function that we can use to potentially round these values there was a little hint there, to their closest integer value. So for example, if one of these was— let's see if this actually outputs 9.852. I'm not going to type out the whole thing because that's going to take way too long. What does this actually look like? There we go. So is that closer to 0 or 1? Now you might be wondering, of course it's closer to 1, so this one would go to 1, right? And then the same would be for this one, and then the same would be for this one. And then we might have to go on for a while before we find a zero. You might be wondering, what's the cutoff? Well, the cutoff is 0.5, so anything higher than 0.5 will go to 1, and anything lower than 0.5 will go to 0. However, this is a value again you can tune, but for simplicity's sake, We're not going to use that for now. We're going to just convert our prediction probabilities to binary format and view the first 10. Now again, I want you to try and figure out this yourself. How could we round these to 0 or 1 using TensorFlow? If you guessed we use tf.round, or perhaps you already knew that, you're like, Daniel, come on, I'm about 50 videos into using TensorFlow, I know this stuff already. We go boom! What does this look like? Ah, that's what we want. That's looking more like what our test labels are. So remember, whenever we're comparing things, again, one of the biggest issues that you'll run into is your tensors or your data types being of the wrong format. So all it is is about thinking about how you can get them into the right format. So that's what we want. So this is what you'll often have to do with a classification problem. The outputs of your model will come in prediction probability form, and you'll have to convert them to human-readable form. In other words, integer form. Now, let's see what our confusion matrix looks like. So let's create confusion matrix, confusion_matrix, y_test, and then we want to go tf.round. And we're off. Hmm, okay. So we got down the diagonal. If we go back to the anatomy of our confusion matrix, let's see. All right, so we've got the same values here. But this isn't as pretty. This is just an array. What we might do in the next video is prettify our confusion matrix to look something like this. We know what the y-axis is. We know what each row is. We know what each column is. We know what the x-axis is. So we can look at this one, but it's not the ideal type of confusion matrix. I mean, if you imagine you were trying to share that with a colleague, what is even going on here? So next video, we'll see how we can prettify our confusion matrix. So in the last video, we made our first confusion matrix. However, we said that it's nowhere near as pretty as the confusion matrix we have here. So how about we write this down? How about we prettify our confusion matrix? Because there needs to be more beauty in the world. And excuse me, I'm getting a little bit poetic here. But code can be poetic too. So the function we're going to use, or the code we're going to write, is— I'm gonna put a little note down here. So the confusion, confusion matrix code. We're about to write is a remix of scikit-learn's plot_confusion_matrix function. Now we've got this up here, so if you look into this, we can click on the source code and we could follow through with that fairly extensive source code. Scikit-learn is beautifully documented, and you can go through that I tried out this plot_confusion_matrix function and I found out it only works with estimators. So a scikit-learn model is referred to as an estimator, whereas we want to use TensorFlow. So we want to adapt it to our TensorFlow code. So you'll often run into this, right? You'll often run into— it's like where you want some sort of functionality, but it exists somewhere else, but to get it working for your use case, you kind of have to tailor it. So that's what we're going to do here. So just follow along, and we're going to make a pretty confusion matrix with this here, and with our TensorFlow model. So let's set up a fig size, just so we can use this again. Now, because we're writing, we're going to be writing a fairly extensive amount of code here. What's our principle? Whenever we're writing something, if you get stuck or you're not sure what's going on, or my explanation isn't as great as it could be, always pause and rewrite it yourself and see what happens. That's what I do whenever I don't understand something. So we want to go— we can just create our confusion matrix just like we've done up here. So nothing new so far. YPreds, beautiful. You might have noticed I've imported itertools. We'll see where that comes into play in a second. We're also going to create a little normalized feature because if we come back here This is how we're going to get percentages here, right? So this label, 98% of them are correct and 2% are incorrect. This confusion matrix works out well because we have 100 examples. Might be different depending on the amount of samples you have for each label, but let's not get distracted, Daniel, let's code. So we're going to go cm as type float now, because remember, our confusion matrix so far is just an array. A very non-pretty array, that is. We can just divide it by the sum to normalize it. Axis equals 1, and then we're going to go here, np.newaxis. So this is going to be— this will normalize our confusion matrix. Again, if we wanted to see what that looks like, we could go cmnorm. Wonderful. There we go. And now let's go in here. We want to set up number of classes. So we can do this by getting the shape of our confusion matrix. Again, to see what something does, cm.shape. This will be helpful for if we had multiple classes. So right now we're only dealing with binary classification, but what if we had 10 classes? We may see that later on, spoiler alert. So let's prettify it. I've got my dyslexia kicked in there, got my F's and T's mixed up. So we're going to create a figure and an axe using plt.subplots, and we're going to set our figsize to equal figsize. We hardcoded this up above here. You don't have to do that, but I just decided to. And then we're going to create a matrix plot. So cax equals ax matshow. So this is a matrix plot. We'll go here, matplotlib matshow. What does this do? Display an array as a matrix in a new figure window. Beautiful. Again, we're remixing some code from scikit-learn here. So matshow, what do we want to pass it? We want to pass it our confusion matrix, right? So CM, and then we'll set the colormap to equal plt.cm blues. A lot of CMs here. Don't confuse our confusion matrix CM with plt.colorbar blues. Wonderful. And then we'll go fig.colorbar, cax. Wonderful. And finally, we can create classes. We need to set up a bool. So if we do have multiclass, we want it to do something. And if we do only have binary class, which is what we're working with now, we want it to do something else. So that's why we're creating a conditional. If classes, labels equal classes, else labels equal np.arange . Wonderful. So if we have a list of classes, if it exists, we'll set the labels to equal classes. But if it doesn't, which is our current scenario, We'll set the labels to be just a range of our confusion matrix shape on the 0th axis, which is just 2. So it'll be 0 to 1. Wonderful. And now let's label the axes. ax.set, because we're going to be writing a bunch of labels here. We're just going to use a shortcut, ax.set from Matplotlib. Confusion matrix, that's a title. And remember, We're prettyfying it so it looks like this. That's our title, this is our x label, this is our y label. So these are the ones that we need. X is the predicted label, and then y label equals true label. Wonderful. We're also going to set up the x ticks to be np.arange, and then the y ticks is going to be the same thing, np.arange. That's the number of little dashes that we want to have on our figure. We'll see what that looks like in a second. We can also set the xticklabels to be our labels variable, which in the case of binary classification, it's just going to be the range. If we don't have a list of class names, the labels are just going to be the class integer values. And same thing for the yticklabels equals labels. We need to set the threshold for different colors. This will make sense in a second. So the threshold equals cm.max, so the max value in our confusion matrix, plus the minimum, and then we want to divide that by 2. So that's the threshold. So this is going to give our confusion matrix different shades of squares depending on how many values are in there. So a typical confusion matrix is you want the diagonal axis to be really dark where the correct predictions are, and all of the other ones where there's not many prediction values to be light. This will make a lot more sense once we visualize our pretty confusion matrix. So plot— we're going to plot some text on each cell. So we can go for i, j in itertools So itertools is going to iterate through whatever we pass it. So we want range , and same thing for range . I'm going to go boom, plt. Now this is— we're going to set the text for each square. So j, i, this is giving ourselves coordinates, and then we can set up an f-string for our confusion matrix. For i, j coordinates, i, j index, that is. We want that. And then it's also going to be, and we want in brackets, cmnorm, again, the same locale index times 100. We want one decimal point here, F, and then squiggly bracket, fix up the F string. And then percentage. Oh, our bracket's correct here. Fingers crossed. There's going to be an error here somewhere, isn't there? Let's traverse back through. Where have we got an extra? Because we want this to be the same color. Okay, wonderful. Yep, that should be like that. Oh, our f-string has ended too early. That's what we want. Wonderful. And now we can have a little comma after that. We want horizontal alignment equals center, and then we want to go color equals white. This is going to be the color of the text if our confusion matrix at a particular index is greater than the threshold, else We want the text to be black. Wonderful. And the size can be 15. There's a fair bit going on here. But again, if you're not sure what's happening in any of these lines, remember, break it apart. We've got a lot going on here. This took me a while to remix this function here. So I didn't just pull this out of the hat. Let's see what it looks like anyway. See where our errors appear. Wonderful. Text has no property horizontal alignment. Oh, I have a no n. There we go. Woo! So we've basically just replicated our pretty confusion matrix. See what I meant with the color threshold here? We want our squares with lots of values in them to be darker, and the squares with not many values in them to be lighter. And the more values— you can't really see it with this confusion matrix, but we will see it later on. The higher the values, the darker the shade of the square will get. Maybe if we go confusion matrix images, there we go. This is a good example. So this is without normalization. So we see here, all these squares are light, and all these squares are darker because they have more values. So with that being said, we probably could increase the text size here. So that's a bit more visual. How about we do that? What do we want to do? We want to set the x-axis labels to the bottom. That's what we'll get that to the bottom. See those zeros and ones? We want to get those down there. So we can go ax.xlabel position. And then we're going to just type in bottom. Then we go x-axis.tick. We want to have the ticks down there as well, correct? And then we want to adjust the label size. We can do that by going ax.yaxis.label.setSize. And then we're going to ax xAxis .setLabel. And then we can do the same for the title. ax.title.setSize. Now of course, beautiful, that's looking way better. Have a go at this. We've just prettified our basic confusion matrix. If we come back up here, albeit it took a fair bit of code, we have this array here, and now it looks like this. Now, of course, we could functionize this. Maybe that's a little task for you. Could functionize this to work with any y_test values and y_pred values. So maybe we look at that, doing that later on, or maybe you could try that now. But as you could see, this is a really great and visual way that we could quickly show someone how our model's performing and see how it does on different classes where it gets confused. Now, our model's doing pretty well right now on our data because, as you can see, the diagonal is very dark. But going forward, it'll make more sense once we work with a multi-class classification problem and make a bigger confusion matrix to see where our model messes up when comparing, say, if we had 10 different classes. So that's where we'll finish this video. We'll get rid of these two cells here. And then in the next video, we're going to start tackling a larger example. More specifically, multiclass classification. So go back through, see if you can turn this into a function. Maybe you want to turn it into a version of plot_confusion_matrix. So you could do def plot_confusion_matrix, blah, blah, blah, and see if you can pass it something like true labels and predicted labels and have it come out with something like this. So give that a shot and I'll see you in the next video. Alrighty, welcome back. I'm super excited for this series of videos because now we're going to start— let me write this down actually— working with a larger example. In other words, or more specifically, multi-class classification. So we'll just make this little heading here. Actually, it's probably worth making it a size 1 heading. So we'll turn that into markdown. Oh, we've spelled multiclass classification wrong. That's all right. Now, what we're going to do in this series of videos is, so far we've created our own dataset. Now, we created a blue and red circle dataset, which is a fairly simple dataset. And then we went through a whole bunch of different steps to model it. We learned how we can improve our models. We learned about non-linearity, which is important if we want to model data that is non-linear, so has different shapes. We learned how we could evaluate and improve our classification models. We learned how to plot the loss curves, find the best learning rate, and a whole bunch more classification evaluation methods. So now to really drive all of these concepts home, let's start with a new problem, a multi-class classification problem. So Let's say you were a fashion company and you wanted to build a neural network to predict whether a piece of clothing was a shoe, a shirt, or a jacket. So in that case, you have 3 different options. So let's write that down. When you have more than 2 classes as an option, it's known as multi-class classification. So 2 classes on their own is binary classification, but more than 2 is multiclass classification. So that means— this means if you have 3 different classes, it's multiclass classification. It also means if you have 100 different classes, It's multi-class classification. Now, the good news is, with a few tweaks, everything we've worked on so far in our binary classification problem, we can apply to a multi-class classification problem. We could just look at here to see the steps that we've gone through, or we could just jump back into our presentation and remind ourselves of the steps in modeling with TensorFlow. I love that little animation. Look at this colorful little picture. So we need step 1 is get data ready, turn it into tensors. Okay. Step 2 is build or pick a model. We've built a lot of models so far, so we are pretty familiar with this, yeah. Fit the model to the data and make a prediction. Okay. Evaluate the model, improve through experimentation, and save and reload your trained model. Wonderful. Let's start with step 1. Let's get the data ready. So I kind of hinted at what kind of dataset we're going to be using before. We're going to pretend that we're a fashion company and we want to build a neural network to classify different images of clothing. So to practice multiclass classification, we're going to build a neural network to classify images of different items of clothing. Now, the beautiful thing about this is that we can use the Fashion MNIST dataset, which is built in to the tensorflow.keras.datasets module. So let's have a look. If we go tensorflow fashion mnist, beautiful. Fashion MNIST, TensorFlow datasets. So the TensorFlow datasets module is up here. It has a whole bunch of different built-in datasets that you can use to practice on your own problems. So they're great to use to sort of get familiar with how a neural network you can build with TensorFlow and get it working before you adjust it to your own dataset. That's a really important concept. A lot of the time in machine learning and deep learning, you'll work on problems that have existing outlines and then slowly adjust them to whatever you're working on. So we've got a description here: Fashion MNIST is a dataset of Zalando's article images consisting of a training set of 60,000 examples. So it's going to be our biggest dataset yet, and a test set of 10,000 examples. Each example is a 28 by 28 grayscale image associated with a label from 10 classes. You could go through that documentation there, or you could just follow along with the code here. To get started with the TensorFlow datasets, we'll re-import TensorFlow as tf, and we'll also get the tensorflow.keras.datasets module. We're going to import— oh, that needs to be datasets— we're going to import FashionMNIST. Now, the good thing about the TensorFlow datasets is that all of the datasets, or basically as many as you can, or at least the ones I've worked with so far, which is a fair few, have already been split into training and test sets. So we can go, the data has already been sorted into training and test sets for us. So to import it, we can use tuples. So it's going to be train_data, train_labels, and then another tuple for the test_data and test_labels. So we'll set that up as equals fashion_mnist, which is this module we just imported here, and then we'll use the load_data method. So of course, if you wanted to see this in the documentation, you could come back here, and the split says that it's already in test and train, 10,000 in test, 60,000 in train, and then there should be an example somewhere of how to import it. I guess not. That's all right, it's somewhere here, maybe in the overview section. Let's get back to focusing right in code. So we Shift+Enter that, and it'll download. It'll be pretty quick because it's a relatively small dataset and it's stored on Google Storage. Now to check out an example, we can go show the first training example. Okay, printf training_sample, and I'm going to put it on a new line just so it's nice and neat. We'll go train_data, and then we'll just get the 0th index from that, and then I'll get a new line after that. We need to finish that f-string. There we go. And we're going to do the same thing for the training_label. We'll just do the same index here. Of course, we could have done this randomly, but I'm going to just get one out of there because that's one of our first steps, right? Whenever we're downloading data, we want to become one with the data. We want to visualize, visualize, visualize. Okay, so what have we got here? Training sample. We've got some sort of array of numbers. Beautiful. And the numbers are varying. From— it looks like from 0 up to about 255. Okay. And then the training label is a 9. All right. So this array of numbers or this matrix or this tensor of numbers represents training label number 9. So if we go here, we look up Zalando Research's Fashion MNIST. These are the images that we're going to be working with. So there's 10 different classes of image. We've got shoes, we've got dresses, we've got shirts. Come down, we can get the data from there. Okay, here's the labels. So labels— each training and test example is assigned to one of the following labels. So number 9, this is an ankle boot. Wonderful. So this first sample is an ankle boot. Now of course we could keep looking at numbers What else do we want to investigate whenever we're working with a new dataset? We want to look at the shape, we also want to look at what it looks like. So let's do that. Let's go check the shape of a single example. So let's go train_data, we need to get the first sample,.shape, and let's go train_label, labels . .shape. Wonderful. So we've got a 28 by 28 tensor here, and the train labels is just a scalar, so it has no shape there. Now how about we get visual, plot a single sample. You come in here, so to plot it we can import matplotlib, .pyplot as plt. I'm gonna go plt because we're working with an image. imshow, and then we can just pass it a single example. I'm gonna go 0, and because its training label is 9, what should it look like? It should look like an ankle boot. Let's see if this works. Oh, okay, so fairly pixelated here, but you can kind of see the outline of a boot there. And how about we try another one? Wonderful. So that looks like a sweater. And then check our samples label. So if we go train.labels and pass it the same index as here, what label does this get? All right, it gets a 2. And if we come back here, oh, it's a pullover. Okay. So I said that this was a sweater, kind of looks like a sweater to me, but again, depending on what dataset you're working with, it will have different labels. So we've gotten pretty familiar with our data. We're probably going to have to set up a little list to index on our training labels. So let's start doing that in the next video, and then we'll slowly keep getting more and more familiar with the data before we start to build a multi-class classification neural network. Let's keep going, getting familiar with our data that we're going to be working with for the multi-class classification problem. So far it looks like that our labels are in numerical form. And while this is going to be fine for our neural network, we probably want them in human-readable form. So what we're going to do is create a small list of the class names that we found on the dataset's GitHub page. And that way we can index onto that list so that instead of just having a train label as 2, this actually reads as pullover. So right down here, create a small list so we can index onto our training labels so they're human readable. Again, this is all part of becoming one with the data that we're working with, understanding what kind of problem that we're working on. So we go here, all I'm going to do is just copy these in the same order. So it'll be t-shirt, top, trouser, pullover, dress. So let's just write that down. Might speed this little section up. Wonderful. Now we've got a list of class names that reflects the actual human-readable name, and we've got their labels in integer form, which will be great for our neural network. So let's have a look. How many classes are we dealing with? 10. So again, anything over 2 is classified as multi-class classification. So now we've got our class label names. Let's plot another example. So we go here, plot an example image and its label, and go plt.imshow. And which number or which index should we pick this time? How about we go 17? And then we're going to set the color map to be plt.cm for color map to be binary because this is going to be grayscale. We'll set the title plt.title. Now this is where we can index onto our class names list, class names list with train label and the same index here. Or actually, we can probably just create an index of choice variable. Boom. So that we can just update that index of choice. Again, you could probably just set this up to be just a random number generator if we really wanted to. But this is how we're going to plot different examples. Wonderful. So there's a t-shirt/top. We can do it again. 20, there's a dress, 10, t-shirt top, 100, 2,000, there's a bag, and there's a coat. Wonderful. Now again, we could keep going through these and visualize different examples so that we know the data that we're working with. But what's next? How about we set up, rather than just running this cell dozens of times, we set up just some code to plot a bunch of random examples so that we can get familiar with them. Now, with a dataset like this, you could probably just look at what it looks like here and start to understand, okay, this is pretty straightforward what it is. It's 10 different types of pictures and they're all grayscale. But if you had a larger dataset with 100 different categories and your images weren't grayscale, they had a whole bunch of other details in them, you probably want to have a look at multiple or hundreds, if not thousands of samples before you start building a model. And when I say that number, again, that's an arbitrary number. It's just the number of samples that you start to feel familiar personally with the data that you're working with. That's the most important part. So let's write some code to plot multiple random images of Fashion MNIST. Import random, plt.figure. And right now we're working with an image dataset, but the same would go for any type of multi-class classification dataset that you're working with is to visualize, visualize, visualize as many samples as you can. So we'll probably loop through 4 samples at a time. We'll set up an axe with plt.plot. Actually, we need a subplot, don't we, because we want to plot multiple different things. We'll give it 2 rows, 2 columns, and the index can be i 1. Because we're looping through. We'll select a random index using random.choice, and we'll set up a range for the length of how many training samples we have, which is 60,000. So all this is going to do is just pick a random number in the length of our training data. So it'll be from 0 to 59,999, and then plt.imshow We're going to show the training data, um, sample that appears at the random index. And the cmap can equal plt.cm.binary, so it comes out in grayscale. And then the title— what do we want the title to be? We want class names. And then again, we'll index on our training labels where the random index occurs. And we don't want any of the ticks because we don't need them. We just want to see the images. So hopefully this works. We should be able to run this cell and visualize. Okay, shirt, bag, sneaker, trouser. Wonderful. T-shirt, shirt, dress, bag. So you can imagine if we went through this, this is a bit easier than what we just did here. So this is what I like to do whenever I'm working with a fairly large dataset is never underestimate the power of randomness. Because I like to look at samples randomly and just keep going through the dataset, start to build an image of what these look like in my own mind before I start to build a neural network that's going to distinguish patterns in them. Now I want you to have a, a careful look at the data that we're working with. What kind of shape does it take on? Are there many straight lines? Are there any curved lines? How does that relate to what we've learned with linearity and non-linearity? So do you think, with this type of data that we're working with, are we going to need a neural network that uses just linear, like straight lines, or will we need a neural network that's going to have some kind of non-linearity in it? So that's something to think about, but have a play around with this little section here. Become one with the data, run it about 25 more times so you've seen at least 100 different samples, and then next video, we come back to our modeling. We've got our data ready. Luckily, it's already in tensors for us because we've downloaded it from the Keras dataset module. However, oftentimes your data won't be ready, turned into tensors, so we'll tick this step off. Now we're up to building or picking a pre-trained model to suit our problem, so we'll get on to that in the next video. Okay, so we've started to become familiar with the multiclass data that we're working with. Now, let's build a model. We're building a multiclass classification model. Now I said earlier, well, we've already built a fair few binary classification models, and I did mention earlier that the multiclass classification model, we're only gonna have to tweak a few things to get it to work with our multiclass classification data. So if we go back to the keynote, we look at the typical architecture of a classification model. So we've got multiclass classification over here is what we're working on now. So the input layer shape, same as binary classification, depends on the number of features you have. Number of hidden layers, again, same as binary classification. Neurons per hidden layer, It's the same. We've got another difference here. It's the output layer shape. So for binary classification is 1, 1 class or the other. Multiclass classification has 1 per class. For the hidden activation, we can use the same. In binary classification, we use the nonlinear function ReLU, and I asked you in the last video to think about whether or not we might need nonlinear activation functions. The output activation, okay, that's different too. For multiclass, we're going to need the softmax function rather than sigmoid. We also need a different loss function. So for binary classification, we use binary cross entropy. But for multiclass classification, we use categorical cross entropy in TensorFlow. And then finally, the optimizer can stay the same, so we can use our trusty Adam. All right, let's write some of these down. We go here for our multi-class classification model. We can use a similar architecture to our binary classifiers. However, we're going to have to tweak a few things. Namely, first of all is the input shape. So what is our input shape? What, what shape is our data? This is what we explored before. So the train_data sample 0.shape. Okay, so our input shape is 28 by 28. So let's write that down. 28 by 28, the shape of one image. Wonderful. And we're also going to have to modify the output shape. What was the output shape? If we come back to our typical architecture of a classification model, the output layer shape is 1 per class. So how many classes did we have again? Lang class names. Beautiful. So the output layer shape is 10, 1 per class of clothing. That's what we want. And what else was there? Ah, the loss function. Loss function equals— has to be tf.keras.losses.CategoricalCrossEntropy instead of BinaryCrossEntropy. I'll put that there. And then I think that's all we'll have to do. Beautiful, because we can keep the other things the same. Ah, the output activation. That's what we need to write down. Output layer activation equals softmax, not sigmoid. All right, so with this being known, how about we put together our first multi-class classification neural network model? So we're going to go through the exact same steps that we've been through before. So set random seed, tf.random.set_seed, we'll give it 42. Wonderful. Let's just try replicate the exact same model we've been using in previous videos. So we'll go create the model. I think we're up to model 11 now. tf.keras.Sequential. Wonderful. And then what did we have? We had 2 dense hidden layers. tf.keras.layers.Dense with 4 hidden units and a ReLU activation. So we'll just make two of those. tf.keras.layers.Dense and a ReLU activation. Wonderful. Nonlinear, because this is the answer to the question I posed before. If you guess that we're going to need nonlinear activation functions for our data, you'd be correct, because it is composed of straight lines and non-straight lines. So that's why we need nonlinear activation functions. For our output layer, oh, this is going to be different, isn't it? We can't use the exact same one. How many output shapes do we need? We need 10. Wonderful, we can do that, 10. And we're going to change our activation from sigmoid to softmax. So we can do it like this, softmax, or we can also do it like this, tf.keras.activations.softmax. Softmax. Now I'm not sure if it's a capital S. Let's look it up. TensorFlow softmax activation. There we go. So I believe we might just be able to put it like that. Let's see if autocomplete helps us out. softmax. There we go. Wonderful. Now what else do we have to do? We have to compile the model. As always, compile the model. We're up to model 11,.compile. Loss equals, what do we have to change here? We need to change it to categorical cross entropy. tf.keras.losses.CategoricalCrossEntropy. Wonderful. We can do that. And then we can go optimizer. We don't have to change that. We can do our trusty Adam optimizer. We'll leave all the default parameters there. And finally, let's set up metrics. We'll just set as accuracy, our trusty accuracy. Beautiful. So let's fit the model, and we're going to save this as non-norm history. And you might be asking, why is that? Well, I will reveal all in an upcoming video. Now we can fit directly on the train data and the train labels. And we might go just for 10 epochs, so 10 passes through all of the data. And we're going to introduce a new parameter here to the fit function, which is validation data. So this is where you can put in— we don't have a validation dataset, but we already do have a test dataset. So what, what we can do here is at the same time the model is fitting and trying to find patterns in the training data, so the relationships between the training data and the training labels, it can evaluate how well those patterns are on the validation data. But in our case, we don't have a dedicated validation set, so we'll just use our test data here. So this data will remain unseen, and that way we can evaluate how good our model's patterns are that it's learning in the training data, ah, when we use them on unseen data. So let's see how this goes. Oh no, what have we got wrong here? Value error, shapes. Ah, the classic shape error. Hmm, what have we got? Shapes 31, 32, and 32, 28, 10 are incompatible. Where are we getting these shape errors from? Ah, 4. So what's 28 plus 4? That's 32. You know what? I'm gonna have to introduce a new layer here. And this is a layer that you're gonna often need. So it's worth exploring it a little bit. We need to flatten our data. What does flatten mean? I'm gonna give it here input. What's our input shape of our data? Input shape is 28 by 28. So the input shape, we're gonna pass it here as 28 by 28. 28. This is telling our neural network that, hey, we're passing you some images that are 28 by 28. And you might be wondering, what does flatten do? Well, let's explore it. We'll get a new cell here. We'll go flatten_model equals— we're just going to create a model with a single layer. tf.keras.Sequential and tf.keras— of course, if you want to skip ahead and just look up the documentation rather than listen to me talk about about what the flatten layer does. You can 100% do that, but I like exploring things by writing code. 'Cause I'm not sure about you, but I tend to read documentation 3 different times and still don't understand it. When I write the code, I start to understand it. Let's check the shape of this. What is happening here? Ah, None, 784. Where did 784 come from? I'll give you a little hint. What's 28 times 28? 784. All right, well, now that we've seen the Flatten layer in action, now let's look it up in the documentation. So TensorFlow Flatten layer. Flatten layer flattens the input. Okay, does not affect the batch size. Do we have a demonstration here? Okay, so this is the original shape, 1 times 10 times 64. And if we use flatten on that, it's going to turn it into 1 times 10 is 10, 10 times 64 is 640. Ah, I see. So what it turns it into, instead of being a 28 by 28 array, it flattens it all so that it's now of shape None, 784. Because what you'll often find is a neural network likes everything to be in one long vector, and then we pass that through these other layers here. and we get to the outcome that we like. So our data needs to be flattened, and we'll go here from 28 by 28 to None, 784. So if you ever run into a shape error in your neural networks and you find that you haven't flattened your data into one long vector, could be because you're not using a Flatten layer as the very first layer of going into your neural network. Some layers can flatten your data automatically, but typically you'll need to tell your neural network that, hey, here's the data, here's my input shape, I want you to compress that, flatten that into one long vector, and then pass it through your other layers. So now that we've flattened our data, let's see if it gets rid of this value error, the shapes that are incompatible. And again, as I said, the shape error is one of the most common errors you'll come across. Oh no, what have we got here? Value error. Shapes 32, 1, and 10 are incompatible. Hmm, you know what? I think it's our loss function. You might be thinking, Daniel, that loss function is the exact same that we have in the multi-class classification, in the architecture of a classification model. What's going on? Well, there are two types of loss function. Now one is for if your data is one-hot encoded. So if we go, what's our training labels look like? Train labels, zero. So our training labels are in the form of integers. So maybe we'll have a look at the first 10. There we go. So 9, 0, 0, 3, 0, to 7255. Let's have a look at the documentation for categorical cross-entropy. I'll just copy and paste that in there. Computes a cross-entropy loss between the labels and predictions. Use this cross-entropy loss function when there are two or more label classes. We expect the labels to be provided in a one-hot representation. If you want to provide the labels as integers, please use sparse categorical cross-entropy loss. So that is where we're getting our value error from now. The shape error is coming because the loss function categorical cross-entropy— now I get confused with this all the time, but this loss function expects our labels in one-hot representation. But if we change it to sparse categorical cross-entropy, it should work. So let's try that. sparse_category_cross_entropy. Let's see what happens. Oh, yes! Look at that! Our neural network is running. Beautiful! So, two little tidbits to take away from this is that our binary class classification model that we've used before, this is very similar, this model 11, to all the other models we've been building throughout this entire section. Can work for multi-class classification data with a couple of tweaks, namely defining what the input shape of our data is, changing the output layer activation function as well as how many classes we're after, and updating the loss function to reflect the problem that we're working with and also the style that our labels are in. So I wonder if we can one-hot— tf.one_hot— tf.onehot, depth equals 10. What happens if we do that? There we go. Oh, I wonder if that would work. Categorical cross entropy, change that. And then we just tf.onehot, depth equals 10. And then we can do the same. We're going to have to do the same for the validation. This is our experimenting on the fly here. Depth equals 10. Man, we need an extra bracket here, don't we? Boom. Our neural network starts to run as well. Wonderful. So let's put a little note down here. If your labels are 1, hot encoded, use categorical cross entropy. And if your labels are integer form, use sparse categorical cross entropy. This one has tripped me up a whole bunch of different times. Just takes a little bit of practice. So again, typo there. If you get any shape errors with your models, the 3 things you have to look at: input shape, output shape, and the loss function that you're using. So they're the 3 main value errors or shape errors that you're going to come across. Of course, it could be more, but they're the 3 that I most often run into. So how exciting is that? We've already built our first multiclass classification model. Let's continue on with where we were in the next video. In the last video, you built your first multiclass classification model. So you should be very proud of yourself, give yourself a little pat on the back. But there was one thing we, we kind of forgot to talk about. We did code it, but we didn't really explain it, is this validation_data parameter. So you might have noticed a change in the output of our model training log here. If you see loss, accuracy, val_loss, val_accuracy, you might have sort of figured out what the val_loss and the val_accuracy is. If not, that's fine. The loss here with no prefix is the loss on the training data. So how wrong the model is trying to figure out the patterns between the training data and the training labels. The accuracy here with no prefix is the model's accuracy on the training data. But the val_loss and val_accuracy here is the model's loss on the validation data and the accuracy on the validation data. Now, this is important because this is the data that a model has never seen before. So it trains on the training data and then validates itself to see how good its patterns are on the validation data. So whenever you pass the validation data parameter with some kind of data, you're going to get these extra outputs here. And so this is a way to tell— remember, A model's results on the training dataset don't necessarily reflect how it's going to perform in the real world. You really want it to perform well on data it hasn't seen before to sort of get an idea of how it's going to perform in, say, your application or something like that. So right now, our model is getting a score, an accuracy score of about 35%. Now, that's better than guessing because we're working with 10 classes. So if we do 10 or 100, 100% accuracy divided by 10 for 10 different classes. If our model was just guessing, it would get about 10% accuracy. So okay, we're getting about 3.5 times that, but let's see if we can improve it. First, let's get a, a model summary. Check the model summary. I want to highlight something else before we move on to try and improve this accuracy, is the input and output shapes of our model, which is A very, very important point. Check the model summary, model11.summary. We go here. So we can see that the Flatten layer takes our 28 by 28 images, flattens them into a 784 long vector. It passes through these two Dense layers. So Dense layer 1 here, Dense layer 2 there, and then it gets output into a size of 10 for 10 different classes. So now, do you recall back when we were preprocessing data? We haven't done much preprocessing data with this problem because the data we got from the Keras datasets module, the Fashion MNIST dataset, is already numerical. However, we spoke about in a previous video the concept of scaling or normalization. If you're not sure, if you can't remember it, that's okay, we're going to go back through it. So we come back to our keynote. I deliberately left this arrow here so that you would think about, hmm, what's Daniel missing out on there? Well, number 1 is turn all data into numbers. Luckily, we've already done that. Number 2 is make sure all your tensors are the right shape. Well, we've already been through that too, so tick, tick. Number 3 is scale features. Hmm, what does this mean? Normalize or standardize. Neural networks tend to prefer normalization. So let's remind ourselves of what that is. Better yet, let's remind ourselves of our training data. Check the min and max values of the training data. train_data.min and train_data.max. Alrighty, so 0 and 255. Now If we said in this slide that neural networks prefer normalization, what is normalization? Well, it's also referred to as scaling. So let's write this down. Neural networks prefer data to be scaled, or which is also referred to as normalized, or depending on what circle you're from, this means They like to have the numbers in the tensors. They try to find patterns in between 0 and 1. However, right now our data is between 0 and 255. So how might we get our training data, and we're going to have to do the same for our validation data, between 0 and 1? Well, we can do that by dividing all of the data by the max number. So let's have a look. So write this down. We can get our training and testing data between 0 and 1 by dividing by the maximum. So this is referred to as scaling or normalization. Again, sometimes you'll find different names for different things, but if I use the word scaling or normalization, I'm referring to getting our dataset between 0 and 1. So let's go here, train_data equals train_data, and we'll divide by 255 as a float because that's the max value, 255. Then we do the same for the test_data equals test_data divided by 255. Or actually, what we might do to save the fact that we're not overriding our original variable, we'll do train_data_norm and test_data_norm. Wonderful. And then we can check the min and max values of the scaled training data. train_data_norm.min and train_data_norm.max. Boom. What do you think they'll be? 0 and 1. Beautiful. Now that our data is between 0 and 1, let's see what happens when we model it. So we're going to actually just use— we'll change nothing, absolutely nothing, from Model 11 except for the data that we're using. So the only thing that it will change is we'll use train_data_norm and test_data_norm, and everything else will stay the same. So you can go ahead and try to replicate that before I replicate that in the video. But I want you to think, now that our data has been normalized, it's one of the things that we can tune with our neural networks to make the performance better. What do you think will happen with a different data using the exact same model? It's all right if you're not sure. Let's find out. So now our data is normalized. Let's build a model to find patterns in it. So we'll come here, set random seed, tf.random. Set seed, and then we'll go create a model, same as model 11. We'll call this model 12, actually. A dozen models, how good is that? tf.keras.Sequential. I'm gonna know how to do this off by heart. And then we're gonna need a Flatten layer to get our data from 28 28, flatten, input shape equals 28, 28, telling our model, hey, I'm passing you images that are size 28 by 28. And then we'll create our two hidden layers with nonlinear activations, ReLU, tf.keras.layers.Dense, so that our model can find non-straight line patterns in our data. And then the output layer needs a shape of 10 because we have 10 different classes, and an activation of softmax because we're dealing with multi-class classification. And we can compile the model. We go model12.compile. We'll set the loss function. It's going to be tf.keras.losses .sparse_category_cross_entropy, because why? Because our labels are in integer form. I say integer like differently every single time I say it. But if our labels were one-hot encoded, we just want to get rid of sparse. The optimizer is going to be tf.keras.activations, our trusty friend Adam. Beautiful. And then the metrics is going to be accuracy, which is a great baseline metric for all classification problems. And then finally, we can fit the model. We want norm history. So we saved our model's training history to non-norm history. I said I'd come back to this variable, but this time we're going to use norm history. model12.fit on train_data norm, and then train_labels is just going to be the same thing. We use 10 epochs. Equals 10. Wonderful. And then the validation data is going to be test_data_norm and test_labels. Look at us building multi-class classification neural networks like a boss. All right, do you reckon this will work? What do you think is going to happen before we run it? We've normalized our data. That's the only thing we've changed. Have a think about it, and I'm gonna run the code in 3, 2, 1. Hopefully no errors. Oh, of course, activations. Activations, did I spell that wrong? Oh no, this has to be optimizers. Oops, honest mistake. Again, what have we got wrong here? Expected float32 passed to parameter y. 'Got type string.' Hmm. Test data norm, test labels. Oh, this is an error I haven't seen before. What are we missing out on here? You know what? I believe it's just a simple brackets. Wow. Yeah, so those are the type of errors you're gonna run into. Just simple little things like that. That was kind of a lucky guess, but you can spend hours sometimes troubleshooting them. So don't worry if you get stuck on something like that, because even I myself, who have built a fair few of these models, still run into simple syntax errors like that. But look what's happening here. 10 epochs, we've run this just as before. What has changed between our two models? The data, we've normalized it, that's it. The val accuracy has shot up from 0 point, what was it before? Come back up here, has shot up from 35% to 70, to 80%. So it was over doubled. So almost 2.5 times as good. Now all we did is we normalized our data. So that's something to keep in mind. So let me write this down as a key. So we'll get our key emoji out. I'll write here, note, Neural networks tend to prefer data in numerical form as well as scaled/normalized. So numbers between 0 and 1. Beautiful. So just by normalizing our data, we got a fairly dramatic increase in performance. How cool is that? So that's something I want you to keep in mind. So in the next video, what we'll probably do is we've saved our norm history, our model training history from norm history, as well as nomnorm history up here. So that's a bit of a tongue twister to say. So if you want to, you can have a go at plotting those loss curves between each other from the history variables and see how they look compared to each other. We've trained two neural networks using the exact same architecture, but except one has— or one was trained on non-normalized data. And then we said neural networks tend to prefer data in numerical form. That's definitely— needs to be numerical form, but they also prefer it to be scaled/normalized. In other words, the numbers need to be between 0 and 1. So let's compare the loss curves of each model. In other words, the loss curves of normalized data versus non-normalized data. So we'll grab Pandas, we'll import that, and then we'll plot the non-normalized data loss curves. So pd.DataFrame nonnorm history.history, and then we'll go.plot, give that a title of non normalized data. Beautiful. And then we can plot normalized data loss curves. pd.DataFrame, and we want norm_history.history.plot, title equals normalized data. Wonderful. Let's see what this looks like. Wow. So from these two plots, we can see how much quicker the model with normalized data improved versus the model with non-normalized data. So have a look at this. If we go here, our model with non-normalized data, the loss decreased and then it kind of flattened out there. But with normalized data, our model's loss dropped. It even started at a lower value than what this model finished with. and then it kept decreasing as it kept going. And the accuracy, who knows, maybe if we train this for longer, it will keep going. The same with this one actually, but it would probably take a fairly long time to find the same level of results that our normalized data found. Another key point to remember is when making comparisons of different models, so we'll put down here, oh, where's our key emoji? When comparing the results, so we'll write down note, the same model with even slightly different data. So this is the same dataset. All we've done is we've just turned it from non-normalized to normalized. So it's the same data that we're working with, but the same model with even slightly different data can produce dramatically— let's get really dramatic here. Dramatically different results. So when you're comparing models, it's important to make sure you're comparing them on the same criteria, e.g., same architecture. This is the comparison we're making here, but different data, or same data but different architecture. So that's something to keep in mind here when you're comparing, comparing the results of different models. Keep your comparisons to only comparing as small of variables as possible. What I mean by that is don't change like 10 things and then compare one model to the other. Change one thing and then compare the results from one to the other so you know what change is making the difference in performance. So with that in mind, when we're preprocessing data, let's come back here. When we're getting our data ready for our neural networks, we need to turn it all into numbers. We already started with that with Fashion MNIST. We have to make sure all of our tensors are in the right shape. We did that using the Flatten layer, so the input shape to our neural network, as well as the output shape of our neural network. And number 3, scale your features, so normalize or standardize them. Remember, neural networks tend to prefer normalization. With this being known, how about we try tweak another one of our neural network hyperparameters, let's see how we might find the ideal learning rate, just as we have before in a previous video, and see what happens to our neural network training. We've seen how even slightly changing the data we input to our neural network can produce dramatically different results. How about we try, let's go in here, finding the ideal learning rate and see if that changes anything. So just as we've done before in a previous video, we'll set the random seed. So the ideal learning rate is the learning rate value where the loss decreases the most. So to do that, we're going to set up the exact same model that we did before. Create model. We're up to model 13 now. tf.keras.Sequential. And we had a Flatten layer. tf.keras.layers.Flatten. Wonderful. And the input shape equals 28 by 28. Of course, that will change depending on the data you're working with. tf.keras.layers— we need a Dense layer. I'm spelling everything wrong today. Activation is going to be ReLU. Again, I'm not sure what's going on with the code editor. I want to keep our code nice and neat, right? So if someone else had to read this, They're not going, "Holy gosh, what did you even write here, Daniel?" And don't forget, when you're writing code, that someone else could be you, just later. layers.dense, we need a layer with 10 output units and an activation function of softmax because we're dealing with multi-class classification. We can compile model, model13.compile. Everything is going to stay the same as above. Loss function, we're using tf.keras.loss.sparse_categorical_cross_entropy. The optimizer is our trusty friend Adam, optimizers.Adam. Wonderful. And metrics, we're going to use accuracy. Boom. And now we're going to fit the model. Oh, what do we have to do first? I forgot a step. I'm too used to going create, compile, fit. We need to create the learning rate callback. So lr_schedule equals tf.keras— do you remember how to create a callback? We need callback and we need learning rate scheduler, and we're going to go lambda epoch. Let's start at— what can we start at? 1e -3. Let's start at, yeah, -3 for now because our model's already performing pretty well. So we'll start there. Epoch divided by 20. So that just means, remember, this just means start at this value here and slowly increase the learning rate every epoch by 10 to the power of epoch divided by 20. And to fit the learning rate, we can go fit RL history. We'll create another history variable. Model.13 fit We're going to use train_data, we'll fit it on the normalized data, and train_labels. We're going to go epochs, we'll go about 40 this time. And then the validation data, of course, is going to be test_data_norm and test_labels. And then finally, the callbacks is LLRScheduler. Wonderful. So this should work. Fingers crossed. There we go. Okay, so this is going to take about 3 seconds per epoch. So what's that times 40? 120 seconds. So I'll come back, I'll speed this up, and I'll come back once, uh, our model has finished finding the ideal learning rate. Alrighty, looks like our model has finished. Let's see what's happened here. Okay, so it looks like it ends up on a pretty Not as great validation accuracy as what it once was. We get a pretty good range of values here, but let's not look at just this training output. Let's— how did we do it before? We plotted the learning rate decay curve. So let's do that again. Plot the learning rate decay curve. So we want to import NumPy as np, import matplotlib, We've already got these in our notebook as before. However, if you were continuing on from this, we'll just re-import them just for completeness. LRs equals— what did we do before? 1e-3 times 10 to the power of np.arange, because we use 40 epochs, divided by 20. And actually, we don't even need to use NumPy. Let's just use TensorFlow, stick in the spirit of using TensorFlow. And we're going to plot. We want to semi-log x because we want our learning rate to be on a log scale. ls, and then find lr_history, our history variable from up before,.history. And we only want the loss component from the history. And then we should make our plot pretty, so we'll add xlabel. This is a learning rate, and the ylabel is going to be the loss, and then the title can be Finding the Ideal Learning Rate. Let's see what this looks like. Alrighty, so we can see when it started off at 10^-3, the loss decreased fairly sharply, and then it kind of plateaued all the way up to 10^-2, And then the loss just sharply increases as it gets closer to 10 to the negative 1. Hmm, now if we come back to our keynote, where is our finding the ideal learning rate slide? There we go. So the ideal learning rate is where the loss is decreasing sharply, find the lowest point on the curve, and then go back a little bit. Alrighty, so let's do that. So the lowest point on the curve is about here. Then if we went back a little bit to where it's still sharply decreasing, I would say, you know what, 10 to the negative 3 is— 10 to the negative 3 is probably our ideal learning rate, which happens to be— what optimizer are we using? We're using Adam. So as I said, this is proof, look at that, 0.001, that the default parameters for a lot of the different optimizers and other functions in TensorFlow are pretty darn good. So looks like for our problem in particular, the ideal learning rate is just the default value for Adam. So with that being said, for completeness, let's refit a model with the ideal learning rate. So let's refit a model with the ideal learning rate, and we'll go here, set random seed, tf.random.set_seed 42. I don't want the symbol, getting ahead of myself pressing the Shift key, Shift key. All of the keys are chef keys because what are we doing? We're cooking up neural networks. That's what we're doing. tf.keras.Sequential, wonderful. Now we come over here. tf.keras, we have to flatten our data from 28 by 28 arrays or tensors into a vector of 784. In other words, 28 by 28. And we go tf.keras.layers.Dense. Just the exact same as what we're doing before. Activation equals ReLU. We'll do another hidden layer. Layers, Dense, 4. Activation equals ReLU. Wonderful. And then we can come down here, tf.keras.OutputLayer. What does our output layer need? How many classes do we have? We need a hidden unit for every class, and we also need an activation activation function. That is not sigmoid, Daniel, come on, softmax. Thank you for catching me on that one. And we're going to compile model, model.compile. What's our loss function? tf.keras.losses. We're dealing with integer values, so we need sparse categorical cross entropy. When would we use just categorical cross entropy? If our labels were in one-hot encoded form. Now our optimizer, we don't even, we actually don't even have to change anything here with our optimizer, but we will just because we spent all that time finding the ideal learning rate, so we might as well put it in there. lr equals 0.001. So again, the default learning rate for Adam is actually 0.001, we can check that. By looking at the docstring. Wonderful. And then we can set up metrics equals accuracy, and we can fit the model. Let's save this to just history14 so we're keeping track of what we're doing. model14.fit, train_data, train_labels, epochs— this time, how about we go 20 epochs? We found kind of what the— oh no, we forgot the normalized data here, because remember our model performs much better on normalized data. We'll go the normalized data. We've got the ideal learning rate, which we actually didn't have to change anything for Adam, but we'll fit for a bit longer this time because we've worked out that our model is doing pretty good. So maybe this time we'll— the thing that we'll tweak is how long it looks at the data for. Validation data equals test data. We need to do this as a tuple actually. Test data norm and test labels. Wonderful. Let's fit that. Shouldn't take too long. Invalid syntax, of course. Oh, did you catch that? I didn't catch that. Wonderful. So I'm going to let this fit, but this should turn out to be a fairly well-trained model with close to the ideal learning rate and performing pretty well. So we've got a couple more options based on what we've done in the previous lectures. We can evaluate and improve our classification model. So we've done a bit of improving, so it's probably now time to, once this is finished fitting, is to start evaluating it with some techniques that we've used before. So I'll let this run through and then I'll see you in the next video. We'll start to, to run some more evaluation methods on our multi-class classification model. Welcome back. So now that we've got a model trained with the close to ideal learning rate and performing pretty well. Let's check back in with our workflow and see what we should do next. So we've got build or pick a pre-trained model. We've kind of done that. We've definitely got our data ready, turned it into tensors. We fit the model to the data. We haven't quite made any predictions yet, but making predictions is kind of synonymous with evaluating the model because, again, what happens during evaluation? We compare what the model should have predicted with the— what the model actually did predict. We've actually already improved through experimentation as well. So this is another key point to highlight here, is that although this is kind of like a linear step-through way to do things, this is just like a rough overall guideline to steps in modeling with TensorFlow. You can always jump back and forth through different steps and suit them to whatever problem that you're working on. With that being said, let's make a little heading here. Evaluating our multi-class classification model. And we'll turn this into markdown. And let's put in here a couple of things that we could do, or actually make this sound a little bit better. To evaluate our multi-class classification model, we could— what are some things that we could do? Hmm, evaluate its performance using other classification metrics such as a confusion matrix. Assess some of its predictions through visualizations. That's always a fun one. We could improve its results by training it for longer or changing the architecture. Now, this isn't really evaluating, but that's just another step that we could do now. And then, of course, referring back to our Steps with Modeling in TensorFlow, we could save our trained model so that we could use it later. So we'll put that there just so we know. Save and export it. For use in an application. Wonderful. How about in terms of sticking with evaluation, we go through this— oh, and I've made a typo. Assess should have a double S on the end. We go through the top two. So we'll write down here, let's go through the top two. So the first things first is create a confusion matrix. Now we've actually got some code up here that we did with the binary classification problem, evaluating and improving our classification— that should be model— but let's find where's the confusion matrix code that we had before. It should be somewhere up here. Far out, we've written a lot of code. Here we go, I'm back again. You've reached the time of the part 2 video where I'm going to tell you about a special code that you can use to sign up to the full course that you're watching, 20+ hours more content on zerotomastery.io for 15% off. So if you do go to the website and go to the academy page where you sign up and you enter the code TFLOW for TensorFlow, it's a shortened version of TensorFlow, TFLOW, T-F-L-O, all capital models, you'll get 15% off. But keep this a secret between you and me. You can share it with your friends, but don't leave a comment on this video because we want to keep it a surprise for people who've made it this far. But that's enough from me. I'm— I'll probably only see you one more time by the end of this video. I'll say goodbye to you. Enjoy. So this is our remix of scikit-learn's plot_confusion_matrix. What we might do instead of typing that all out again I might just break my rule here of writing code, and I'm just going to copy everything that's in this cell, and I'm going to come down to evaluating our multi-class classification model. This is why it's handy to make headings here, you know, so really lays out your work so you can just jump between different sections. Create a confusion matrix and have an idea— why don't we functionize this? Because I kind of issued that as a challenge in a previous video. It'd be cool if we could actually go through it together. So from sklearn.metrics import confusion_matrix, just in case we don't have it. We want this to run as a standardized cell. Now to functionize this, what should we call it? Let's call it make_confusion_matrix. And because the whole premise of an evaluation function or a confusion matrix in general, this function here is comparing the test labels with predictions. At a bare minimum, our make_confusion_matrix should take in test labels, but I believe scikit-learn calls them— is it y_true? Yeah, let's make it y_true so we stick with theirs. And is it also y_pred? Yeah, y_pred, y_true. And then the classes, we're going to set that equal to None. 'Cause if we want a list of classes, we can set it there. And then what else should we set? We've got the fig size, we've hardcoded that. Let's hardcode that into here, 1010, wonderful. Then what else is there? We could do the label size. Mm, why don't we just do the text size? Text size equals, we'll set that to 15 by default, okay. That should be some good parameters there. We'll have to tab all this across. Tab. So it's part of the function. And then now what do we have to update here? So confusion matrix. This is now going to be y_true. And then our preds are going to be y_pred. Wonderful. And then fig size equals fig size. That works there, so we can get rid of this. Yes, yes, wonderful. Classes equals— we're gonna have to set that to— I don't think we need that anymore. So set labels to be classes. So if classes exists, now the label should be the class names. Yes. Else they don't exist. It should just be a range of how big our confusion matrix is. Wonderful. Anything else we need to change? We could change these to text size. Text size. That way I know I set it as 15 up the top, but we'll just make all the text a very similar size. Why don't we do that? That should work. Text size, and we can adjust it. Size equals text_size. Okay, now all the text should be the exact same size. Beautiful. So now we've got a function make_confusion_matrix which takes in a bunch of true labels and a bunch of predictions and classes. That's what's important. So let's run this, make sure all the syntax is correct, remind ourselves of what our class names are. Okay, beautiful. But what don't we have? We have a function, make_confusion_matrix, and we could have really put some docstrings here to tell us what the function does, but that's all right. I'll leave that for you to explore. We have some true labels. So this is our test labels, but we don't have any predictions with our model yet. So let's make some predictions. Make some predictions with our model. And we're going to create y_probs for prediction probabilities. You might be wondering, what does that mean? We kind of covered it in a previous lecture, but we're going to reiterate here. So to make predictions, we just go model14.predict, and then we want to make predictions on the test data. And I'll put a little note here: probs is short for prediction probabilities. Because we've got the activation function of our output layer, the outputs of our model's predictions are going to be prediction probabilities. So view the first 5 predictions, and we're going to have a look at y_probs. Oh, what do we get here? Why are these like that? Hmm. So has it already rounded it for us? That is interesting. So this is Hmm, did we set our model to have— let's come back up here— train labels, test labels, sparse categorical cross-entropy, yes. Hmm, that means it's outputted what label it should be. I'm going to pause the video here and inspect this and then come back to see what's going on here. Now, that took me a while, but I figured out what went wrong here. So I said that probs is short for prediction probabilities, but as we see here, we get whole numbers here. Now, you might have spotted what went wrong. Our model predicted on the wrong data. Now, this is a very important point to highlight here, is that we want our model to predict on the same kind of data that it was trained on. Now, what's the key difference here? What did we do before with test_data and test_data norm. That should be the hint as to what we got wrong just before. So here's our test data and our test data normalized. What did we change? Well, let's have a look at this first example here. Test data 0 and test data norm. What did we do? We normalized it, remember? So our test data samples still have values between 0 and 2.5. Whereas our test_data normalized have values between 0 and 1. So that's why we get interesting outputs from our model when we ask it to predict on test_data non-normalized. But if we passed in test_data norm, which is the variable we created before, we get the outputs that we should get. So that's a very key point. I'm going to write a little note here. Even this is a reminder to myself. Let's get a key emoji in here. Note: remember to make predictions on the same kind of data your model was trained on. E.g., if your model was trained on normalized data, you'll want to make predictions on normalized data. Beautiful. So what are prediction probabilities? So these are different numbers. Let's get the first one actually. Y-probs, we'll get the 0. So the highest number here indicates the index. So we'll get our class names back up, indicates the index that our model thinks is most likely the value. So I believe it is this last one, maybe. So y_prob— we can use the argmax. Let's use TensorFlow, tf.argmax for arg maximum, which is going to give us the index where the maximum probability occurs. So all of these are values as to how likely the sample 0 is t-shirt or top or trouser or pullover. So the first one here is t-shirt or top, so that's a fairly low number, e to the negative 11. This is even lower, so it's definitely not trouser, or at least in the model's eyes it's definitely not trouser. So it seems that the highest value is this one here, which is— should turn out to be about 0.8. So that's the last one. So our sample 0, our model is predicting as ankle boot. So let's have a look here. So NumPy 9, and then if we take that index, tf.argmax, let's get y_prob , and then we index on our class names list. With that, we should get ankle boot. Wonderful. OK, so now let's turn our prediction probabilities from this— from a prediction probabilities array into integers. So we can do that by going— we'll get rid of this. Now convert all of the prediction probabilities into integers. So let's go y_preds equals y_probs.argmax, and we'll go it on the first axis. And then we will view the first 10 prediction labels. Y_pred. Boom! So now our predictions are in the same format as our test labels. So what can we do now? Well, we can compare the two. So I will leave that as a challenge for you, but in the next video we're going to use our make_confusion_matrix function to compare these two. And how about you could try another metric. Maybe look into creating an accuracy score as well. See if you can reproduce the accuracy score we got from model14.evaluate by comparing these two. Wonderful. Now that we've got our model's predictions in the same form as the true labels, let's create a confusion matrix to evaluate our model's predictions. But first, we're going to create a boring confusion matrix using scikit-learn, just so we can demonstrate how valuable our make_confusion_matrix function is. Confusion matrix, y_true is the test labels, and y_pred is y_preds. So let's see this. It's kind of hard to tell what's going on here, but with our confusion matrix, we know that down the diagonal we should have the highest numbers. So looks like our model's performing pretty good across all the classes. So the highest numbers are down the diagonal. Now let's remind ourselves of what an ideal confusion matrix looks like. So the correct predictions, the true positives and true negatives, are down the diagonal. But right now this is pretty hard to interpret. So if we were to send that to someone, they're probably going, what the hell is going on here? And this is where our pretty confusion matrix comes in. So let's make a prettier confusion matrix. And we'll go make_confusion_matrix. Yes, yes, yes. Y_true equals just the same as before, test_labels. Y_pred equals y_preds. Now we have classes is going to equal class_names. That's our list of class names of the Fashion MNIST dataset. Then the fig size, it's hardcoded as 1010, but Let's make it 1515 just to have a little practice. And then textSize equals 10. Let's see what happens. Ho ho, look at that. I think I'm gonna have to zoom out 'cause that's a big dog. Or maybe we change this to 1010, see what happens. Might be the text is too big now. Yes, it is. So 1515 was a good size. So let's go back to that. Alrighty. This one looks much better than before. So this is where you can really see the power of a confusion matrix. See how visual this is? Like you could, you could send this to someone and they'd be like, okay, they'd start to intuitively figure out. So the true label here, let's explore this. For the t-shirt/top. So again, the darker the square, the more predictions the model got right for this class. However, if we come over here, this square is pretty dark. I mean, 16% of the predictions of t-shirt top were shirt. Ah, okay. Well, does it make sense that our model is getting confused between the shirt and the t-shirt/top? I mean, to me, how do you decipher those, you know? And then where's another one? What else is our model getting confused with? So pullover. Okay, again, the darkest squares down the diagonal, that's good, but it's also getting confused a lot. With the coat class. So it's predicting coat when it should have been pullover 188 different times. Now, are they similar? Now, we're going to have to explore that in a second, but you can kind of imagine pullovers looking similar to coats. And what else is getting confused on? Ankle boot and sneaker. Okay, well, at least for ankle boots, they're a shoe at least. So a sneaker is a shoe, so that kind of makes sense. This is the kind of thing that you'll you'll be doing with your own problems. I see the shirt class gets confused a lot with t-shirt top, pullover. It's to explore where your model is making errors. And what you can do with this information is start to improve your models. So maybe the t-shirt/top class should actually be incorporated with the shirt class. Or maybe we need some more data of actual just shirts so that our model really learns to differentiate it between t-shirts and tops. But anyway, investigate this confusion matrix a little bit more. It's nice and pretty. In the next video, let's go back through— what did we say we're going to do to evaluate our model? Let's get visual a little bit more, assess some of its predictions through visualizations. So the confusion matrix is one way to visually explore it, but it's something else to actually look at a picture and look at the label that our model predicted versus the true label. So let's write some code to do that in the next video. We're going to start this video off with a little key. So let's go here. Key note. So what's our motto? Or one of our mottos is visualize, visualize, visualize. So often when working with images and other forms of visual data, It's a good idea to visualize as much as possible to develop a further understanding of the data and the inputs and outputs of your models. So I've also discussed the power of randomness when exploring your data. So how about we create a fun little function for— let's write this down so we don't forget. How about we create a fun little function for— hmm, what should we do? We should plot a random image, we should make a prediction on said image, and we should label the plot with the truth label and the predicted label. Yeah, this is a great, great way to evaluate our model. So let's do that. First, we'll need random from Python because we want to plot a random image. And now we'll create a function, def plot_random_image. I often create a lot of these little helper functions. If I want to do something over and over again, I make sure to functionize it so that I can use it multiple times. So We'll pass it our model. We're right now, I think we're up to model 14. We'll pass it a list of images we want to inspect, what the true labels are, and the class names. Beautiful. So we'll make a docstring here so it's a little bit complete. Picks, uh, what does it do? Picks a random image, plots it, and labels it with a predicted with a prediction, that'll do— and truth label. Wonderful. So we need to choose a random number. So setup random integer. We'll go i random.randint between 0 and len images. Oh, we need a comma there. Does that make sense? So i is just going to be a random number between 0 and the length of images, and images is going to be the images we wanted to look at. Wonderful. And now let's create predictions and targets. So the target image is going to be images . We'll just index on our random number, and then the pred, probs is going to be model .predict. So it's just going to take our model here,.predict on target_image.reshape . So we'll make sure it's in the right shape for our model because we're only predicting on one image here. So our model right now is trained on images in 28x28 size, but we're telling our model, hey, we're only passing you one image at a time this time. Then we're going to do pred_label equals classes pred_probs, and we want to get the argmax. Wonderful. And then the true_label is just classes true_labels . Beautiful. What else do we need? Oh yeah, we need to plot the image. So plot.imshow, we want target_image to be the plot, and we'll set it to a binary color map, so it's just black and white. Beautiful. And now let's do a little something a little bit fancy here. All right, we're going to change the color of the titles, so change the color of the titles depending— if I could type correctly— depending on if the prediction is right or wrong. So how might we do that? Let's just create a boolean. So if pred_label equals true_label, that makes sense. So if the pred_label is the same as the true_label, so the prediction our model has made is the same as the true label, yes, we want to set the color green because green means good. And else the color equals red. Wonderful. And then let's add xlabel information, and we want prediction slash true label. Let's go plt.xlabel, and how can we do this? Maybe we just do it with a.format, pred— what's the pred? Pred, and then we want to change this to— let's put the confidence in there, why not? 2.f— I always get confused when I'm typing out this little section of code. There we go, and what else do we want? We want the true. So the true is just going to be like that, and then chuck a.format on here. We could have done this as an f-string, or could we? Because we have to have a calculation. Yeah, we probably could have. Oh well, that's all right, this will do for now. We want 100 times for the confidence. We want tf.reduce_max, so in other words, find the maximum value in pred_prob, and then we'll just finish off with true. Label, and then I believe color goes down here. Color equals color. We've definitely got something wrong in this function, but— so set the color to green or red based on if prediction is right or wrong. So does this make sense? Plot random image, pass it a model, picks up a random integer, picks a target image from images, Then uses the model to make a prediction on the target image. We have to reshape it to be, because it's only one image. The pred label is going to be classes.pred_prob argmax. So the index of pred_prob, which has the highest prediction probability. The true label is going to be the true labels indexed on i. Yep, that's correct. Plot the image, target image, CMAP binary, yada yada yada. We could keep going through that, but let's just run the code, see if an error comes up. Oh, oh no, we're just defining the function there. We haven't actually run it yet. Let's do this. Check out a random image as well as its prediction. Oh yeah, plot random image. Model equals our current model, model14. Images equals test data because that's the test images we want to work with. Test True labels, sorry, equals test labels, and then classes is class names. How does it look? Oh, nice! Okay, there we go. So the prediction is coat, 100%, so that's a very high prediction probability, but the true is shirt. Does that look like a shirt to you? Kind of. Let's look at some more. Ankle boot, 100% true. Sandal. Ah, is that an ankle boot? I mean, that's a fairly pixelated image. That's kind of hard. Come on, surely our model got something right. Prediction: coat, true, is actually shirt. Wow, that's a bit— see, that's— that one's on the edge to me. As long as our function's correct. There we go, there's one that's right. T-shirt top, 100% true. Wonderful. Oh, What did we also forget here? This should be norm. Oh, come on, Daniel. Remember, always make predictions on the same type of data that your model is trained on. I'm gonna write a note here. Always make predictions on the same kind of data your model was trained on. And by the same kind, I mean preprocessed in the same way. All right, ankle boot, yes, that's what we want. Green, show me some more green. Ah, damn it. T-shirt top, it's a shirt. See, this is what I mean. I'm not sure who created this dataset, but to me, t-shirt and top is kind of the same thing as shirt. Maybe it's not, maybe it's not. Trousers, yes, that's what we want. Shirt predicted, or the true is a coat. Mm, kind of. So as you see here, you can keep going through this for a lot. We could have really just set this up to plot more than one at a time so we don't just have to keep sitting, enter, enter. Maybe that's a little bit of an extension for you, is to see how you could functionize this to plot, say, 4 images just like we did before when we were exploring our data. So going through this, I would say do another 20 or so, go through them, and then figure out, does visualizing these predictions here help you to better understand this confusion matrix here? And what I mean by that is, does it start to make sense where the model gets confused? In other words, like, does the overall shape of a trouser relate to pants? Or does the overall shape of an ankle boot— is the model getting mixed up with a sneaker because an ankle boot is a similar shape to a sneaker? Does that make sense? So we've done a bit of evaluating for our model. How about The next video— we've talked a lot throughout this entire series that our model is learning patterns. So how about in the next video we discuss what patterns exactly is our model learning. So have a little bit of a play around with our visualization function here. See if you can start to get an idea of where the model gets most confused and compare that to the confusion matrix. And then I'll see you back in the next video, and we'll check out the patterns our model is learning to make these kind of predictions. We've covered a fair bit— well, actually, fair bit is probably an understatement. We've covered a lot in neural network classification with TensorFlow. But this whole time we've been talking about how neural networks learn patterns in our data so that we can use those patterns later on. But what exactly do those patterns look like? Well, let's use this video to find out. So first we're going to crack open one of our models. So find the layers of our most recent model. So if you're not sure by now, take this as me stating it straightforward, is that a deep learning model is constructed of layers, and each one of those layers has a specific role in finding patterns in the numbers that we feed it. So let's have a look at model14 layers. So we can see, we start off with a Flatten layer, and then 2 hidden dense layers, and then an output dense layer, because we typically go from this direction, so top to bottom. So this is input, and this is output. And now we can inspect what's going on in a target layer using indexing. So how about we extract a particular layer? I'm going to go with the first hidden dense layer by indexing on 1. Wonderful. So there we've got an individual dense layer there. And now we can find the patterns learned by a particular layer using the get_weights method. So let's try that out. So get the patterns of a layer in our network. So we're going to set this up as weights and then biases. We'll have a look at that in a moment. Model_14 layers .get_weights. You can just search this method up if you want, but I prefer to figure out things ourselves. And then we'll go shapes. So we'll view weights, and then we'll also view weights.shape. Ah, okay. So this is what the internal patterns of a single layer, or specifically, the first hidden layer of our neural network look like. So to you, they might just look like a whole bunch of random numbers, and I mean, to me, that's what they look like. But to our neural network, it considers these as the patterns which contribute to the decisions that it makes. So we're working with Fashion MNIST data, so the shape of this weights matrix corresponds to the shape of our input data. So remember our input data was a 28 by 28 image. That's where this number comes from. So if we go 28 by 28, and then we remind ourselves of what our model looks like, what our model input shapes look like. So if we see here, this is a 784, the Flatten layer to begin with, and then this 4 comes from the number of hidden units in our first dense layer. So that means that for each data point in our input tensor, our weights matrix has 4— because see here, 4— has 4 numbers that it starts to learn and adjust to find patterns in these 784 numbers. So just looking at this off face value can be very confusing, and don't worry if you're not sure of it to begin with. This is the first time we've ever cracked open one of our neural networks, and this is kind of where the term deep learning gets the idea of black box from, as in when you crack open a deep learning model, you get all these random numbers. If you or I would try to interpret them, we can't really interpret what's going on here, but somehow they correlate to our model finding patterns. In the input data. Now, what each one of these values does, so each value in the weights matrix, it corresponds to how a particular value in the dataset should influence the network's decisions. Now you might be wondering, how does our neural network even create such values? How does it learn these values? Well, and let's go back to the keynote to see a high-level overview of how this might happen. So we're working with the input data here, which is grayscale images of the Fashion MNIST dataset. So we have coat, ankle boot, shirt, sneaker. Wonderful. And then we might encode them to— well, we should have actually normalized this data, shouldn't we? We might transform them into a tensor to pass it to our neural network. And then what's going to happen in our neural network, we've seen this kind of overall schematic before, is that it's going to learn the representation. Another word is patterns, features, weights, These can all mean similar things when you're hearing a neural network talk. For now, we're referring to them as weights, but I have been referring to them as patterns. And so what's going to happen is at the beginning, our neural network is going to initialize itself automatically with random weights at the very start. So if we come back, so all of these numbers, the weights, the internal weights, of one of our neural network layers are going to start off as random numbers, just completely random. And it does this, we can look at, if we go to the TensorFlow Dense layer, it does this using which parameter? There we go, kernel initialization. Now a little bit of extension on this video is that you can read into what this is, GlowRootUniform, but it might actually tell us here, Kernel initializer, initializer for the kernel weights matrix. So GlowRootUniform is a form of randomness. So just take that from here. We're not gonna dive into what GlowRoot is, but just understand that that's a form of randomness. Now if we come back, so it's gonna initialize itself with random weights in each particular layer. And then what's it going to do? Well, we're going to show it different examples of the data we'd like it to learn. So we might show it images of coats, images of ankle boots, shirts, and sneakers, and it's going to keep looking at these and slowly update its representation outputs, or weights and biases, into a different kind of tensor, which has come back to here. If we come back to our notebook, this is what it's going to learn over time, these representations, as we continually repeat with more examples. Does that make sense? Our neural network, we feed it input data, it starts off with random numbers, random weights, random patterns, however you want to refer to it. And then as it looks at more and more examples, it's going to go, "Hey, I'm going to try these random numbers and see if they correlate to any of the patterns in the data." And if they don't, well, the neural network is going to try and correct itself, thanks to our optimizer, Adam. And then we're going to repeat it with more examples as we have more data. Then it's going to slowly adjust these patterns to better suit the data as best it can, to eventually— hopefully it starts to output all correct predictions. That's the ideal case. So let's come back. Now, we've only talked about the weights matrix so far, but alongside a weights matrix is also a bias vector. So let's come down here, let's have a look. Now let's check out the bias vector. Let's go bias and biases shapes. We'll go biases and biases.shapes. So again, this is from a single layer, more specifically the first hidden layer in our current neural network. So we come down. What does this correlate to? Number 4. So this is a bias vector. There's only 4 here. So it's this 4 here. So that means for every hidden unit in our neural network in the first layer, it has one bias vector. Whereas for a weights matrix— this is the difference, the key difference between a bias vector and a weights matrix— a weights matrix has one value per data point, whereas a bias vector only has one value per hidden unit. And so what the bias vector does is— we'll write this— every neuron has a bias vector. Each of these is paired with a weights matrix. And so the bias vector also gets initialized, but this time, let's have a look up in the Dense layer. What is our bias initializer? Zeros. So you can kind of intuitively guess what zeros mean. If we come down, what does it say? How does it get initialized? Initializer for the bias vector gets initialized as zeros, at least in the case of a TensorFlow dense layer. Now, where it kind of gets tricky, right, is that sometimes depending on what layer you're using in deep learning, so within the TensorFlow Keras layers module, your weights matrix or the kernel initializer and the bias may be initialized differently. However, as you can see, we never actually set these variables. They got initialized by themselves. So this is what I'm saying, a lot of what TensorFlow does for you, the majority of calculations are done behind the scenes. So of course, you can dive deep into this as much as you want, and I actually am a big advocate for that. But to begin with, I just want you to focus on writing as many neural networks as possible and just getting them working. And then once you want to know more, start diving into the nuts and bolts of what's going on behind the scenes. So we've said what a bias vector is, now what does it do? So the bias vector dictates how much the patterns within the corresponding weights matrix should influence the next layer. Okay. So for every hidden unit, there's a weights matrix and a bias vector. So if we change this to 10, how many weights matrices would there be? And the same thing for this one, how many bias vectors? If we change both of these to 10, how many weights matrices would we have, and how many bias vectors will we have? Just have a think about that for a second. So this is a key point here as well, how much The patterns within the corresponding weights matrix should influence the next layer. Another big thing about deep learning. Let's re-familiarize ourselves with our model architecture. So model14_summary, what it looks like. Okay, so we've built a few of these. We've built a few deep learning models. So now's right about the time to point out the whole concept of inputs and outputs not only relates to the input layer of a deep learning model and the output layer, it relates to every single layer within a model. So let's go to the keynote, go to the next slide. So this is inputs and outputs layer by layer. So if we imagine this is our model here, very similar to what we've just built, Model 14, and if we, we can create this in a second. But if we imagine we have this input layer that takes in our input data, in our case, which is images, 28 by 28, into a tensor. We should have normalized this if we were really preparing our image data correctly. So this is going to take the inputs, and they output it to the flatten layer. The flatten layer outputs to the dense layer, or the first dense layer. The first dense layer outputs to the second dense layer. and the second dense layer outputs to the output layer. So for each layer in a deep learning model, the previous layer is its inputs. So this is the crux of deep learning. This is what makes deep learning deep, is that you have multiple layers, and of course, as you added more layers, the deeper the neural network would go, but each subsequent layer, does its part to find patterns in the original data and then feeds it on to the next layer. So as you keep going through, the patterns get more and more refined towards the ideal, or hopefully they get close to the ideal outputs that you're after. So let's have a look at how to replicate this, and I believe that will probably be more than enough to wrap up this video. So let's check out another way of viewing our deep learning models. So we can do this with from tensorflow.keras.utils import PlotModel and see the inputs and outputs of each layer. PlotModel, model 14, and show shapes equals true. Beautiful. So this is what we have. As I said, this is inputs and outputs layer by layer. So it starts off the input. The None here is for batch size. We'll tackle that in a future video. Students can look that up if they want. Let's just say we put in 28 by 28 images. So that's the input. Then this layer outputs the inputs to the next layer. And then this layer outputs the inputs to this following layer, and then so on and so on and so on until we get our idealized output, which is this 10 shape here, 'cause that's how many classes we have. Now, we've covered all of this very quickly of what's actually going on. I just wanted you to get familiar with cracking open one of our models. Now, if you want to dig deeper on what's going on behind the scenes here and how these calculations actually happen layer to layer, I've put some resources in the extension section, but otherwise, I think that just about wraps up everything in the introduction to neural network classification with TensorFlow. So go back through, pat yourself on the back, because we've covered an incredible amount. Take a break to let things sink in, and when you're ready, make sure you check out the exercises. So I'll just put this here. Next, check out exercises and extracurriculum. Check out the exercises and extracurriculum to practice and cement what you've learned. I've put in some stuff there to not only practice everything that we've gone through, that's in the exercises, but as I said before, the extracurriculum will help you really dive in to what's going on behind the scenes here. So with that being said, congratulations on finishing Section 2, Neural Network Classification with TensorFlow. I will see you in the next module. Holy bejeebers! If you've made it all the way to this endpoint and you followed along, you've gone through over 14 hours of video on YouTube, and I trust you've learned a fair bit about TensorFlow, a fair bit about how to write neural network code to solve regression problems, in other words predicting a number, and writing neural network code to solve classification problems, both binary classification and multi-class classification, which is some of the most common problems in the field of machine learning. So this is me signing out to you, and you're probably wondering, has he worn the same shirt through all of these little clips? And yes, I have. It's the magic of cinematography. But this is So big props to you. Thank you so much for watching through this. If you've loved it, please let me know in the comments. If you have any questions, remember the GitHub discussions page is your friend. Otherwise, if you'd like to sign up to the full version of this course and learn a whole bunch more about TensorFlow and deep learning in general, there'll be a link below. Sign up to the zerodomastery.io Academy, and if you've seen the other videos, you'll know what code to use when you sign up. That's it from me. Congratulations, happy machine learning, happy deep learning, and all the best learning TensorFlow in the future.