How To Do A Classification Tree in Python | Machine Learning in Python | Databytes

Hi, welcome to this supervised machine learning tutorial on how to do a classification tree in Python. Click the link in the video description to follow along in Datacamp workspace. First of all, you might wonder, when should I use a classification tree? And the answer is that you want the response variable. So the thing you're trying to predict to be logical values, so trues and falses or, or categorical variant. Now classification trees have pretty weak predictive power in general, so more on that in a moment. But it means they have fairly niche use cases. So the first case is that you need easy interpretability of the results. So for example, if you're working in a heavily regulated environment like finance or healthcare, you might have to explain your model to a regulator. And so classification trees are great because very easy to interpret what's going on. Secondly, they're really computationally cheap to run. So if you've got a very large data set and you're trying to train your model on something low powered like a laptop, then classification trees are pretty ideal. They also have a very niche but kind of cutting edge use case that rather than try to make predictions with the classification tree, use them as a feature engineering set. So you can use a classification tree to convert continuous features into categorical features. So this is feature engineering. And then feed the results into something more computationally intensive but with better predictive power, like a deep learning model. And as I mentioned, because these classification trees have weak predictive power, you actually, rather than just creating one of them, you can create an on ensemble of classification trees. So that just means lots of trees. And there are two broad classes of technique for doing this. You have bagging methods, which basically means you create lots and lots of different trees and then average them somehow or get them voting on like which parts of the tree are best. So this is how random forests work. There are also boosting methods where you do one tree at a time. So you start off with one classification tree and then you sort of iteratively improving the tree so it gets better and better and better, and that's how something like gradient boosting works. However, both those things are out of scope. For this tutorial, we're going to focus on single trees. Today. In terms of Python packages, we're going to be using scikit Learn today. You can also use Pycurrot if you prefer. Let's move on to a case study. So we're going to be looking at determining the variety of raisin. And this is an automated food detection problem. So automated food detection means you have images of food and you have to try and work out what food is in that image. So this is really useful for things like automated production lines in food production, things like food safety, trying to work out what allergens are present in food, things like dietary monitoring. So apps where you take a photo of your lunch and it tells you how many calories you've eaten. And this process traditionally has two steps. So the first step is you take your image data and you convert it into numeric features. So things like dimension, shape and color of your food. And then second step is to run a classification model on those features. So we're going to be doing step two today. So I've got a raising data set from the UCI machine learning archive and it contains the output of step one. So it's got numeric features. And we're going to be doing the classification part and we're looking at Turkish races today. So we're going to be using Pandas for importing the data, scikitlemph modeling and matplotlib for a little bit of work on the plot. So let's write some code. So we start off importing Pandas as a PD and then from scikit Learn. So from Sklearn we're going to import several functions. So I'm going to do this three times. Copy and paste, paste. So from the model selection sub module we are going to import train test split from the tree sub module we're going to import. So if I start decision. So in terms of classification trees that are actually it's a type of decision tree and there are two types of decision trees. You've got classification trees, classification decision trees when the response is logical or categorical. And then you've got response regression trees or regression decision trees, which are used when the response is numeric. So in this case we're going to be focusing on the classification trees which are called decision tree classifier in scikit Learn. All right, and then finally we're looking in the metrics sub module and take a look at the accuracy score and the confusion matrix. Oh, so one other thing, in order to plot the results, we also want a function called plottree and that is in the tree sub module of circuit learn. And finally we're going to do import matplotlib.pyplot as plt go to these in pyplot. Okay, so let's run this and we have everything we want now. So we're ready to import the data set. So we're going to do PD readcsv and the file is called raisins CSV. So raisins CSV. And let's just assign this to a variable called raisins. I'm gonna. Oops. I'm gonna copy and paste the name of this. You can see the output. And here you can see we've got 900 rows. So each row represents one raisin or at least one image of a raisin. The features that we're taking out of that. And I have to say, at first I got a little bit confused about what some of these columns meant. So major axis length, minor axis length, what's that? Eccentricity, what does that mean? So if we have a look at the data dictionary, it turns out this all makes sense if you think of a raisin as being an ellipse. So the major axis length, that's the longest part across the ellipse, minor axis length, shortest part along the ellipse and eccentricity is how close to circular is it? So an eccentricity of one means it's perfectly circular. Eccentricity with zero means like completely flat. It's just a line. And then if we have a look on the right hand side of the data set, they've got variety and you've got choices of Cassiman raisin and Besany raisin and the other features, just other kind of properties of this ellipse. So first data manipulation task, we need to split the data set into a response column and explain country columns. So the response column is the one called variety. So let's call this variable response in there. So we're going to start with raisins and we're going to get the variety column and then the explanatory variables. That's going to be. We're going to start with raisins again and want every column except variety. So we're going to do drop columns equals variety. So this is going to be a series because it's just one column of a data frame. And explanatory is going to be another data frame, just one less column than raisins. Next step is to split the data into training testing sets. We're going to be doing training the model on the training data and fitting making predictions on the testing data. So let's call train test split and we're going to pass in the explanatory variables. And the response variable and train test split returns four different things. So it's going to return explanatory train, explanatory test, response train and response test. Let's run that. We'll see how we split this up. So next step, we're going to fit the model to the training set. This takes two steps. So the first thing is you need to create a decision tree classifier object. This is going to be our model object. I'm just going to call MDL model. And so this is going to be decision tree classifier. And just to make things easier, when we come to look at the output, I'm going to set one of the parameters to call max depth to three. So that's it. We only have three different layers of depth within our tree. This will become clearer when we take a look at the output. And something you don't need to do in general just makes it easier to see what's going on. Now we're ready to fit the model to the training sets. We use MDL and then the fit method. So I'm going to fit this to explanatory train and response train. The output from this isn't very exciting, just tells you what kind of model we use. What is more exciting is doing predictions though. So now we're going to fit the. So we can do. Sorry, we're going to make predictions on the testing set using models. We're going to do model dot predict and we're get. This time we're going to pass in explanatory test. Let's run this. You see the result is a numpy array and for each raisin in the testing set, it's giving you its best prediction of whether it's a kessimin raisin or a besany raisin. So all very exciting. However, it's not that useful on its own. We don't know whether those are right or wrong. So we're gonna have to do some model evaluation. So let's save this to a variable. I'm going to call it predicted responses. Let's just run that. And so we're going to assess the model performance. Now there are four possible outcomes and either it was a besany raisin and we correctly did that, so that's a good outcome, or it was a keserman raisin and we also correctly predicted that that's a good outcome. And then there are two possibilities where we did wrong, where it was actually qasimin but we predicted besny, or it was actually besny but we predicted keseman and we can see how many times each of these things occurred with confusion matrix. And in this case we pass in the actual and predicted responses. So the actual responses, that is called response test. And the predicted responses are called predicted responses. So We've got a 2x2 matrix, same form as this. So in this case you can see 94 times. It was actually Bezny. We predicted Bezny 95 times. It was actually predicted casements. Those are pretty good. And then there were 12 and 24 cases where we got it wrong. Whether that's good enough accuracy unclear. And actually just from looking at this 2x2 matrix, it's really difficult to get a sense of like quite how good things are. So a better sort of way of figuring this out is we can use accuracy score, which is pretty common metric about the performance of a model. So I'll show you the code and then I'll explain what this means. So let's do accuracy score. And this takes the same arguments as confusion matrix. So I'm going to copy and paste this. So we want the actual responses and predicted responses. So let's run this score of 0.84. So in this case, what that does is the sum of the diagonals. So the sum of the correct responses, 94 and 95 divided by the sum of all the cases. This is how many rows there are on the testing set. So 94 plus 24 plus 12 plus 95. So you can see it guessed right 0.84 or 84% of the time. So that seems reasonable to me. But I don't have a strong sense of like whether that's good enough. All right, so that. So let's visualize the tree and you can see how the decisions are made. So we're going to call plottree and this takes a few arguments. So first of all it takes the model and then we want to set the feature names argument. And this is going to be the names of the columns in the expungery data set, which is explicit explanatory columns takes an argument called class names, which is basically the the names of the different values in the response. So Besny and Kisserman, we could type this manually, but it's sort of better to do this programmatically so we can say it's the unique responses. So response response unique. So the unique values and responses, that's based on your guess. And we're going to use sorted to make sure that they are in alphabetical order. Finally, the plot is a little bit easier to interpret if you use an argument called filled, make that true. I will explain what that means when you see the plot. Let's have a look at this. Now this actually dumps a load of annoying Output there. However you can see we have a plot and it looks like a tree upside down. So this is the root and then these are branches and then you have leaves at the end. Let's just go back and tweak our code a bit to make sure the output is a bit more readable and I'll go through explaining this in a little bit more detail. So first thing, we're going to do some a bit of matplotlib coding. So we're going to use plot, dot, figure, and let's make this a little bit bigger using fig size. So we're going to make it 10 inches by 15 inches. And then if we do plot show at the end, it's not going to dump all this text output. We. One other thing, let's change the font size to 11. Turn it up to 11. All right. Boom. That's a little bit more readable. Okay, so what happens is you start off with a row in your data set and then you check the major axis length and if the value is less than 422.423 you go left. If not, you go right. Spoiler. A classification tree is basically just a flowchart. And then if you went left, you would check the perimeter, see if the value is less than 1125. If you went right, you check the major axis length, that says it less than 466. And you just work through each of these decisions, hence the name decision tree, until you get to the end and you're going to end up in one of these boxes and it will tell you, is this a Bezny raisin? Those are the red ones or is it a Qasmin raisin? Those are the blue ones. So in this case the color is the strength of belief about how right it thinks these decisions are. So in this case, dark blue is. It's, it's pretty confident that these are all Kessman. This one is very pale red. So it thinks they're besny but it's really not sure about it. Anyway, that's how the model works. Underneath, when you're explaining to your regulators how your model works, you can go through this. Okay, so we've classified some raisins here. You want to learn more? Classification trees are covered in this Scikit Learn tutorial which goes through understanding the decision tree structure. So what this plot looks like in a lot more detail. Classification trees are covered in so many places in data camp. Many different courses cover it. I would start with the machine learning the tree based models in Python course And then there are also some industry specific machine learning courses like Machine learning for Finance and Python Machine Learning for Marketing and Python. Those also cover different types of decision trees including classification trees. And then there is a decision tree classification workspace template if you want to apply your skills. I hope this has been helpful. Please do subscribe. Please check out future tutorials. This has been wonderful. Thank you. SA.