Code with me (live): How to make your first Kaggle submission from scratch!

Alright, I think we might be live. Are we live? I'm not sure. Test. Oh, I'm checking the live. Here we go. Yes, it's a little bit delayed. There we go. Alright, we're live. Sweet. Sorry for the delay. I need to pause my own live stream so I don't hear myself echoing. All right. How we going, everyone? I got to remember where to look. This is my first time doing this. Actually, I might have to keep the chat open so I can see what you guys are saying. I'm really excited. We're going to do an exploratory data analysis on a Kaggle dataset. So let me, let me rearrange my screen windows here. So I've got it all, all sorted, all hunky-dory. How y'all going? Hey, hey, Kashuk, how are you, mate? Nice to meet you. Actually, I think the chat will load if I just keep this window open. Can I chat? How are you all? It's a beautiful, beautiful sunny day outside where I'm at. We'll just, just let this run for a little bit to, to load everything up. If you can't make the stream, if you're just checking it out, we can— it's going to live on the YouTube channel forever. So what's the purpose of this stream? Well, since, since becoming a machine learning engineer, there have been a few things that I've really had to upskill myself on. And that's the number one thing is probably How do you explore a dataset from scratch? Right. So that's, that's probably, that's probably number one. And so that's what we're going to do today. We're going to start with a dataset where we're going to go to Kaggle. We're going to get the Titanic dataset. We're going to explore it. We're going to make a Kaggle submission. So if you've, if you've never done a Kaggle submission before, you can go through this notebook. You can go through this live stream. And you can, you can do your first Kaggle, official Kaggle submission on the Titanic dataset, which is, which is pretty exciting. If you, if you want the code, it's all going to be available on my GitHub after this stream. So I'll make sure to put that in the description. Otherwise, before we jump, big shout out to Coursera. Thank you so much for, for sponsoring this video. A lot of the stuff that I've I've done in this notebook has been from the Applied Data Science with Python course. I mean, learning pandas is fundamental to exploratory data analysis, and the Applied Data Science with Python by the University of Michigan— bit of a mouthful— is an incredible course. It's available on Coursera. And also, due to this partnership, Coursera, if you want to sign up any type of computer science course, they're offering a 1-month free trial till December 11th. So if you sign up before December 11th, you get a 1-month free trial of any course in the computer science section. So be sure to check that out. The link's in the description as well. That should already be there as long as I've got this right. Now, if you can't hear me, if you can't see me right, if you can't see the screen, please leave it in a chat. Oh no, the stream is cutting out. Empiricist, we're just starting. Hey, can you hear me? Oh, please tell me this is working. All right, it should, it should be working from my end. It should be working. Alright, so if anyone watching, if you give me a heads up, we'll get this started because I want to make sure it's all good. So just leave a message in the chat. How we going? Otherwise we might have to work out another, another way to, to fix this up. It's working fine. As I said, I've never done this before. It might have been better to record it, but we're going to try to do it live. And if all else fails, we can do a recording and do it later. But I want to have the chat here so, so you can, you can ask me questions along. Alright, so I'm getting a heads up from Empiricist Academy. Thank you so much, Ahmad, for tuning in. Let's, let's get started, shall we? So the goal of this is to predict who would have survived on the Titanic. So if we look at, we look at Kaggle, there's a, there's a project here, right? It's called Titanic Machine Learning from Disaster. This is where you should start if you want to jump into Kaggle competitions. So predict survival rates on the Titanic and get familiar with some machine learning basics. Um, so there's some descriptions here. Uh, you can get this at kaggle.com/c/titanic. Um, otherwise, If you want the data, you can download it here. There's a couple of different things, but we will eventually be creating a submission here by the end of this stream. Right. So let's, let's go through with that. Now we go to the, go to the precooked notebook that I have here. All right. Can we predict who would have survived on the Titanic? Oh, there we go. Now, as I said, by the end of this stream, this will be up on my GitHub so you can go through it yourself. Expand on it. I put some challenges in here that I will explain, and I'll do my best to explain each step and what I've done, why I've done it. And by no means is this an exhaustive list of things that you should do with the dataset, right? It's, it's just an example to get you started. I've put a bunch of challenges through here, and I mean, I'm, I'm still learning as well. So pull me up, hit me up in the chat if, if, uh, if I'm doing something wrong, all right? So why EDA? So what is EDA? EDA is exploratory data analysis. So not all data comes in a neat little package ready to be modeled by the latest and greatest machine learning models. Most of the time you'll get a dataset you don't know much about. So before you can make any solid predictions, you'll have to try and find out more about the data. Right. And what are predictions? So predictions are running your machine learning model over, over some historical data and then trying to predict the future based off what's happened in that historical data. All right. So this is where EDA comes in. When you first get a dataset, you've got to start exploring it, start figuring out what, what is the question I want to answer. And now remember, this is exploratory. So we're not writing the crystal clear code. We're just trying to go through it step by step and figure more things out about the data. Okay. So step 1 is download the data. You can download the data from, from Kaggle. I showed you that before. So boom, you go to this little data tab and here it is. Right. So that's kaggle.com/titanic. If you just Google Kaggle Titanic data, you should be able to download it, but we'll get this submission tab ready. We'll go back. Neural, you're awesome, man. Thanks, bro. You're awesome too. All right. Be sure, guys, if something happens to the stream, hit me up in the chat because I've got it paused, my live version, so, so I can, I can keep this going. So step 2, work through the notebook. All right. This is what we're going to do. We're going to go through step by step of, of what you would do if you're exploring a data just a dataset you've, you've never seen before. So I've got some, some basic import dependencies here. All right, so you should be able to run this cell in the Jupyter Notebook regardless. We're doing this matplotlib inline so we can see some graphs in the notebook. If you don't run this little, if you don't run this little line here, matplotlib.inline, you won't be able to see see what's, see the graph in the actual notebook. Okay. And then we get some Python imports, standard ones like math, time, random, datetime, data manipulation, manipulation, data manipulation, Python, pandas, NumPy, visualization. So this is where the graphs come in. This is where we want to see our Seaborn graphs. Seaborn is a graph library, visualization library. It's really pretty. Pre-processing. So we've got some one-hot encoding, we'll get to that. Machine learning, a whole bunch of different machine learning models. We'll get to those as well. And we're going to hide the warnings here that Pandas sometimes throws because we don't want to see all that red stuff throughout our code. You can comment this out and see what warnings they give you, but comment it out, and by comment it out, I mean, you can just put a little hash at the start. But we won't need those for now. Trust me, I've, I've gone through this notebook. The warnings are just unrelated. All right. So we're going to run this one, run this cell, Shift+Enter. Oh, actually, to make sure it's really live, let's restart the kernel. What does that do? So that means anything that's stored in memory is going to be deleted. So where this is as if we're starting it, you downloaded this notebook on your computer. And you were starting it from fresh. Now, if you were to run this cell running it from fresh, you might get some errors here because you haven't installed some of these. Now, if you get, if you get an error that says you haven't installed NumPy, Google how to install NumPy, right? How to install the thing you have and install it in your, through your terminal or something like that so you can get it running and then repeat that for each one that you need. So we're gonna, first of all, we're going to load in the data. Hassad, how are you, mate? Hamad, yes, I made this notebook. Merlin, how's the health app coming along? Still, still in a very much planning process. All right, so loading in the data. After we downloaded the data, we need to get into the notebook. We need, sorry, we need to get the data in here. So this is training test gender submission. So what is this? So if we look back at the, the Kaggle data, we go back here, you can download 3 different data sources, right? Gender submission test CSV, train CSV, right? So these are all— CSV stands for comma-separated values. And now this is what it looks like, right? So you'll have 892, comma, 0, right? So that's what this gender submission file looks like. This 890— but what is that? 892, comma, 0. This is why we bring it into our notebook so we can see it a bit better. So we're going to download these 3 files, right? So you've got these in your, in your home directory or wherever you, you've downloaded them. So I've downloaded them. I'll show you here. I've downloaded them into my Titanic folder and then you've got data here. And so see these 3 files, gender submission, test submission, train. Etc. We've got it here, right? So it's in my data folder. Now we're going to put that back over there and then we go back into our notebook. All right. So this is where, where it's stored. So we're importing the train data, data.train.csv, data/train.csv, test equals. Now what's this function here? pd.read_csv. So that means Pandas, read this CSV and save it as test. Now we do it for all, all 3 of them, right? So now we've got them, we've got them as variables, as Python objects, and they're in a Pandas DataFrame form now. Now what's a DataFrame? Now remember that weird way of looking at things over here? So this is how a CSV looks, right? But then we move into back to here. And this is how a DataFrame looks. So there's our gender submission. So passenger ID and survived. So this is an example submission of what we want to, what we want to get to at the end of this notebook. All right, and this is the data we're going to use to build that, that submission. Now if we look here, this is the training data. We have passenger ID Ahmed, thank you so much. Cheers, my friend. All right. So, sorry guys, I got distracted reading the chat, trying to maintain both of this at the same time. We got chat, we got notebook. All right, I'm going to be bouncing in between them as best I can. So we have the training dataset here. So this is what it looks like. We have passenger ID. Now, now how long is this? All right. Actually, we need to import first, need to run this cell, run this cell. All right. How long is this training dataset? So length, len is just length of the data frame. So this is, we have 0 to 4 examples here, but there's actually 891 rows of this. So we've got Mr. Owens, we've got Miss, Miss Leanna, We've got Mr. William Henry. And now this column here, survived, says is 0 for if they did not survive on the Titanic dataset. So, that means these are the actual passengers on the Titanic, right? So, this is, it's kind of sad, right? We know that these people, some of these people perished from the Titanic. It's a tragedy, but it can help. It wasn't in vain. They didn't die for no reason. They can help us. Practice machine learning and hopefully then we can, we can use those skills to bring some value to society, right? So, they didn't just board it on and drown or pass away for no reason. At least we can learn something from this. Alright, so, survived. So, Mr. Owen Harris-Brawn, 22, male. That's a young lad, tragically didn't survive. So this is his information here. He was in P Class 3. What does that mean? We'll explore that in a sec. 22 years old. He, Sib SP. What's Sib SP? P Arch 0 ticket. That's his ticket number. He paid $7.25. Probably a lot of money back then when Titanic was around. Cabin Nan. What does Nan mean? Nan means not a number, right? So there's no value here, but we'll look at that in a second. Cabin C85, that's for, that's for Mrs. John Bradley. John Bradley. Hmm. And embarked, S, C, S, S, S. What does embarked mean? We'll explore that. That's part of this exploratory data analysis, right? But this is just an overview of what the training data looks like. And now we have the same for test. Now test.head. Head means to view. Let's do this. Head view first 5 lines. It's a real way, real good way to view the DataFrame without having to see the whole thing, right? So if we just type in, um, test, it'll show us the whole thing. But what if this one only has 418 rows? What if you had one that had 1,000? Or actually, no, not even 1,000 because computers are fast, they can deal with 1,000 pretty well. If you had 10 million Right, you might not want to look at all 10 million rows at once. Um, let's see the length of our test dataset. Oh, type that in wrong. See, that's the beauty of doing this live, right? There's going to be some errors. So there's 418. So here's what we have to do, guys. We have to use the 18— 891 rows of, of the pass— of the training dataset. So we need to learn information from these rows to predict on the test dataset whether these passengers survived or not. So this, as you can see, training has a survived column, but the test dataset doesn't have a survived column. So what we want to do is build a machine learning model that is able to learn across this dataset and predict this column. So survived on the test dataset, right? And that's what we're going to submit. So our end submission is going to look something like this. So it's going to have passenger IDs. So we're going to take this column from the test dataset and then use our machine learning model to predict the survived column. All right, and then we're going to submit that to our Kaggle competition, and then we're going to get glory. Well, Maybe depending on how well our submission goes. All right. So back to some data descriptions. So remember how we looked at these columns? We have SIBSP, PARCH, ticket, fare, cabin. What do these actually mean? Right. That to me, that's just numbers. Like, is that, is that anything significant? Why are there so many NaNs in cabin? You should stream in it. Yeah. I'm not sure what— I'll have to fix it for the next stream. If I'm not streaming in high enough quality, I'll make sure to fix that by next time. 'Cause this is my first one, I'm not quite sure why the quality is not that great. Might be my internet connection. But we'll do our best for now. But thank you, Nikitan, for the comment. All right, so back to data descriptions. So remember these? SibSig, pH. Let's have a look. This is what comes in the Kaggle. Kaggle dataset. So it comes with this, a data dictionary. So it tells you survival is— gives you definition, gives you the key. So if we look at this, so sib_sp, number of siblings/spouses aboard the Titanic. Alright, so that means if I, if I'm aboard the Titanic and I have one little brother on there, this should be 1 for me. And p_arch is number of parents/children aboard the Titanic. Alright, so if I'm on there and I have— I'm a father I'm a father and I have 2 kids. My Parch should be 2. All right. And then embarked ticket is ticket number. Passenger fare is fare. Cabin number is cabin. Embarked port of embarkation is C equals Cherbourg, Q equals Queenstown, S equals Southampton. All right. Now we know a little bit more about the data. Now I've put these here, right? I've just manually wrote them in. So now we sort of know. All right. So Sib SP, Mrs. James Wilkes must have had— we look here, number of siblings/spouses. So she must have had either a husband or a sibling on board. Right. So at least one. But we'll get to that. We'll get to that in a second. Maybe there was, maybe there are some people who had more. but we can't see that because we're only looking at the first 5 lines. That's where we run some, some more, more, more, more in-depth exploration. Now what you can first run is a train.describe. So this function will, will just pull out all of the, all of these little information bits here. This is a function built into Pandas. You can run that on any DataFrame. So if we remember what train looks like, so train is is our data frame here. And we're going to describe that here. So if we look at this passenger ID, we don't really mind about that because that's just, we know that's just labels of 1 to 891. So of course there's going to be 891. Oh, but what's this? 714 in the age. All right. So maybe there's some missing values in the age column. We'll check that in a second. Sibsp, these all have 891. That's good, right? So that means there's values in the columns, 891 values. That's what we want. 'Cause that's the length. That's the overall length of our dataset. So if we look in here, length train, 891, beautiful. So this is also going to give us a mean, standard deviation, min, 25% threshold, 50% threshold, 75% threshold, and the maximum. We're not really going to worry about those at the moment. We're going to do a bit more of our own exploration, but this is a really quick function that you can run on any DataFrame to quickly get a bird's-eye view of what's actually going on. All right. Hey, Shaikh, what's happening, brother? Thanks for joining in. We're doing some exploratory data analysis on the Titanic Kaggle dataset. But thanks for showing up, man. It's good to see you. All right. So first of all, in any dataset, what I like to do is find out where the missing values are. Okay. Why, why would you do that? Because we want to figure out where the holes in our data are. Right, because you can't make a prediction when there's no data there. You have to figure out how to plug that hole. Or if you do make a prediction when there's no data there, how actually valid is that prediction? Right. So first things first, you can run this great little function missing_no.matrix on train figsize equals 1. What is this? What is this line doing? All right. We come back up here. We imported this missing_no. We can just think of this as missing, missing numbers or missing information. Let's just call it missing information for now. So missing information, aka missing_no,.matrix.train. What's train? Train is our DataFrame here, this one, this information. And figsize is just the size of this little graph here. Right. So if we change this to maybe, maybe 10, it'll be a bit, bit higher, I think. Yeah, there we go. Beautiful. So we're running missing_no, a matrix, feeding it the train dataset, train dataframe, and we're telling it we want it to be 30 pixels wide by 10 pixels high. Now, what is this? What does this look like? Now, again, always, always be looking at your data. Always check it out. We're going to look at that here. So train.head. Always keep paying attention to your data. You want to make sure you're doing the right thing. So, okay, we can see we've got a column here. Passenger ID, survived, P-class, name, sex, age. Oh, hold on. Age has got a lot of missing values, right? So, so if you imagine the black columns are fully black columns, no lines here. That means that there's— these have values, right? But the ones that have white gaps, that's a missing number, right? So our age column has a fair few missing numbers. Alright, so we'll have to figure out a way to deal with that. Potentially not in this EDA, but in your future EDAs, you'll have to go through missing numbers. And then we have SibSp. That's good. That has lots of data. Parch. Ticket, fare, cabin. Oh, cabin's maybe not even worth looking at because look how many— look at that. That's just most of it is not there. Put it that way. Embarked, that has, that has 2 missing values. This is a really good way of visualizing quickly where the missing data is in your, in your data frame, right? And you can show that to someone like, say you are working with the client, you're like, You go to the client, you're like, hey, I'm gonna run this quick function on your data and, and show you this. And you can, you can send that to them and go, your age column has plenty of missing values, your cabin column has plenty missing values. If you have that information, it would really help us for, for, for making predictions on your data, because machine learning engineers and machine learning models love more data. All right, so We clearly see some missing values, especially in the column— cabin column. It's important to visualize these missing values early so you know where the holes are in your dataset. So important. Alright, we'll revisit— we might not— we'll revisit this in a second. Alright. Now, find missing values. So what I've done here is another form of this. Not as pretty, right? But it does the job. I've wrote my own function. Okay, it's called find_missing_values. Nice and simple. You pass it a DataFrame. What's a DataFrame again? It's this, so train, and you pass it the columns you want it to scan over. So what it's going to go through is going to create an empty dictionary. It's going to go print number of missing or NaN values of each column, and then it's going to loop through. It's going to actually— it's going to create length DataFrame. So that'll be— can you remember how long our training DataFrame was? We can figure that out really quickly. All right, so length train. So length, length df, df length is going to be 891. So then it's going to loop through columns. So what's the columns? So, so we passed it train.columns. So it's going to— maybe we pass through this, it's going to loop through all the columns. So for column in columns, so it's going to go total column values equals df column.value_counts.sum. SUM, all right, then the missing column values is going to be the total length minus the column values that it's found. So this total column values, what will that do? All right, so that's going to go through the column, all right, so maybe it goes through the age column, okay, and it's going to total up the amount of, of black lines, okay, so it's going to total them up and then it's going to sum. Sum all those black lines up. All right, so this might, for age, for age, it might, might turn out to be, uh, who knows, maybe there's 150 missing values here, but we can run this function to find out. And then it's going to, going to create a dictionary. It's going to update the dictionary to say df.length, uh, the missing values column, df.length is going to equal the total length of the data frame. So this, this total length minus the number of values in the column. Okay, so let's run this and see what happens. Okay, so here's our dictionary. If we revisit up here, this should be 0 for the first 5 columns. Yeah, we have 1, 2, 3, 4, 5. Yep, 0. And now age should have a fairly high amount of missing values. Let's check that. Age, 177 missing values out of 891. Okay. We're going to have to do something about that. And then we have SibSp, pH. Okay. These all have values there. And now cabin, 687 missing values out of 891. Woo! A lot of missing values there. And then embarked has 2 missing values. So not that bad. All right, so, so maybe we, we can easily discard those or fill them up, but we'll have a look at that in a second. All right, so we've gone through the missing values of the data frame. We've worked out where the holes are. Okay, so now what we're going to do is go through each column and check out more in depth what the actual data contains. So the information that is there, right, or isn't, what can we do with that? All right, is there stuff that we can, we can manipulate it in some way? What's, what's, what's holding us back from running a machine learning model? Because that's, that's what we want to do, right? All right, so to perform our data analysis, we're going to create two separate data frames, one for discretized continuous variables and another one for continuous variables. So what does this mean? What's a discretized continuous variable? Well, a discretized continuous variable is, is a continuous variable that's just been chopped up into bins. All right, so let's just say, what's a continuous variable? First of all, a continuous variable, um, is, is say like fare. All right. Say fare, you have values. We can do a simple example. We don't even need to use fare. So say you have fare values range from 0 to 100. Right. So you could have any value in between 0 and 100 for fare, and that would be a continuous variable. But if you wanted to discretize that. You could bin them up into separate buckets. So what does that mean? So maybe you have a section that's 0 to 10, you have a section that's 10 to 20, and you have a section that's 20 to 30. So now a continuous variable would be all numbers between 0 and 100. But it would be binned in a discretized continuous variable if you put all the numbers between 0 and 10 in, in this bucket and have a label of 1. So now you would change your values from being 0 to 100 to being values from 1 to 10. And if it gets a number 1, it means it's, it's from 0 to 10. If it gets a number 2, it's from 10 to 20. And if it gets a number 3, it's, it's 20 to 30 and so on. All right. So that's what we're going to do. We're going to make 2 types of data frames. We won't necessarily need both of them for this type of analysis, but it's good to separate your values and you'll see how we separate them in a second. It's good to separate them into 2 sub data frames so you can potentially run, um, different types of models over, over each data frame. So we'll create those. And now, what data types are in the data frame? Before we do that, we'll check the chat. What's up, Stefan? How you doing, man? Um, discrete labels. We are still survive. Please make a short video for the Kaggle competition in future. Great idea. I will definitely do that. Um, but I want to make this one an extended one just to go through this whole notebook. Um, but we'll make a shorter version of this in the future. Um, what data types are in the DataFrame? So you can do that, train.dtype. Remember, train is our main DataFrame we're looking at now. So passenger_id, that's an integer. Survived, that's an integer. Pclass, an integer. Name is an object, sex object, float, int, int, object, float, object, object. All right, that's all right. We got, we got a good mixture of integers, objects, floats. Okay, let's explore each of these features individually. So we're going to start with survived. All right, so survived. Let's, let's remember what our DataFrame looks like. Always remember what it looks like. What columns are there? What, what information is there? Because we don't want to lose track of the problem we're trying to solve. We're trying to predict whether someone survived the Titanic based on this information. Okay, on the test dataset, always define your problem at the start of any exploratory data analysis and use that problem to guide your, your exploration. All right, you might not find the answer, but you might also raise different questions that will help you figure out what's not the answer, which is almost as valuable as as figuring out the exact answer you're looking for. All right, so you're eliminating things that aren't the answer. So we're gonna go through this in an iterative process. Now that is what an EDA should be, an iterative process. You're not writing pure, clean code. You should strive to as best you can, but we just wanna figure out as much as we can about the DataFrame as quickly as we can. Well, not as quickly as we can, but sometimes you have project deadlines, right? Kaggle competitions don't last forever. This one lasts forever. Not a good example, but when you're doing something like client work for a customer, you may have 6 to 8 weeks. So you don't have all the time in the world to explore. I know for myself at Max Kelsen, a lot of our projects go for 6 to 8 weeks, right? So we have to get this exploratory data analysis done in the first week so we can start to build models and get them back to the client. So first column first, makes sense. We're going to go through the target feature, which is survived. So description, whether the passenger survived or not. And it's really good to communicate this stuff. Okay. So, so the reason being is, is say your colleague or say one of your teammates had to read your code, can they understand what you've done? All right, that's what, what I need to really work on as well. I'm gonna— I think I've communicated this one pretty well. You can give me some advice or some feedback once you see it on GitHub. And, and yeah, it's important to communicate, always important to communicate. This is the variable we want our machine learning model to predict. So we're going to make a little plot here. This is using the Seaborn library. Remember, we imported that way back up here. Import seaborn as sns so we can use sns whenever we want. Now we come down here. How many people survived the Titanic? Now this might be a bit of a graphic. Yeah, it's going to be— it is going to be a graphic, but it might be a bit graphic considering I've seen the data before and it's, it's, it's not the best case scenario. Boom, there we go. So what are we, what are we actually printing here? Now we're getting sns.countplot. So the y value we're giving it is, is survived. So the y value on here is going to be 0 or 1. Now countplot just shows the total amount of numbers and the data is our training data frame. So this is a data frame here. And so it's going to bring back the total number of zeros in this column, right? 549. But you can, you can slightly see that I've printed out the value counts here so you can see the exact number. Now that's just train.survived. So train.survived.value_counts. So 549 people in the training dataset did not survive the Titanic crashing. So rest in peace to those. And then we have 342. So thankfully a fairly large amount of people did survive. So then we can see that here. So now we sort of know what the ratio is of how many people survived, how many people didn't. So, so we know in the future if our machine learning model is predicting on the test dataset that more people survived than didn't, then we probably got something wrong, right? Why? Because we can see from this training dataset that the number of people who didn't survive is clearly larger than the number of people who did. All right. So that's, that's just a little tidbit to remember when we're doing our first predictions. So we're going to add, add this column to our subset data frames. So now we've got, we've got two, two subset data frames that we can use later. Boom, df_bin and df_con. All right, so they just look the same right now. All right, now we're going through the next, next column. So pclass. Boom, there we go. Pclass, what values are in here? The key, the ticket class of the passenger. Now key is number 1 is first class. So those people are a bit fancy, right? Fancy. First class, I'm not sure really, I'm just making this up. Second class, third class, nice and simple way of labeling things. This is unaltered from how we downloaded from the Kaggle dataset. So first of all, we're going to plot the distribution. Now, why would we plot the distribution? Look at the distribution of each feature first. If we can understand what kind of spread there is across the dataset. Now, what do I mean by spread? All right. Now, we can do that. There won't be many outliers in here. And what do I mean by an outlier? Okay. So we've got a little highlight of this graph already. All right. But say you had some column way out here. Right, and it's, it's 10 instead of, instead of 1, 2, or 3. Do we need 10 in our dataset? And this is not a good example, right, because the data is, is pretty, pretty well close, but 10 is out here. Imagine if that had one— you had one variable that had 10 out here, all right, out of, out of 891, uh, rows in our training DataFrame. If you had one that is totally out here when everything else is here, you might not want to use that in your dataset. Okay, so that's the concept of outliers. We won't dive too much into that, but that's something— that's a little challenge. That's challenge number 1. Check out what an outlier is. We're going to have challenges all throughout this notebook because I don't want to just give you the way of how to do everything because really there is no way of how to do every single thing. You need to explored. That's the role of a machine learning engineer. That's the role of a good data scientist, is to figure these things out. Okay. So we can see with this feature, the values are numerical, 1, 2, and 3. So they're numerical, but they are also categorical, right? Why? Because what is that? If you— so a numerical feature, you could say, say this, these were numerical. So that means if you're in class 1 and someone else is in class 2 and you add those 2 people together, they equal someone in class 3. Does that make sense? No, it doesn't make sense, right? So they're categories. So you have 1, 2, 3. They're not necessarily related to each other because they're, they're separate, separate categories. Okay. So how many missing values are in our P class? 0. How do we figure that out? Because we got our missing values dictionary up here. All right, we're just calling this dictionary missing values pclass 0. So we come down, we go, go to that, and we're going to add our first feature that is not our predictor variable to our subset data frames. So now we've got survived and pclass. We're going to keep working through the training data set feature by feature. All right, it's, it's as I said, iterative process. Sometimes it can be tedious, but it's important to understand the data you're working with. Here we go. What's up, Daniel? Hey Raymond, good to see you, my man. Good morning, Mohit. Put this up on your channel, Shaikh. I'll definitely put it up there. It's called a nominal variable when the categories are not ranked. If they can be ranked, then it's an ordinal variable. Hamad is 100% correct. Yes, I did not— the actual names of the variables escaped me. So if we come up here, so, so this is— these are nominal variables. All right, as Hamad said, when the categories are not ranked, if they can be ranked, then it's an ordinal variable. Okay, so like an order. All right, these are nominal. That's, that's homework. That's homework number 2. Figure out what the difference is between nominal and ordinal. And I'm half saying that because I need to work it out completely in my head too. Okay. Feature name description, the name of the passenger. All right. How many different names are there? Oh, actually, this doesn't show us exactly. All right. Look how many different names there are. So value_counts is a beautiful, easy, easy-to-use function that will give you the counts, number of appearances a certain value appeared in a, in a, in a, in a column. All right, so train, from the train DataFrame, get the name column, count how many values and how many times they appeared. So all of these are going to be 1 because no one has the same name. So there's 891 different names. And how many rows did we have? We check it, 891. So every single row has a different name. All right, what can we do? So every row is a unique name. This is equivalent to the passenger ID, but name could possibly be used differently. Can you think you could reduce the number of different names or create new features out of the names? Now we're not going to go through that at the moment because we want to, we want to get through this EDA fast. Now because there are so many different names to keep this EDA fast, we won't move forward with using the name variable. So we're not going to add it to our subset, but I'll give you a little hint. This is challenge number 3. I'm going to lose count of the number of challenges. See how all of these have Mr., Mr., Mr., Mr., Miss, Mrs., Mrs., Mr. Could you create another column, right, on this data frame which uses the name column and then creates a class, so like p class, that if you have, say you have Mr., you get a number 1, or if you get Miss, you get a number 2. If you have Mrs., you get number 3. Can you do that? All right, so your column might look like name, 1, 2, 4, 5, 3, 2, depending on what, what, what what numbers you give these values. So you've got a master there, you got Miss, Mister. That's a little bit of homework, all right? So we won't use name, but now we're up to sex. Okay, let's check it out. Sex is nice and simple because it was just male and female. We're doing the same thing as we did for survive. We're plotting the count. All right, so we have a fair few more males on there. So that's just under 600, so 575-ish, and count for females is just over 300. And then we have, uh, 0 missing values in this X column, which is great. Um, so since this is already a binary variable, male or female, let's add it straight to our subset data frames. Beautiful. So sex, we can add it straight there. We're going to add it to df_bin as 1 or 0. So how do we do that? We use np.where. So that's going to go through the um, the df_bin. So actually, let's just, let's just check out the train, train sex column. All right, we'll just print the head of that column. So this is going to say df_bin sex column np.where. So df_bin sex, we just added it here, just added it to the, to the, to the df_bin, um, where it equals female. Give it a 1, and where it doesn't, give it a 0. So that means we'll have a column that goes, uh, 0, 1, 1, 1, 0. Okay, let's do that. Boom. And now, now we've added it to our subset data frames. We can use the sex variable and see how it looks like compared to survival. Why? Because they're both binaries. Okay, you can do this with other type of variables, but this is pretty easy to do with, with binaries. And what's a binary? A binary is simply 0 or 1. So we're going to plot the distribution here. Let's check it out. So we've got 2 plots happening here. If it equals 1 in df_bin for survived, let's look at df_bin actually. Okay, so df_bin.head. All right, so this is what we're doing. We're grabbing from df_bin, we're grabbing this column and we're comparing it to this column sex, and we're giving the— we're giving it the plot two little labels here. So this is what this is doing. Now, what does this look like? Oh, and by the way, this is the size of your graph here. So sex is down here. So 0, we have 0. This scale is not that great. You could probably improve that. Okay, that's another challenge. Improve the scale on here. So maybe these columns are a bit closer together. So we have 0, right? What's 0? So 0 for sex. What do we give that? That was males. Wonderful. And then we have 1. So this is females. So males, females. Now we have 2 colors. All right, predicting— showing us, sorry, survived or did not survive. So if we look at this 0, Males survived. Look at the proportion of males that survived. Not very many, right? That's, that's, that's— if you're a male on Titanic, you had a— your chances of survival were far lower because look at this, did not survive is right up here. Okay, and now we got females. So females are pretty close. Some of them still perished. So not, not all of them. More, more did not survive than actually survived. But the ratio is certainly a lot closer. Alright, so if we look at this, this is survived, did not survive, way bigger. And this is the same over here. So these are pretty close. That's, that's really good for the females, not so good for the males. Not many people survived, but of those who did, more were female. Okay, so we've gone through the sex, the sex column. Now we're going to go to age. Now you remember how many missing values did our age column have? 177 way back up here. All right, we have this graph like so many missing values. Oh, now we quickly look at age. So out of a total of 891, that's almost a quarter of the data set. But to make this quicker, where I'm going to give you a big challenge, this is your first big challenge, right? Along with the name column, how would you fill in the missing variables of the age column? Could you replace them with the average? But what's the pros and cons of just replacing the age values with the average? What I mean by that, so you have 177 missing values. What if you just took all the information from this, this one here, these, these black lines, and filled up the white ones with the average of the black black lines. Can you do that? That's pretty easy to do. But, uh, what's wrong with that? Or would you get rid of the, the age rows, the whole row that doesn't have values? Probably not. So we won't answer these questions in our initial EDA because we're trying to iterate over every column as fast as we can to run some models and make some early predictions. Then we can add in age later. But once age has been fixed up, there's some code here that you can add it to the two sub data frames. So this is your first big challenge. See how it's in bold? How would you fill the missing values in the age column? I'll leave that for you to check out. Now we're going to create a little function here. Before we check that, we'll check the chat. Daniel Bourke, it's not good practice to use sex as a label. You should use male or female depending on what is one. Hmm, are you talking about the graph there? Maybe, maybe that's important. Yeah, I'm not, not quite sure what you mean. Oh, okay. Yes, sorry. So it should be male, female. You're right. You're right. Yeah, that should be a better way to graph it. Alright, so we've got a little function here. Not the best, the best code to do. It's just simply a binary that says if, if we have a true value here, use bin_df, else don't use it. This is just for some graphing. You'll be able to check that out. We won't go too far in depth. Essentially, it's just going to create two different types of graphs side by side to each other. Now, how many missing values? We're up to the SibSp column. How many missing values does that have? It has 0. Beautiful. All right. Now, what other value counts are there? Most of them are 0. All right, then we have 1, then we have 28, then we have 18, then we have 16, 7. All right, so Sib or spouse. So there's some people on there with 7, 7 people on there have 8 siblings or spouses on there. Wow. So this whole family's on there probably. But a lot of people are just there with, with no spouse or sibling. That's, that's strange. I would have thought a lot of couples go on Titanic, but, but apparently not the data. Remember, this is the description here, the number of siblings, spouses the passenger has aboard the Titanic. All right. Now let's add sibsp to our subset data frames. We'll do that as usual. You're used to that now. And then we're going to run a visualization here. Now, this is, this is what this function does here, this big messy function. But we're going to run this graph a couple of times. So we don't want to be rewriting the same code over and over again. Wherever you can to save yourself writing the same code over and over again, create a small little function and use that to, to, to run similar, similar snippets of code over and over again. It's good practice to start functionalizing your work. So, oh no, we forgot to define our function. So we'll define it here and then we'll, we'll plot it. So what, what does this graph on the right tell us? So we've got the counts here. We can see that that's, that tapers off, that, that, that mimics this little, little, little graph here. I mean, little table here. What does this one tell us? So this is on the, on the x-axis, we have 0, 2, 4, 6, 8. So these are these values here. And we have survived and did not survive as well. So did not survive. So 0 have a much higher ratio of people who did not survive compared to people who, who did survive. And then here, if you did have a, have one spouse, or one sibling, it seems like more of them survived than didn't. So what does that tell us? Well, maybe if you had a wife on board or if you had— yeah, I think that might be it. Maybe if you had a wife on board, your chances of survival were higher, or if your husband was on board, maybe. I'm not entirely sure, but we'll see. So that's it. That's another good way to visualize your binary variables. Remember, survived is just a binary variable, and start to see the distribution of, of the different column values versus the two values you're trying to predict. So next we're up to Parch. So the description is the number of parents to children the passenger has aboard the Titanic. All right, so since this feature is similar to sibsp, we'll do a similar analysis, right? Because so it's a number of parents or children that passengers aboard the Titanic, and this is the number of sibling spouses the passenger has aboard the Titanic. So we're gonna missing values none, value counts same, same sort of distribution as before. Watch this, same graphs, right? That's why we made a function plot_count_dist. You feed it a DataFrame, you give it a bin column, you give it the label column, which is this one here. You give it a target column, which is P arch, which is down here, and then you tell it how big you want this figure to be. All right, now that is our function up here. All right, that's what you can check out yourself. So what's this? What's the story here? Similar, similar case scenario, right? You got 0 if you If you had zero number, if you have zero parents or children aboard the Titanic, your chances of survive, of dying, did not survive, is higher than your chances of surviving. If you had one or two, your chances of survival is higher than if you, if you, if you didn't, right? Sorry, your chances of survival is higher than not. So you can see that survived, did not survive. All right, so that's some, some information there. We've added them to our— we've added both of these to our subset data frame, seed, seedsp, and parch. Now they're both in our subset data frame. We can have a look at that here. So we've got df_con.head. Beautiful. All right, so we see how we've got, we've got df_train We're slowly working through the DataFrame train. We're slowly working through column by column, adding it to, adding it to a DF, a subset DataFrame to start making some predictions. All right, so we've got a few columns left to go, but ideally this is going to be filled out with similar columns to what's on here, uh, in a similar way. Then we can run some machine learning models on, on our subset DataFrame. All right, because we want to keep train as the ground truth. You want to keep that, uh, as it, as it, as you would download it. All right, because we don't want to alter that because we need to figure out if in case we have to go back to what our original file was. We want to keep that original. That's why we're making subset data frames so we can, uh, or supplementary data frames so we can alter those and not, not destroy our ground truth information here. Now, ticket. How many missing values did ticket have? 0. Beautiful. Now, how many, how many different types of tickets were there? Now, this plot might take a while. The 1, 0 data is perfectly fine. If the data 1 is male, then you should just label that variable male. If you label it sex or gender, we don't know if— oh yes, I see. Yeah, 1 and 0 for male and female. Yeah. Yep, I think that's what we got. Okay, if we go down now, so there's way too— as you can see, this is, this is clogged up. Why? Because there's so many different types of tickets now. Oh, actually, I think we do that here. We don't need to screw that. So that doesn't look too good. What about another way of looking at this? All right, let's, let's see all the value counts of ticket. Wow. Right. So remember our name variable. Okay. Remember our name variable had 891 different types. Ticket has 681 different types. So that's still a lot of, of, and what this is, is, is high, high dimensionality. All right. So that's the curse of high dimensionality. You have, have one variable. So one feature, ticket. But it has 681 different types of things that it could be within it. So the idea of machine learning model is to reduce that, right? So that's what we're trying to reduce it back to. But is there a way you can quickly see that on here? I can't really. Maybe some that were, were, we could group them by if they had CA. There's a few with CA there. We could group them by that. Or if they, if they were over, if they were over $20,000 or maybe if they were over $100,000, I can't. So essentially we're trying to group these into different, into different smaller, a smaller amount of categories than 681. Right. Because that's just too many across 891 rows. That might be a challenge. I think it is a challenge. How would you reduce the ticket feature? Is it even possible? If you can't see a clear way immediately, you could probably skip over it in your initial EDA. So remember, we're just, we're just running over this quickly to try and get a prediction, to get an idea of how well we can, we can build a model first. Okay. So there's 681 unique ticket values. That's too much. We're not going to use the ticket data, data ticket column for now. There may be some way to reduce this down, but I can't see one immediately. You might be able to see one. And that, that is definitely a challenge for anyone who decides to work on this, this notebook themselves. Now we're up to fair. Path Dave, how to get an internship in AI and business? Great question, my friend. The best way I've— that worked for me is to develop some skills. So whether it be machine learning, data science, and be good at those skills and communicate those skills online and then apply rigorously to, to as many internships as you can. There's no structured way to do it, but there are two things you will definitely need. You will definitely need the skills and you will definitely need to be able to communicate those skills. And so they're the two things. You can communicate them through, through what I'm doing. You could, you could build something like this and show, show someone how well you can go through a dataset from scratch. You could enter a Kaggle competition. You can make a video like this. You could do a blog post. But first of all, you need the skills. You need to be able to communicate those, and then you need to apply. You can do that through messaging people on LinkedIn or going through a company's website. But the two things you definitely need are skills and communication skills. Back to this. All the best, Path. So we have a feature fare, no missing values. Beautiful. We're doing a count now. As you can see, A lot of this process is similar, right? So we've done it, missing values, 0. We've done the counts. We've done the data type. We're going to continually do this for each column, right? Because that's what we do. We do high quality work and we leave no stone unturned. All right, you go through a DataFrame iteratively. That's— there's really no other way to do it. If you have a better way, let me know. But of the problems I've worked on, It's just a matter of putting in the time and putting in the work to explore the data as best you can. So there's 248 unique fare values. Now what we're going to do is we're going to, we're going to create our first bucket. But because fare is a float number, let's add it to our continuous sub data frame and add it to our categorical sub data frame. And to add it to our categorical sub data frame, we'll cut it into bins. Now, what does this mean? All right. So if we go here, so we've done a bin, we've, we've pd.cut train fare into 5 different bins. So this is bins equals 5. But as for df_con, we've just added the column. Why would we cut it here? All right. Well, remember, bin is our— oh no, this is This is not discretized. This one is discretized. This is the beauty of doing it live, right? You think you've checked the notebook for errors, but you find some. All right, so that means it's cut into bins. So if we look at, look at df_bin.head, so now we've got different groupings for fare, whereas in df_con We've got fare just exactly how it is. So remember how I was saying we'll have fare zero between negative— that's weird, someone's fare was negative 50 cents. That's— I'm not sure what that value is in there. That might be an outlier you want to remove. So we have fares between zero, 50, negative 50 cents, and $102. Now if we have a look at how many different types there are here, value counts. All right, so we have negative, negative 50 cents to $102 is 838. Now these bins aren't exactly, they might not be the best. So what FARES_BINS does is it just, I'm not actually sure how it decides to cut them into bins because I haven't used that function very often. You might want to look into that a bit more as a little homework project because I'm certainly going to be doing that as well. But as you can see, they go up in about $100 increments. Okay, so $100, $100 to $200, $200 to $300, $400 to $500, $400 to $300, etc. All right. So now we have, we have two subset data frames. We've explored fare. Let's check out a similar graph on, on the fare data on the fare column. So remember, this is that function we defined before. We've called it 3 times. Imagine if we wrote all this code 3 times. It wouldn't look too great, would it? So we go down, we go back to fare. Where is it? Here we are. So now we have did not survive. All right. So, so most of the values are clumped up in here. So again, this is, this is maybe not ideal. This is, but for now, It'll, it'll do. So it seems that the closer it was to $100, the more chance you had of survival. All right. But is that information useful? Maybe, maybe not. We'll keep going. There's a little bit— let's put this as a challenge, actually. How would you change the fare bins? Right? And once you find out, let me know. So we have feature cabin. How many missing values are in cabin? 687. Oh yuck, out of 891 rows. We don't want that. So look, only 147 different value types in, in, uh, well, out of 687 missing values, there's 147 So that means there's what, 891 minus 687. So what's that, 200 and 204, 204. And there's 147 different value types within that 204 values. Not ideal. Since there are too many missing values, we're not going to use KABIN for our missing, our initial models. But how would you modify KABIN to be, to be more usable, right? Could you maybe go A, B, C, or D. So you'd have a column, you'd have a column instead of— let's put it here. That's for your train. So how would you change this cabin? So maybe you could change it to C. So all the people in the C cabin had a value of C, all the people in the D cabin had a value of D. So anyone with D here, D33, D20, or anyone with E or anyone with C. I mean, there's many different ways you could do this, but it's all about trying to explore different ways. All right. We're not going to add cabin to it. We're on to our last feature, I believe. Embarked. Yes. And then we can run some models. All right. Thanks for sticking around. Thanks for— I mean, this is— I'm just trying to show you that it's not all just going to happen instantaneously. The reason why I'm going through this step by step is to to really, really give you an example of what it looks like to explore a dataset from scratch. This is something that I do on a daily basis with, with data. And I mean, there's no perfect way to do this. There's so many different ways. I'm still learning. So of course, don't take this as being the best way to do it all. If you have a better way, I'm always open to feedback or advice or whatnot. But we're on to our last feature, then we'll run some models and then we'll get a submission. All right. How many missing values does Embarked have? 2. Not too bad. Over 891. So 889 total values. So what kind of values are— we don't want that. We don't want— we don't want some. So there's 644 S, 168 C, and 77 Q. What does that mean? So Southampton was the most popular point at where the passenger boarded the Titanic. Queenstown. Does this influence whether you survived or not? Who knows? We'll see. We'll see. We'll run a little count plot over there to just look at this in a different way. These count plots are handy because a lot of people might not understand this sort of stuff. Like if you're doing a presentation, people can quickly interpret that, right? They might not be able to go look at that. I mean, there's only 3 here. So that's quite easy. But definitely if you wanted more people to have a look, more people to be able to understand it quickly, you want to visualize as much as you can. I know that's a big request from many of the clients we deal with. Can you make it visual? Yes, of course we can. We're just, if you don't, you're not communicating properly. How can we deal with the 2 missing values of Embarked? One option is to drop the 2 rows. So remove that row where Embarked is missing completely. So if we go back up to our missing number, so we could just drop these two, get that, get that whole line. So take this, draw it across, drop that, and draw that across and drop that out of the data frame. Do we want to do that? Maybe, maybe not, depends on what it is. But we go back to our, our little, uh, little embarked column. I think we have, yeah. We've removed the values which are missing. What's another way to do that? How can we do it? Another option would be to randomly assign a value with C, Q, or S. How would you do that? How? What are the pros and cons of each? For now, because we're just rushing through this, this EDA, I know it seems it might be going slow, but believe me, if you were to code this whole pipeline, it might go a bit slower. We're just trying to get, get a model out to see, to see what kind of predictions we can, we can try and make. All right, so we've removed the 2 rows, missing values from Embarked. We can now add Embarked to our sub data frames, which we have done. So now we have 2 sub data frames. df_bin.head. Beautiful. We'll view that and we'll do df_cont.head. Now, of course, we're missing some features from the training data set, which is, which is here. We got that. Boom. All right, look at that. It's a lot smaller. But why? Because we've, we've gotten rid of some things. We've got rid of name, we've gotten rid of ticket, we've got rid of cabin. And now, as I said, there's some challenges for you in this notebook to figure out a different way to use the name column, to figure out a different way to use the ticket column, to figure out a different way to use the cabin column. These are all projects that, that are that, that you would have to work on as a machine learning engineer, right? And I've left them as challenges because it's good practice. All right. It's good practice to try and think about those things and, and visualize new ways and then running the rest of the notebook to see how it affects, affects your models. All right. Before we get on to feature encoding, we have in here— hey, Yash, nice to see you, brother. Welcome to the chat. If you have any questions, please ask. Otherwise, we're gonna, we're gonna keep going, pushing on with our exploratory data analysis, eventually leading into a Kaggle submission. All right, feature encoding. We now have our two subset data frames ready. We can encode the features so they're ready to be used with our machine learning models. So we're gonna one-hot encode our binary variable— sorry, our binned variable. So this is our binned data frame here. We're going to one-hot encode that. What does that mean? Now, one-hot encode means giving— creating a new data frame that's going to give things, um, a 0 for what it isn't and a 1 for what it is. Now, example. So we take this first row. Let's have a look here. df_bin.head. One-hot encode. So we have survived 0. It's given it a 0 because it's not, it's not a— it's a 0 here because it's giving— remember, one-hot encoding gives us a 0 for what it, what it isn't and a 1 for what it is. Now we have Pclass, so it should be a 1 for Pclass 3. Boom, it is. Sex should be a 1 for sex 0. Yes. All right. Now what can you think about what the disadvantages are of doing this, right? So we have our data frame here. It's 1, 2, 3, 4, 5, 6, 7, 7 columns. And this one is 28. So what have we done, right? We've got the same, it's just gonna be the same length, but we've just added a whole bunch of columns. Hmm, what could that do? Alright, I'll let you think about that. What's, what's, what's the disadvantage there? Now for the continuous DataFrame, we're going to label encode all of the values using LabelEncoder. So this is an sklearn, right, function. We imported that right up the top. So if we go label right there, so there we go, from sklearn.preprocessing import OneHotEncoder, LabelEncoder. Alright. So we go, we'll find the next incident. There we go. That's what we want. All right, now we look at that. So what, what has that done? What is this label encoder done? So we have df_con. Okay, so now it's changed all of these into different labels. So we have sex has been turned into 1s and 0s automatically. Parch and civsp hasn't been changed. Fare's been changed. Hmm, do we need to do that? Maybe, maybe not. We'll see how the models go. But usually with, with these kind of values, fare, the numeric, you don't usually have to change them. But we'll, we'll leave this for now. Um, it's just what it's done is it's given, uh, every, every fare that's $7.25 will have the code 18. Okay. And then same for, for, for every fare that's $7.92 and 92.5 cents. I don't know how, how that happened, how someone got charged 92.5 cents. That'll get a number of 41. And then embarked has simply been changed to 0, 1, or 2. So maybe if we blow this out to head, and then we have 20. Boom, 0, 1, 2. So anything with an S has been given a 2, anything with a C is 0, and anything with a— I can't see that one. So maybe— and anything with a Q is a 1. Alright, so that's just replaced our variables with, with numbers. Because what's the goal of getting— before we run machine learning models, it's to Bum bum bum! It's to turn all of our values into numbers, right? Because what do computers work best with? They work with numbers best. So we can't necessarily feed it— where is it— information like this, like name. It's not going to understand that. So we need to turn it something into this, right? So now what our models can do is search through this search through— actually, it's going to search through these columns, missing out the survived, and figure out what pattern in these numbers gives a 0, gives a 1, gives a 1, gives a 1, gives a 0. All right, can it work out a pattern in that? We'll see. If you were to guess if someone survived or not, what do you think you would get? Think about that. If you were to guess just by looking at this, Guess 0 or 1. How accurate do you think you could get if you did it across the whole data frame? Think about a coin, heads or tails. All right, we won't answer that yet. So we're going to start building some machine learning models. Yes! All right, so first of all, we're going to separate the data into train and test. I mean, sorry, labels and data. Now what, what does that mean? So the selected data frame is this one here, the one we've encoded. We're going to use the continuous data frame for this, for this EDA. You might want to use the bin, bin data frame in your EDA, but for the sake of simplicity, we're going to use the encoded version of the continuous data frame subset, which is, which is this bad boy here. So select data frame, so we don't change this, we're going to change it, put it into another, another variable called select dataframe. So just split this into data and labels. So x_train, so our x variables, because this is a machine learning model, you're going to have a whole bunch of x variables and you're going to try and predict the y. Alright, so x, x_train equals selected_df.drop survived axis equals 1. So this is our data. So what is this doing? It's taking the selected dataframe. Okay. So selected_df.head, it's taking this DataFrame, dropping that column, and just using these variables. So these are our X variables. So let's have a look at X_train. Run this. All right, so see how this— we've dropped the Survive column, and now we're going to make our Y. So this is, this is the labels of our data. DataFrame. So this is the just 0 or 1. So it's practically this, just this column, y_train equals df selected_df survived. So it's this column. So now we're going to use this and these to build our model. So this is the data and this is the labels. All right, that's what we've done. We've just, just, uh, gone through each feature, turned it into a set of data and a set of labels. Now we can run some machine learning on it. So we've built a little function here because we're going to be running multiple different machine learning algorithms called fit_ml algo. Now this is— it takes an algorithm, a type of algorithm. It takes a set of data. It takes some labels. It takes a number of times you want to cross-validate, right? So cross-validate is important. We'll get on to that in a second. It's going to do— it's going to create the model, then it's going to fit fit the algorithm to the training data and the labels. It's going to give us an accuracy, accuracy score. It's going to perform cross-validation automatically, and it's going to give us a cross-validation accuracy metric, accuracy cross-validation. And then it's going to return training predictions, the accuracy, and the cross-validation accuracy. So we're going to start off with logistic regression, then we're going to move through through a bunch here. Okay, now why would you run so many different, different types? Well, you're trying to figure out which model is best suited to the data. So you're going to try and see, run a whole bunch of models. And it's not hard to run these. Look how quick it runs, right? Oh, we didn't define our function. I say this now and it takes a little bit while, a little while. There we go, 4.5 seconds. Not that, not that long. Doing 10-fold cross-validation. So 78% accuracy on cross-fold validation. That's beautiful. Now we've tracked the time here by creating start time. We've done our function here, fit_ml algo. And then we're logging the time. That's just there. And then we've got just some print statements for the metrics we've returned here. So then we've got accuracy 79. That's just on one pass. Accuracy of cross-validation on 78.5%. Now, why would you want to do cross-validation? We'll get to that. We'll get to that in a minute. We're going to run all the other algorithms first. So k-nearest neighbors. Now, these are what— actually, what are these? What algorithms are we running? So we go up into sklearn, and if you, if you look here, we've imported a bunch of different algorithms here, okay? Now these are some of the most common machine learning algorithms that you'll see. All right. Now if you Google most common machine learning algorithms, these will probably come up: linear support vector classifiers, gradient boosting classifiers, k-nearest neighbor classifiers. Now we're not going to go through what each of them does. All right. Now, because of our, our need for speed with our EDA, we're just going to run them, see which one's the best, and then we'll choose the best one. and then you can improve that from there. All right. So we've run our logistic regression. By the way, if you want to find out which algorithms and what they do, I would highly suggest looking at the sklearn documentation. So just Google sklearn machine learning algorithms and you'll find a bunch of stuff there. So we have here, look how quick that ran, under a second, boom. 70, 73.68% accurate. Again, under a second because there's not much data, right? There's only 891 rows and these, these are great algorithms. Computers are fast at doing math. Now we have support linear support vector machines under a second again. Stochastic gradient descent under a second. What's it? A 92% accuracy. Oh, but it's fallen down to a 79% accuracy in the cross-validation. So stochastic gradient descent didn't do too well. Gradient Boosted Trees, 81.1 on Tenfold. That's the best so far. All right. So these are some of your most common algorithms, but I want to introduce you to, to one of my favorite, one I've discovered recently, and it does an amazing job, right? CatBoost, the CatBoost algorithm. So it's by a Russian company, Yandex, who created it as their in-house algorithm, but now they've open sourced it. It's It's similar to— well, it's similar to— I'll just read you the line. It's a state-of-the-art open-source gradient boosting on decision trees library. So this one was a gradient boosted trees algorithm and that got 81%, right? So this is a state-of-the-art gradient boosted on decision trees. Well, maybe this is— this is— CatBoost is a combination of these two. So I wonder how it's going to go. Okay, so it's a decision tree and and a gradient boosted tree combined. All right, it's simple and easy to use and is now one of— regularly one of my regular— one of my go-to algorithms for any kind of machine learning task. If you want to check out more, I encourage you to check out the CatBoost docs. You can find that at catboost.ai. All right, but we're not going to dive too much into the algorithm because we just want to try and run it. So this one's going to have its own special kind of implementation because it— the fit_ml algo function won't, won't work on it. The There's a website with CatBoost. Check that out. We have here, we'll view the training data one more time. We'll view the training labels. Excellent. Now CatBoost has a little section where you define the categorical features. Now that, that might be what CatBoost stands for, categorical boosting. I'm not entirely sure. What do we have here? I think you need test data for testing your algorithms. Yep. Yes, you're completely right. We're going to test our data. We do have a test dataset. So we have a— we write back up here. So we have a training and a test dataset. This notebook is actually bigger than I thought. So we have training. This is our training data here. And then we have test test data. So we're going to train on this and then we're going to test it on that, which is what our submission will be. Our submission will be our predictions on the test dataset. And yes, Zach, the video will be up to see later. It will be live on my channel once we finished here. All right, let's go back to CatBoost. Okay, so CatBoost has a really cool feature where you can tell it what categorical features you have of your dataset. Let's go right back down to CatBoost. I should probably just search CatBoost anyway. We're here. Excuse me. Okay, so categorical features for the CatBoost model. In the CatBoost documents, it says feed CatBoost the categorical features you have, right? So what's a categorical feature? Alright, so these aren't— none of these are categorical features. But a categorical feature might be, might be something that's non-numerical. Okay. So let's think about it. So say you had cars, cars, a category might be the different type of make of car. So you could have a category for Toyota, a category for Ford, a category for Mercedes, something like that. Okay. So all of our data is numerical. So if we run this little function here that says catboost equals np.where x_train, so this is, this is our x_train here, does not equal np.float. So what's a float? So float is, what is a float? So if we have a look here, so if we have, actually, you know what a float is. Let's just, let's just, let's just put it there. This is not to tell you what type of variables are. So a float is a number like that, right? So it's got decimals. And an integer is probably something like this. Okay. It's just a 6. So int, no decimals, float, decimals. Okay. So none of ours have decimals, right? 'Cause they're all integers. So we don't have— CatBoost is seeing these as all categorical features. So what's it gonna do? CatBoost has its own methodology of encoding these. So I'm not going to dive too much into that. Essentially what it does is CatBoost— all values— sorry, all values except fare. What's happening? I'm doing this wrong. So this is the beauty of doing it live. Let's see. There we go. Okay. So 0, 1, 2, 3, 4, 5. All right. All the categorical features. So what CatBoost is going to do is when you when you create your training data, all right, you tell it what the categorical features are. Okay. So these are going to treat them all as categories, right? Because we've encoded them all into categories. And then CatBoost is going to figure out its own way of best dealing with those categories. So remember, like up here, we one-hotted our data in dataset, right? Now that might not be the best way. So CatBoost has its own way way, and you can read about more in the docs, of one-hotting a DataFrame, but in a different method. I'm using— don't even think about one-hotting because CatBoost does it in a different way. Okay, just think of it as you tell CatBoost what your training data is, what your labels are, and what the categorical features are of your training dataset, and it will create a training pool, which means CatBoost has its own way of manipulating this data into a way that its algorithm can best explore it. Okay, that's, that's how I understand it. There is some, some, some math you can find in the docs about how it actually does that. But for the sake of this, of this video, we're just going to leave it at that. CatBoost figures out based on the information you feed it, the best way for its models to run over your data. And let's, let's create the training pool. And now we're going to fit our CatBoost model. So we have CatBoost model equals CatBoostClassifier. So classifier, what does that mean? So classifier means zero— well, it's going to classify something. Now in this case, CatBoostClassifier means it's, it's going to It's, it's 0 or 1 classification. Now that is our, that is our goal. Do we want 0 or 1? Yes, for Survived. All right, so it's going to classify the training data, the training data into 0 or 1. That's what we want. And we're going to give it a loss function of accuracy. So we want to figure out how accurate this CatBoost model is, and then log The loss function it uses is LogLoss. All right, we won't dive too much into that, but that's the default that comes with it. You could actually comment this line out and it will still run, but we'll leave that there so you can see what it does. And then, so we've created our CatBoost classifier model, and then we're going to fit it by calling CatBoostModel.fit on the training pool that we just created. And we're going to plot so we can see what happens. And then we're going to save out the accuracy much in the same way as we have for our previous machine learning algorithms. So round_machine.model.score, X_train, Y_train, same, same way we've done that for CatBoost. Okay, let's go down here. We're going to run, we're going to watch this beautiful graph. This is what you can see with CatBoost. Check that out. You can actually watch the accuracy climbing. So this is what this is doing is running over our training dataset. So going through this and figuring out which label it should have, right? And this is, this is how accurate you can see it in lifetime. CatBoost is beautiful. I actually love CatBoost a lot. So, um, we'll see how this goes. Going to take a little bit while to run. Now we're going to run CatBoost cross-validation after. After this is done, because this is just fit, remember? Right. So, so in our fit_ml algo function, we did fit and then we did cross-validation. So up here, while CatBoost is running, we go up here, fit_ml algo. Boom. So we did fit, algo.fit, and then we did cross-validation. So we need to do the same for CatBoost. So we're doing CatBoost.fit down here and then running cross-validation. All right. So CatBoost.fit and then running cross-validation down here. So CatBoost has a function built in called cross-validation. It's just cv. We imported that right at the top. There we go. From CatBoost import CatBoostClassifier. That's the classifier model that's running right now. Pool. That's how we created the training pool data and cross-validation from CatBoost, which you'll be able to see in a second. So now that's running. Apparently it's going to take a little while. Hopefully it goes a bit quicker than that. But in the meantime, while we're waiting for that, we'll check out the chat. Oh, sorry, just joined the stream. Any clips from Silicon Valley? Alok, how are you? I haven't actually divided into a training dataset. So, so the, the Kaggle competition gives a training and test set. So what we're doing is we're going to model the training data. And predict on the test data. So we've already got a training and test set from Kaggle. Usually, if we didn't have a train test set already, we'd, we'd split, we'd split it. Yep, you're right. 80% training and 20%, 20% test, or a little different depending on your problem. So CatBoost is running cross-validation. It's going to take a little bit longer than our, than our previous algorithms. Because of what it does. And now I actually can't— couldn't explain to you what it does on the inside. But just in my experience of using it multiple times for multiple projects, it does take a little bit to run because I think, I think it's doing a combination of different types of models. So it's not just doing one, one, one single way of, of fitting on the data. It's doing, it's doing multiple different ways. So it's doing a decision tree classifier and a gradient boosted tree all rolled into one. So we're fitting that here. Now, why would you want to perform cross-validation? All right. So I'm going to explain cross-validation, how I understand it. I would, I would look into it, into it yourself as another piece of homework. But if you imagine, if you imagine, let's, let's do this. Let's use a piece of paper. While CatBoost is running, we've got a little countdown here. So we've got some time to play with. Imagine you have your data here, right? Just this one simple piece of paper, right? And you do one— your model goes down and it fits all the rows in one hit. Okay, it might have done a really good job, right? Because some of what it does is randomly initialized. What does that mean? So it's used— it could use random random numbers, it might have done a really good guess, right? So if you actually say you had this piece of paper and you had to guess 0 or 1 for each, each different line, say you went down your first try, your first shot, you did really well, right? You got, you got 80% accuracy or 85% accuracy. You got all the, all the 0s and 1s pretty much spot on. but then you reshuffle, you shuffle all the lines. So you move the piece of paper around and then you guess again and you only got 60% accuracy, right? That's essentially what cross-validation is doing. Instead of just running one, one time over the data, it's, it's running multiple times over different versions of the data. So it goes once. All right, boom, we got 80%. Mix the data around again. We got 65%. Mix the data around, you got 79%. And then we're going to do that, do that 10 different times to really get a good idea of how well our model is actually doing on 10 different types of the same data. Not types, but 10 different ways of looking at the same data. That's how I understand cross-validation. And why, why is that important? Well, as I said, look at this. This accuracy without cross-validation is way higher than that with cross-validation. So see how we've done 10-fold? That's, that's going over the data in 10, 10 different ways. So this way, that way, do it again that way, right? All right, not exactly how this piece of paper is looking, but just imagine reshuffling the rows in different orders. So this is a lot higher. Look at this one, 92%. So it got really lucky on that one hit. And then on the cross-validation, not as high, right? Because it's cross-validation, a much more robust metric because you know it's gone over the data in several different shapes and sizes. All right, how's our cross-validation CatBoost model going? Alright, we still got a little bit of time. How we have here? Hey, just arrived. Don't know if you spoke about that already. When trying to build a model, how do you pick your features? Great question. Let's get into that. Speed, speed, trend, maga. Alright, so how do you pick your model, your features? We're going to answer this question while our CatBoost cross-validation model is running. So let's look at our initial training data frame. We have some features. Thank you for the question. We go up here. We have— this is our training data frame. So if you imagine each column is a feature, so we have survived, that's our target feature, Pclass, name, sex, age. Now for this one, for this training data frame, there aren't that many columns, right? So there's only, let's say, 10. Doesn't really matter, might be whatever. So how do you pick which ones to use? In this case, we've decided to use as many as we can because there isn't that many to go over. But what happens if you had this and it just kept going and kept going off the page? You've got maybe, maybe 1,000, 1,000 different columns. That, that could be hard. That could be hard to go through them one by one. All right. So what you would probably do is go through them, do something like this, do something like this, figure out where all the missing values are, get rid of all the ones with missing values and just, just run it on, on your, on your best ones. Or you go through it step by step. You go through each, each feature individually, work out which one's the best. Or so, so there's— we got a few different ways here. So this is, this is going through each feature step by step. The other option— where are we going? We've got to go right down to the bottom. There's two more, two more options that really stand out to me. So we've run some models, we made some predictions. I'm jumping a little bit ahead of where we actually are up to. So our CatBoost model is still running. Look how beautiful this graph is, by the way. Look at that. CatBoost is awesome. It shows you these plots. Maybe if, if I didn't have this plot on, it would probably run faster. So how do you choose your features? Back to that. So we'll see how it is. We're going to get to that in a second, actually. Feature importance. So which features of the model were most important in making predictions? There's another way that you could do it that I've set in the challenges, which is called PCA. Principal Component Analysis. That'll be in the extensions. It'll also be in these resources that I've listed down here. Daniel Furoso, awesome name. He's done some epic work on how you can do this notebook on a bigger dataset. This one I built is actually based on his, his notebook. So all that'll be linked in the description as well as in the GitHub version of this notebook. So that's PCA, feature importance, iteratively going through each feature one by one. Or another big one is talking to a subject matter expert. Now, what does that mean? Now, say you had a project, right? So you had a project and you want to predict the prices of houses and you have a dataset like, like we do. You have, you have the dataset like we do. You have, you have, let's say, 1,000 different columns, right? And you— all of them go from number of bathrooms to suburb to what color the floor is and stuff like that. To work out which features of those columns you might want to use, you might talk to a real estate agent and say, hey, I'm trying to do some data analysis on this. But why? Why would you talk to the real estate agent? Because they sell houses. They know the value of what contributes to different house prices, or they at least have some insight. The data might be able to tell us more. But to get started, they might be able to say, okay, you really want to pay attention to the number of bedrooms. You want to pay attention to the, the suburb. You want to pay attention to how close it is to schools, because these are the things that my customers really ask for. You might say, okay, I'll start with those. You start with those and you build a pretty good model, but you want to make it better. And that's when you can start to go through each of the features and start adding more and seeing if it improves your model. So step by step, you can do feature importance once you've made some predictions. We'll show you that in a second. Or you can go through, see where the missing values are and just drop all the ones that have missing values because you just want to work through it fast. Or talk to a subject matter expert to get their advice. And now there's more ways, but these are just the ones I can think of. All right. There's some of the ways that we literally do day to day at Max Kelsen. So for some of the projects, we'll talk to the client like, hey, you've got all this data, but just some information. What is most important when you, you, the expert, the people who've been there for 20 years? Because as a data scientist, I may come along and I've never seen this problem before, but you have someone who's been doing it for 20 years, maybe I had to tell you straight away, oh, don't worry about the color of the floor. No one cares about that. All right. Of course, the data will tell you more once you figure that out. But that might be a way to eliminate the amount of features you have to deal with. Otherwise, there's this PCA, different ways of analyzing features with code, but we won't go through that in this notebook. That'll be in the, in the extra resources down below. If you want to look it up, just Google SKLearn classification notebook by Daniel Bourke. Alright, our CatBoost model has run. Beautiful. Look at that. Look at that, all that green, and you can run over it and see where it's most accurate and stuff like that. Let's do this. Alright, CatBoost metrics. As I said, it's a higher accuracy. Alright, but we did some cross-fold validation and it's lower. So that first guess it ran through the, our piece of paper for it, ran through, it got really good results. But then we decided to go, you know what, I don't trust that first result. I'm going to run it multiple times. And it averaged out to 81.2% and it took 11 minutes. So a bit— a lot longer than what our other models took. So this is where we can just make a little table. So we've just gotten the accuracy scores from all of them. So look at that decision tree right up there, 92.46%. But as I said, we're not going to pay attention to these regular accuracy scores. We care more about the cross-validation accuracy scores. And there we go. Our baby boy CatBoost is right up the top. I seriously love CatBoost. I'm not partnered with the index or anything. I just, I just love their, their algorithm because it's so easy to implement and it gets incredible results. So CatBoost got the best cross-validation accuracy score. So we're going to use that. And now it got close to 82%. Now, if you're trying to guess 0 or 1, what do you think is a baseline? Maybe 50%, right? Because heads or tails. So if we were to guess who survived and who didn't, we could probably get around 50%, maybe a little bit less or more because the dataset is not evenly, not evenly spread. It's not 50 survived, 50% survived, 50% did not survive. So 82% is a pretty good score in my opinion, just for this exploratory data analysis. And remember, there's always a challenge. How could you improve the CatBoost model? So that will involve you checking out the CatBoost docs. I've put a little tutorial down here on CatBoost by the authors of CatBoost themselves. You can check that out. It's really great. They work on the Titanic dataset as well. So we're going to run feature importance. That's what I was talking about before. This is just a little function to create a feature importance graph by, by sucking out the, the feature importances parameter of the CatBoost model. And then we go here. So if we look at this, this is, this is how this, this axis here is how much it contributed to the model. And this is our feature, what, what contributed. So we have— this is in reverse order. So sex is number 1. So male or female is really important to, to whether you would have survived on the Titanic according to our model. So sex is number 1, P-class is next. So does this make sense, right? Have you seen Titanic? Have you seen the movie? This is where you can ask yourself, is my model talking bullshit or talking real? Is the data showing us the right thing? Right. And you can, you can use your own intuition here to kind of spot check to whether your results are okay. So P-class, what was that? That was the class. So that was first class, second class, third class. So you— it makes sense. Maybe people Actually, let's go through sex. So remember in Titanic the movie, women and children had priority. That makes sense. And a lot of in our analysis, right back up in sex, the feature, more males. If you're a male, your chance of survival was way less. Look at that. Survived, not many. Did not survive, lots. Remember, zero is male. Alright. So that makes sense that sex is a big contributor. And then P class is if you were in the first class, maybe you get an advantage because you're, you've paid more for your ticket and you're a higher class person or a citizen. Maybe they see you as more important. I'm not entirely sure. Fare is there as well. P-arch. All right. How many parents you had on there. So if you're a parent, you, you that it valued more. These are all things that you can do to spot check your model, right? And what you may do is— why would you, why would you want this feature importance? Well, what you could do for your next version of running a model is maybe remove this bottom one. All right. Ideally, we want to add a few more because we missed out on the age and the name. Remember right up here, we didn't use the name and the age. But say you did use the name and the age and you find out that SibSp really doesn't give you much. It's under 10, right? So, well, in this case, 10 is pretty significant. You might be able to remove the bottom feature and run your model again and see if it gets better results. All right, but in our case, in this smaller data set, you could probably use more, more the more features, the better, because there really is only 10. All right. But if you had, if you had 100 of these and you have the bottom 50 are giving you a little sliver, so absolutely nothing, you might want to get rid of those and rerun your model. Okay. Another option is trying to improve your best features. You can't really improve sex because it's just 0 or 1. You can't really improve P class. There aren't many here that you could really change because as I said, these are already ideal parameters. But you might want to, say you did add name here as it was without altering it and it just didn't give you anything, you could change it to how we discussed before by reducing it to like Mr., Mrs., Miss, Dr. and it might boost it up, might give you a bit more value. All right, but that's something you'll have to explore yourself. But yeah, what feature importance does is it gives you an insight into what information is most important when making a prediction. Okay, and you can use it to improve the number of the features you're using or use it to remove the number of features you're using. Okay. Precision and recall. I'm going to mention that here. We won't go through it in entirety. I have linked down below a great article. Actually, no, it's up here on what precision and recall is. So when would you use precision and recall? So we've used accuracy as our, as our measurement here, right? So maybe accuracy is not the best metric. right? Because if you had 100,000 people and only one of them gets a certain type of disease— this is another example, not Titanic— and you, and your model predicts that every single person doesn't have that disease because only one person gets that disease, your model is 99.999% accurate. So that's a really accurate model, but it's not that helpful, right? Because your model, all it does is just predicts that everyone doesn't get this disease, but there's 1 person in 100,000 that gets it. So this is where precision and recall come in. Now we can run precision and recall on our CatBoost model pretty easily. I'll let you read up about precision and recall and why that's a challenge, actually. What are some ways to visualize precision recall, right? You may want to look into what a confusion matrix is. Now remember, here's where you would use precision and recall instead of accuracy. If you had cases where you have an imbalanced classification problem, right? So ours is pretty balanced. We have, although there are more zeros than ones, so did not survive versus did survive, they are pretty balanced. So they're not like, they're not like this. They're pretty close, right? But if you had a problem like this where you have 100,000 people and only one of them, only one of them has a disease, accuracy is not your best measure, right? So if you go into your own problem and you have, you have an imbalanced class, so say you have 100,000 examples and you're trying to predict 10, which 10 of those 100,000 are the ones you're looking for, right? So you have 100,000, you have 99,990 zeros and 10 ones. You want to find the 10 ones, you might not want to use accuracy. You might want to use precision and recall, right? So look into precision and recall. There's a little link here that you can check it out. Otherwise Google Beyond Accuracy Precision and Recall, really great article. We won't dive into the specifics of precision and recall in this notebook. I just want to leave it there. So you think about it in the future, You're not just using accuracy for every single, uh, every single one of your models. So our model got pretty good precision, right? So close to, close to 1 there. Recall not as great. Um, F1 combined, so, so these metrics are going to be between 0 and 1. Closer to 1 is good. I won't go too deep into it. So our recall is low, so this means there's a higher amount of false negatives. So recall is the ratio of true positives over true positives plus false negatives. So that means our model has a higher amount of false negatives predicting did not survive when the person actually did survive. And our precision is, is higher. So that means there's less false positives. And precision is a ratio of true positives over true positives plus false positives. So that means a false positive is predicting survived when it actually did not survive, right? So our model's recall is not as great as its precision. F1 is a combination of precision and recall, and area under a curve is another, another way of checking out the ratio between false positives, true false positives, and true false I'm getting it all mixed up. Best to go check out that article. Much better job at explaining it than what I do. Remember, I'm still learning these things as well. I'll try and get better and teach them as I get better. All right, we're ready to make a prediction on the test dataset. Finally! You're like, Daniel, I've sat through this, this video for 2 hours, whatever, however long it is, and just show me how to make a submission to Kaggle. All right, well, we're going to do that right now. So We're creating a variable wanted_test_columns because let's have a look here. Why do we want to do that? Our model was built off certain amount of columns. It was built off these columns. However, our test dataset has not the same columns. So our model, if we tried to run it on this right now, it would not work. We need to have our test dataframe in the same version as our selected DataFrame actually. Oh no, it'll be the same as X_train because we want to predict, uh, the survived column, so 0 or 1. So we need to convert test into something that looks like this, and it's going to be a lot easier because now we have our, our wanted test columns, right? So we're going to— these are the only columns that we want, so df_con So we use df.columns. So let's, let's actually change this to selected_df because— or actually we can just change this to x_train.columns. So we want the same columns as x_train. Much easier, right? There we go. So this is just a list of columns that are in here. And then we're going to make the predictions by choosing catboost_model.predict. Remember, we saved our CatBoost model out up here. Somewhere. CatBoost model. So yep, CatBoost model equals this, and then we use it to fit. Boom. We're going down here. Now we're going to use CatBoost model to predict on the test data frame. So test, we're choosing the wanted columns from that test data frame, and then we're going to apply apply— remember that label encoder fit transform that we did for the encoding df, df_con— right back up here. Same, same thing we're doing because we want to predict on data that is in the same format as our model trained on, because the model is not going to be able to predict on data that's not in the same format. So we're just applying this one here. So what we're saying is get me that test data frame only the columns that I trained the model on and run this function on them. So that's what we're going to do. We're going to do that. We're going to make predictions doing that. We'll go down here. Boom! Look how quick that is. Our model just ran. Now what does it look like? So it's just a bunch of zeros and ones. Why? Because that's what we wanted our Survive column to be. So our model has just output the same type of data as it was trained to predict, right? So zeros and ones, that's all it is, zeros and ones. Now to create a DataFrame, what does it have to look— has to look like this. So we need passenger ID and survived. So we need this, this from the test column. So passenger ID and whether they survived or not. Based on our model's prediction. So let's do that. Create that. Oh, look at that. But do they look the same? What's wrong? What's wrong with this one? Right? What's wrong with these numbers compared to that? Remember we talked about floats and ints? Has to be the exact same format. Otherwise, I ran into this issue before. Otherwise the Kaggle submission won't take it. So although it's still zeros and ones, we need to convert them into integers. So this is what we can do. We can do submission_df. So this is this one. This is our submission_df. And we need to turn the survived column into integers. So we're just going to go as_type_int. That's a function built into pandas. Boom, converted survived column into integers. Let's have one last look at that before we, before we get it up there. submission.head. Wonderful. Now, what is this, by the way? This is our example, right? Remember we imported an example right up the top? This is what it looks like. Do they look the same? Yes, I think they do, except for these values are a bit different here. But remember, this is not, uh, this is not, um, not the true reflection of what the label should be. Okay, are our test and submission data frames the same length? Why? Because if they're not the same length Kaggle won't submit it. Kaggle has to be the exact same length as the test data frame, so we have to use every single row. And now to submit to Kaggle, what do we need? We're going to submit this. This is exciting. Submit predictions. So upload files. Your file should be in a CSV format, the same way we downloaded them. So we can do that. Right, we can do that. Uh, let's check it out. Boom. Oh no, boom. Submission CSV is ready. So what have we done? We've just taken our submission data frame, said to CSV, we've chosen where we're going to put it, right? And we, we, we, we're not including the index and that, so we don't, we don't care about this, right? Because that's just, that's just how Kaggle likes it. So our submission CSV is ready. Let's do a quick check on that just to make sure what it looks like. How do we check that? We're just going to do the same thing. Submission check, reading our CSV file, and then we got it up here. Now it's time to submit your Kaggle.csv. Are you ready? Are you ready? Look what we've got here. We've got— we go into our Titanic folder. We've got our notebook in here. That's it there. Kaggle Titanic data analysis. This is our submission file right here. How are you? Everyone to the chat. What did you do right out of Andrew Ng's Coursera course? You can see, you can see that in the AI master's degree curriculum. How long does it take to get started, sir? Get started with what? It depends. It takes as long as you need. All right, let's submit our Kaggle CSV, our predictions submission.csv file to Kaggle. Where we going? We're going to Chrome. We're gonna upload a file. We go into desktop, Titanic. Oh, we don't want to upload that. Submit. All right, now this is live. Hopefully it works. What do you reckon we'll get? Uploading file complete. Now, how do we How do we check our score? My submissions, 2 submissions. All right, it may, may have a little bit of a wait time. Oh wait, did that go up? Catboost_submissions.csv. We're trying this live. Boom, we got a tick. Excellent. Bites fingernails. Hey, Ayush, how are you? After Andrew Ng's Coursera course, which one, the deeplearning.ai or the machine learning? We're waiting for our Kaggle submission to be processed. It takes a little bit. Where is this? Well, as you can see, my submission 6 days ago was, was, was successful, but I'm not sure where, why this one hasn't been. Maybe I have to refresh the page. Well, let's just take that 70, 76% accuracy. Thank you very much, Kushik. So what have we done here? Let's go back. We have gone through step by step, downloaded a dataset from Kaggle, done a full exploratory data analysis. So we've got the dataset here, done a full exploratory data analysis, built several machine learning models, used our best machine learning model to predict whether people would not— would have survived on the Titanic. Then we've, we've submitted our DataFrame. We've output our predictions, right? And then we've submitted them to Kaggle, right? All in the space of however long this video has gone for. But as I said before, this is not an exhaustive list of the things that you could do, right? There are— I've listed some possible extensions that you could do of this notebook if you want to go through this one yourself. Things like, what could you do with the age feature? What can you do with the name feature? How about the cabin? Can you combine SibSp and Parch features to see if the person was alone or not? Right? Can you do a PCA? That was the person's question before. How do you choose what features to use? Or is there a way for you to improve the CatBoost model? I would suggest looking at the Hyperopt library. And then, There's a bunch of references that I use to create this notebook. sklearn classification notebook by Daniel Bourke. This is what this one was inspired by. Really, you should probably check out his. It's a high quality of work. I just decided to make my own version and go through it for you. His goes a bit more in depth on a lot of different things. So, so be sure to check that out. There's a great blog post on how to encode categorical features in Python. By Practical Python Business. There's a CatBoost tutorial on GitHub. Now that's, that's the challenge. That's how you could complete this one. The CatBoost Python tutorial on GitHub is great, all linked in there. And then SHAP library is another alternative to viewing feature importance, which is this one here. But as I said, this— all the code will be available on GitHub. There will be a blog post that will be about a 5 to 7 minute read of, of talking through what we've talked about in this video. I'll link it all below. And last but not least, if you do want to sign up to, um, Coursera did sponsor this video. So if you do want to sign up to a computer science, uh, specialization, look at all this, right? Mobile web development. They're currently offering 1 month free up to December 11th for anyone who wants to sign up to, to any of these courses, I've left a link in the description. You can check that out. All other relative links will be in the description. As always, if you have any questions, if you find any errors in this notebook, if you're having trouble, tweet me, send me a message. My email, you can always email me, is daniel@mrdbourke.com. Otherwise, leave a comment below. If you'd like to see a similar video to this in the future, How can I improve? I'd love to know. But most of all, thank you so much for joining in and all the best exploring your future datasets. But most of all, keep learning. And what do we have here? Hey man, can you give me best learning machine learning? Ricardo, the ones you use, are using right now. Ricardo, my friend, great, great question. There are some great references in here. Send me an email, daniel@mrdbourke.com, Ricardo. I'll be sure to send you a link to everything that I've been using in the past few days. But thank you so much. We're going to end the stream now. We're going to figure out how to do that. As always, keep learning.