Pandas in Python for Data Analysis
Learn pandas basics for data analysis: create Series and DataFrames, explore data with head() and describe(), clean missing values, combine datasets with merge and concat, and filter rows with boolean conditions. The tutorial covers installing pandas, working with CSV files, and deciding when to drop columns or rows.
Chapters
Hello everyone. Am I audible to you?
Yes.
Okay, so in the last session we discussed about NumPy. Right now, in this video, we are going to discuss— in this session, basically, we are going to discuss about Pandas library, which is must for data analysis. Okay, before we begin, if you have any questions to ask, then you can And go ahead. All right, so let's begin. So again, if you wanted to use pandas library, you have to complete the installation. So pandas is basically powerful open-source data analysis and manipulation library. When it says data analysis and data manipulation, is you can make analysis on the data that you have selected, and if you wanted to still manipulate your data or you wanted to make some changes in the data, maybe deleting some stuff or adding new content in it, everything is possible using Pandas. Pandas works on two data structures. First is Series, second is DataFrame. To work with the data, you have to create any one of them. It depends on which type of data you are selecting. If suppose you have a data which is in the 1D format, it means directly a single 1D array, then you have to go with the data structure Series. When you are dealing with rows, columns, or you have complete Excel sheet, you have complete CSV sheets, then you have to go with the DataFrame. Every time you have to create a DataFrame with that particular data. Okay, so two data structures are important. Both of them are important. It is not like if you like moving onwards you will be working on Excel sheet, CSV files, so you need not to learn Series. No, it is not like that. You have to learn Series because Sometimes for testing purpose or for training purpose, we have to create 1D arrays. We have to generate some samples and then we have to operate on those samples to make our analysis more effective. So now let's begin with the pandas one. Okay, so I'm just creating day.ipynb. Pip install pandas because we wanted to use pandas library. Now once this library is installed, you have to import this library. Again, if you wanted to import the library, you have to— you can also give it a nickname. So import pandas as, let's say, pd, which is the short form pandas. Okay, now let's talk about the first core data structure of pandas, which is Series. Series is one-dimensional array. It can hold any data type, and you can also assign some indexes. So let's say s is equal to pd.Series, okay. Now over here I wanted to create a series of these numbers. When I'm creating a series of numbers, I'm writing it within the square bracket because there are more parameters that can be added in this same line. So you can see, first is the data that I am providing, second is the indexes you can assign if there is anything none or hashable, or you wanted to assign data types. More features you can keep on adding, more parameters you can keep on passing, but those are optional ones, so it is not mandatory. That you add them. So let's suppose if I'm just going to write this much and I will just print the s variable. So this is our 1D array where we have these numbers, and along with the numbers, these are the indexes. 0, 1, 2, 3 are the indexes. If you wanted to change the indexes and you want the indexes of your own type, then what you can do is you can simply write down the indexes. Make sure When you are writing the indexing, you are focusing on the number of elements. So in this, in our case, we are creating a series of 4 numbers. It means indexing should also be 4. If you are going to write something less than that, then you will encounter some errors. Now I have written 4 values and then I am giving the indexes for 3, so it shows ValueError saying that length of value 4 doesn't match the length of index. So both should match. If you have made these kind of mistakes, you will get these errors. So now you will get the series with A, B, C, D as indexes. Any question on this? Next that we have is the DataFrame. When it is DataFrame, it means we are talking about tabular data structure like a spreadsheet where we have two-dimensional arrays. So you can also create DataFrames using dictionaries. You can use any data type to create any data type to create data frames in pandas. For an example, let's say d is equal to pd.DataFrame. Now notice over here, in Series, S is capital, DataFrame, D is capital, and F is capital. So you have to like it, write it in the same way. Now here we have to provide the data. Right now we are just starting with the DataFrame. It means we have to create some sample data, right? So it can be anything. You can create any DataFrame just like we have created dictionary. Same way you can write over here as well. So before this also you can create— let's say I'm writing di is equal to curly brackets name, and I'm going to just write some names. So I've created a simple dictionary. Then I'm going to pass the same dictionary in the DataFrame because now I want that particular dictionary to act as a DataFrame in pandas library. And then you have to— then I'm just going to print it for now. So now this data has been passed to the pandas library, and the pandas library has converted it into DataFrame. And here we have the DataFrame. These are two columns and these are the rows. Any question in this? You can also assign the indexes in this also. So same way, you have to just go over there, add comma, then let's say index is equal to, and then we have to write out the indexes. So initially, by default, your indexes will be 0, 1, 2, 3. If you want something else, let's say data1, data2, comma data3. So then your indexes will be data1, data2, data3. So that's how you have to change the indexes. Now let's talk about if I have the data already. Every time you need not to create the data. Pandas library is used because it has all the built-in functions and you need not to write separately to calculate anything, right? So what you can simply do is you have to first download one dataset from anywhere online. You can go and you can download the data. Now when it comes to datasets, for sample datasets, you don't have to You cannot take everywhere, like if any you are extracting, let's say, you cannot go for that data because that data needs to be public data. You only have to download public data. Is there any question up till this? No. Okay, so simply what you can do is for downloading the data, you can use the Kaggle website. I'm just going to share another window. Okay, so you have to go to the Google and you have to search for Kaggle, and on the Kaggle website you will get all the public datasets. Over here you can see the datasets and you can download whichever you need. Now when it comes to selecting the datasets for your projects, you have to understand which type of data you wanted to get, what you are trying to achieve, what you are trying to create, okay? According to your interest, like right now you're not working under any internship, you are not working for any specific company. Or anything related to this. So you have to go with your interest or your future goal. Suppose one person is interested in finance, so you have to go for the finance-related datasets. Or if you are interested in e-commerce, you can go for the e-commerce. Or if you are currently working in healthcare, go for the healthcare datasets. And then you simply have to download it from this website. There are many more resources for dataset, but this is the more live-on ones. So you can— let's say I'm just going to download this one. So now you can simply use this download button to download the data. So now the data has been downloaded. What you have to do next is you have to take that data to the folder in which you are working. So obviously, if you are downloading it, by default your downloaded file will be in the Downloads. You have to cut it from there and you have to paste it directly in this folder. Okay, so I think that file, that folder, that file which I have downloaded is in the zip file, so I'm just going to take the different one for now. To just explain, yeah, let me just start it up in the folder. Okay, so I've taken this DisneyMovies.csv. So you also have to notice that which one you are trying to import. Okay, let me just reshare everything. Okay, so now our file extension is .csv. .csv means comma-separated file. If I'm going to open it in Notepad, every value in this file will be comma separated. So whenever you want to load the file, you first have to understand the file name. Once you know the file name and the extension, then accordingly you can import it. Now there are built-in methods to import the file, so you have to use pd.read, pd.read. Now you can see, according to your file extension, you have to select Suppose you wanted to read HTML file, you go with read_html. If you wanted to go with the CSV, you have to go with the read_csv. Then you have to again open the bracket. The same thing, you have to paste the path over here. So this is my file. You can directly type the name of that file as well. So this is disney_movies.csv. Now I'm going to just say print df. Now when you will see print df, everything will be in the result. You can see this is the type of a data that I have, okay? This is the dataset which I have selected, which has movie titles, release date, and the rating, total gross. Everything related to this data will appear over here once you are going to print it. But this is not the ideal way of printing the data for view. The ideal way to print the data for view is you have to use two more built-in functions. When you wanted to view the data from the top, you have to go with your variable.head. Let me just increase your variable. So you can see my variable is df, so I'm using the df variable. df.head. df.head will do what? Will print the first 5 rows of your data. When you wanted to view your data from the bottom side, then you have to just use the another function which is tail. This will give you the last 5 rows of your data. So first 5 rows, go with df.head. Last 5 rows, go with df.tail. Any question up till this? Now when you wanted to get the, uh, information about the data that you have, so you can simply use another function like information in terms like, let's say you wanted to know the number of rows and number of columns that your data consists of, so you go with the shape method. Df.shape will give you the number of rows and number of columns. So in total we have 579 rows and 6 columns. Now when you wanted to print the data from one single column, you don't want entire data to be printed, then you have to go with the variable which is your df, df dot, and then you can, uh, there are two ways basically. This is dot method and another one is you have to go with the square one. Now over here you have to mention that column name. So you can see when I have added the quotation, it automatically will show you the columns. So when I am saying df.this, it will show you that particular column data along with the indexes. Okay, when you want multiple columns, then also you have to follow same method, but in this case you have to go with one more square bracket because now you are trying to get more column data. So let's say, um, I need to check the name of the column. So let's say movie title. Yeah, I think we're not closing one method. Okay, so now this is the format in which you will get the data, because now you are trying to get the data from two different columns. Any question in this? Now, when you wanted to delete anything from the data, let's say I wanted to delete one column which is the rating column which I feel is not required, so you can directly drop that column. So df.drop will help you in this case. So let's drop this MPPA rating one. df.drop bracket MPPA. This is the column that I wanted to delete, and then you have to also mention the axis. Okay, so for now I'm writing X as 1. Now this is your updated data where you don't have that particular column. If suppose you like, whenever you try to delete everything from data, you have to be 110% sure that why you are deleting it. You should have the reason for that. Second thing is You should be aware that you are deleting it because you are directly making changes in the dataset. And if you are making changes, maybe in future you need that same thing again and you cannot get it because you have dropped it. Okay, so better ways— you go with another variable and keep a copy of it. df1 is equal to this. So now you are not making the direct impact in the original data, but now you have df1. And df1 will have your updated data where you don't have that particular column. So deleting data is not every time recommended because whenever we make analysis, we have to think of alternatives that you have to be specific. Let's suppose in this data we have the data related to movies, right? So now there comes up a question that which type of analysis you wanted to make from this. Whether it is related to recommending movies to the users or whether it is related to, uh, checking that in 2020, like 1940, which movie was on top, what exactly you are trying to identify from this data, then only accordingly you can change, delete it. Okay. Now let's say you wanted to delete a row. This is the way you can delete a column, but now when you wanted to delete row— so when you are deleting a column, your axis will be 1. When you are deleting a row, your axis will be 0. Same method. So I'm writing df1 is equal to df1.drop bracket. Now you have to mention which indexes you wanted to delete. Suppose I want to delete 576, comma, you have to mention the axis. Axis is equal to 0 because this time I am using the same function but to delete this. Now you can see after 575 I don't have 576 because I have just dropped that particular row. Any question in this? Y-axis is zero because we are going with the horizontal way. Because we are deleting the rows. When you are deleting, deleting the columns, then you have to go with the axis is equal to 1.
If I don't mention any axis, so then it will—
maybe it totally depends which version you are using. Okay, so let's say I'm deleting 577. Okay, so it totally depends how you are deleting it. Okay, so generally, like in cases of small datasets, right, it is a bit easier for pandas to work on and go line by line and go to that particular index and delete it. But when you are specifying the axis, it becomes more specific for the library to work on and will give you the quick results. Okay, okay. What do you think, when should we drop any column?
When there's null value.
When there is null. Okay, how much null? Like 5, 10? Suppose there is a data set in which there are 5 null values in one column and the dataset consists of 1 lakh rows. What should we do? Do we need to drop the column?
We can drop or fill. You can—
yes, so it totally depends how many null values do you have. First thing, whenever it comes to the point that you have to drop something, right, whenever you have analyzed, okay, these are the features of your datasets, these are the number of rows, this type of data type you have in the dataset, after that you have to think of irrelevant features. When it comes to irrelevant features, it means you have to go to your outcome, that what exactly you are expecting in the outcome. If your outcome is related to finance and you have the data related to product description, let's say, or product details, let's say, or reviews or comments, then in finance analysis these features are not required. So you can simply drop those irrelevant features because these columns are not going to contribute to the analysis or your target prediction that you are going to make with the data. So first you have to go with the irrelevant features, and then you simply have to drop them without thinking anything, because you are going to make the changes in the copy of the data. You're not making the changes in directly in the data. Okay, second thing is when you have high missing data percentage— high, it means if the missing percentage is above 60 or above 70. Like, most probably go with the 70, but in some cases we go with the 60+ also. So when a column has too many missing values, or you can say imputation is not possible, then at that time you have to delete that column. Okay. Another dropping scenario can be like, let's say there is a table, there is a dataset in which you have one column which is related to, let's say, gender. Male, female has been mentioned over it. First way is you just directly convert male, female into numerical data, 0 and 1. It will be more easy for you to making analysis. Okay, now let's say there are two different columns. One is for male, another one is for female. In me, male column, everywhere it is mentioned male, male, male, male. And in female column, wherever it was female, it is mentioned, and other times it is just null. So in this case, there is a low variance. It means your data is not varying row by row. So column with the consistency— sorry, so columns which are not consistent, basically, which are having just some fixed values, maybe one value or two, two value then it means that column consists of some constants, or near some constant— near some constants values means they just have 2 or 3 values. Then first option that you have is you convert that data into numerical data using 0 and 1, or else you drop that because it is going to provide very little information and it is going to just give you 0.1% impact in your analysis. So rather than making that a main feature, just go and delete it. Okay, you can also drop some duplicate columns if you have any. Like let's say you have a dataset in which you have 3 different columns: first name, last name, and then full name. So you can simply delete either full name, or you can delete first and last name because you already have full name. So better option is go with deleting full name. Got it. Any question in this? Yeah, anyone, any question? Okay, now let's talk about the next type of function. Now let's say you have two different datasets, dataset 1 and 2. Both are important for your analysis. So at that time, what you have to do is you have to merge your datasets. Merging your datasets, again, have two methods. One is using CONCAT function, another one is using MERGE function. With CONCAT function, you have to mention how you wanted to merge them and with— sorry, CONCAT them. And when you are going with the MERGE function, you also have to mention a common column, okay? So pd.concat, then you will mention your data frames, comma, that you have to mention the axis. And this way you can concatenate two data. Okay, now you can see how many rows in total you have, but you still have 6 columns which are the common columns. Okay, now if I am going to write axis is equal to 1 and I'm going to just run this, now you can see, now we have more number of columns and the same number of rows. When I was writing axis is equal to 0, then we were just getting the double of the rows that we have right now. So that's how axis impacts on your data, and that's how you can concatenate two different datasets. When it comes to merge, there is one more method of joining two data, which is merge. When you're going with merge, you have to mention— first is you have to mention the data frames. So first is, let's say, df, and then second is df1. df and df1 is not having any common column, so merging is possible only if you have something in common. It can be anything in common. Let's say there are two different tables but movie title is same in both the tables. Movie title is the one column which is available in both the datasets. Then you have to do what? You have to write down movie title over here. Over here you have to mention the movie title as a column which is the common column between both the tables. And then you can easily merge. Any question in this? Now let's say I want to filter my data on the basis of certain criteria, or you can say on the basis of some conditions. So I have to— you can use like, I'm going to create one more new variable for the filter data. So df3 is the variable, and I'm going to apply the function on df. Now df is my original data. From df, now where your condition is, for which particular column your condition is So let's say my condition is for— do we have anything numerical? Yeah, I think we can go with this total gross one. Total gross is greater than, let's say, Amin bidi ditshtasata. So this is the data which is our filter data. Wherever the total gross value is greater than this, we are getting that data printed on our screen. So that's how you have to filter the data. For filtering the data, you just have to follow the syntax, that is, from the data set from the entire datasets, and then I am applying the condition. Now here you have to notice the square brackets because square brackets plays a major role in this. When we are going with the columns, you have to use square brackets. Okay, so df, then the column name, and then the operator using which you have to apply the filter, and then the value.
Why 2 DF?
One is for entire data because this is from data I'm applying the, the, yeah, for entire data I'm applying the condition for, and for applying the condition I'm using one of the data in DF. Okay. So filtering is bit easy when you wanted to just filter the data based upon the condition. You can go with this. Now let's say you wanted to add a new column in it. Adding new column is very, very simple. It is just you have to use df, which is our initial dataset variable. So let's say df, uh, let's say new column. Is equal to— now this will give us some error because we are trying to give the wrong length. Okay, so when you have the data which is with the same number of values, then you can simply add this particular column. Got it. So adding column, dropping column, or getting some information about the column. Then we also have one more function which is df.info. df.info will give you the information about your data. It shows that, okay, which type of data you have. Okay, you have now it in the DataFrame. Then what is the range index? So this is the range index. It starts from 0 and ends up to 578, and in total there are 579 entries. Then it shows data column, how many total number of data column it has, then column name showing non-null count and dtype. Non-null count means basically it's going to show you how many non-null values are there. So wherever you are getting this 579, it means that column consists of zero null value. But if like in this MPAA rating we have this 523 or 562 or 579— sorry, yeah, these two only. So these two columns have null values. This also shows you the data type. So just in the beginning of Python I told you that those data type which we have learned, which is string, integer, float, are just for the basic understanding of the data structures or data types. In pandas, we have the data type object. Object means string. Object is similar to string. Then we have int64, it means integer with the 64 bits. And then in the total also, it will show you how many object column you have and how many integer values you have. Integer value or string value. So there are 4 columns which have string values, and there are 2 columns which have integer value. And how many— what is the memory usage of this particular dataset? Then that also shows you within the info. We also have one function which is called describe. df.describe is going to show you some statistical analysis of your data, that what is the total number of count Let me just add it up in the print. Yeah, so now we have the total number of counts, which is 579. It was supposed to be— and then the mean value, standard deviation, minimum value, and then 25% value, 50%, 75%, and max value. Everything you can get at once with the describe function. Got it? Any question in this?
Hello? Yeah, in this info, I mean, how do you check the total number of not null value and the percentage of null value. Let's see in 2 and 3, January and MPPA rating.
Yes. So for now, I have this INFO function is going to just give you the overview of everything, right? That how many, like what is the non-null value and all. So what you can do next is you can use, uh, some more functions to identify missing value. Okay, there are two different types of missing values. Whether in that particular column's like data cell you will get NA or you will get null as a value written over there. Okay, so you have to use the functions called is_na and is_null. Okay, so df.is_na will give you the data in the true and false format. Wherever the null value is, it will add a true at that place, or if you have no null values, then it will add the false. Okay, but this is not the effective way of viewing it. You have to target the particular column. Okay, so let's say my column is gender1. Okay, so I'm going to add up and then ISANY. Again, it will give you the values in the true and false. You can simply add a sum over here and it will give you the total number of values where NA is present. Okay, also to make it more accurate, or you can go with the ISNULL also. So wherever the null value is, it will show you that value. It is considered as an NA or null, then only it will present. If suppose there is an incorrect data present— incorrect means, let's say, an MPA rating, and rating was supposed to be G or E or something, and some characters has been mentioned which is totally out of context. So that won't be, you know, counted in this particular function. So this function is just going to count the main NA and null values. Got it. So like right now in this data, if I give you the example of this Disney one only, so you can see we don't have more— like it is just, I think, above 80% of data we already have. Only just in some cases there is 20% of missing data, or in some cases maybe there is 10% of missing data or 5% of missing data. So you can simply impute some values. In this case, to impute the values, you can go with the mode value from the entire data, which is like frequently occurring. Then you can add that data value at the missing value, in the place of missing value. Okay. That's how you have to work on this datasets. All right, now your today's task is that you have to practice the data Pandas library, and then you have to take one dataset from Kaggle, and then you have to try to get some information from the dataset. And also you have to get ready with the analysis, the type of analysis that you wanted to make. Okay, and then you have to start with going ahead with the missing values and handling it up.
