Day-10 Python Data Ninja: Master Data Manipulation in Minutes! (For Beginners)

Ladies and gentlemen, hello everyone, welcome back to my YouTube channel. In today's video, we are going to see how to do the data manipulation and what exactly data manipulation is. So basically, data manipulation is something for the data itself. In this stage, what we are going to do is we are going to transform our data, we are going to clean it, we are basically going to restructure our raw data into the meaningful and the organized data which makes it ready for the analysis and the visualization, or we can say that the data is ready for decision making. There are multiple operations using which you can do the data manipulation. Let's explore few components of data manipulation. The very first is the data transformation. In the stage of data transformation, what you are required to do is you are required to convert one data from one format to another format, or you might have to change its structure as well. So change its structures means suppose there is a column in which we have some values and the values are like after decimal there are 3 values and somewhere it is written after decimal 2 values. So to make it linear, you have to modify its structure by changing the, uh, 2 decimals into like 3 decimals into 2 decimals so that it will have the linear values entire the data. Another is the data cleaning. In the stage of data cleaning, you have to identify and rectify the errors that you might have in your data. You have to figure out what are the inconsistencies you have in your data what are the missing values or where the missing values are exactly. So for doing that, what you have to do is you have to handle— you may have to handle the duplicate values, you might have to do the data formatting or dealing with the missing values, you have to remove the outliers, you have to do the standardization and normalization of the data. Another is the data enrichment. In this stage, what you have to do is you have to enhance the existing data with the informational— like information that is required for more good insights, we can say. So for this, you might have to add a column or add a feature, or you might have to integrate it with the external data source for some external data, and so on. Another is the data validation and the quality assurance. So for this one, what you have to do is you have to see whether the data is complete or not, whether the data has the perfect consistency or not. The data is— this is— the data is something like with a good accuracy or not. So you have to validate your data, you have to check the quality of the data, you have to— you might have to do the data profiling, or you might have to do the outlier detection, validation rule, or error handling as well. Next step in the data manipulation, or we can say the next component in the data manipulation, is the data aggregation with the summarization. Here you have to group the data, you have to calculate the summary statistics— mean, median, mode— and yeah, then you have to generate the reports or the dashboards. Then you might have to do the filtering, which we have learned in the last session, that how to do the selection. You might have to extract the subset of the data based on the certain criteria or the condition. Another part is the data integration. In this stage, you might have to combine multiple datasets into the unified dataset. Let's take an example. You might have to work with the 5 different Excel sheets, but in case of working with the 5 different Excel sheets, your analysis are not going to be the uniform or the linear. So in that case, what you are required to do is you are required to create the sixth sheet in which you are going to combine all 5 datasets and make a unified dataset for the comprehensive analysis. Then next step or next key component of the data manipulation is data sampling. You might have to extract the representative subset of the data for analysis, and especially the large dataset. Suppose you have the 10 lakhs or millions of rows data and you have to work with the, you know, you have to work with the data but your system isn't supporting or your analysis do not require that much large data. So in that case, what you can do is you can extract the subset of the data for that particular analysis and come up with a decision required. Another is a data reshaping. Then this says what you have to do is you have to transform your, transform your raw data into the structure of the data to facilitate with the analysis and the visualization. For this purpose, what you can do is you can do pivoting, melting, stacking, or unstacking of data. Another is the data reduction. You might have to reduce the dimensionality of the data, but while doing this stage, while completing this task, you have to make sure that you are preserving its essential characteristics. That's the must thing that you need to take care. So for this, what you have to do is you have to do the feature selection, dimensionality reduction techniques like PCA or t-SNE. So let's see how we can implement it within the data. So I am using the same data which we have used in the last video, which is NBA.csv, and I have imported the pandas library as well. Now I have imported the data. Here I am printing the first 5 rows of the data. So you can see we have the same number of rows: 1, 2, 3, 4, 5, 6, 7, 8, 9. We have in total 9 columns in our data and multiple rows. Now let's see how we can add a row or column, how we can do this. So suppose our task is to add a new row. So for adding a new row, what I am doing is I have created a variable. In this I am adding all the details that are required. Make sure that you are writing all the details for all the required columns. So like For name, I am writing name. For name, I am writing name, then team, then number, position, and so on. I have written for every, but in case of college, I have kept it none. It means this is a missing value, and I have to append it. It means there are two methods to do this. One is using the append, it means to add that particular row in the end of the data. And second is you can use the CONCAT. CONCAT is also going to do the same thing for you. It will add your data into the end of the data. Frame that you have already. So for doing that, I am creating a variable df. In the same variable, I am trying to get the data. So pd.concat, and in the concat, I have to write down pd.DataFrame, and in the brackets, neuro, and index I am ignoring so that there won't be any error. So now let's see how we can add a column. So for adding a column, what I am trying to do is I am trying to create one column on the basis of certain, you know, parameters. So let's say in this case I am trying to— I have already have the column height and weight. So when I have height and weight, I am not sure, it just says 6'5", 6'6", and something like that. I want this height to be converted into inches. So for that I am doing df, and this is my column name that I am expecting it to present. Then I am saying from the height column, I am applying one function to convert it into the inches. So for converting it into the inches, I am using the lambda function where I have written x and x.split, then multiplying it by 12 to convert it into inches. And then same way I am doing it for the 1, splitting it with 1. And if there is any missing value, I wanted to come up with the NA only. Now another column that I am trying to add is on the basis of the multiplication of the multiple columns. So I am trying to add one column which is BMI column. So for adding the BMI column, I'm going to use the weight, height. So the formula for the BMI is weights in pound, height in inches to the power of 2 multiplied by 703. So I am writing the same thing. I am creating a column which is df bracket BMI. In this column, I'm trying to add on the values df weight divided by df height in inches to the power of 2 multiplied by 703. And that is my data which is generated. You can see we have two more column columns like height in inches, and these are the heights in inches. Wherever the missing values were there, it just presented it with the NA. And in case of BMI also, it's doing the same thing. It generated the BMI for us using the height and weight column. I hope this makes sense to you guys. If you— yes, then you can drop your comments. And let's continue with the next task. The next task is removing the rows and the columns. So removing the rows and the columns means if you wanted to delete something, delete a row or delete a column. This way I was adding the columns and rows. Now let's see how to drop it. So you can use the drop method to do this, to remove rows, to remove any— suppose there is a missing value in a particular column and you wanted to drop all those rows which consist of missing value in that particular column. So what I have done is I am using df.drop_any. drop_any means dropping any values from the salary column. If there is in the salary column, I am dropping it within the salary column if there is any, any values. Another is to drop a column. To drop a column or to remove a column, you can use the directly drop method with the axis 1. You have to write down the axis 1 if required, else you can continue. df is equal to df.drop. I am writing column is equal to this. If you are not writing column is equal to this, then you might have to use the method of axis. I am writing directly column is equal to college. Then I am reprinting out the data to check whether it is done or not. So if you can see over here, we do not have the college column right now because we have dropped it. Then also from the salary one, I have also dropped the row which consists of any— see, if you have observed that 0, 1, and then directly 3 comes up. That's because the number 2 index was having any missing value in the college column or salary column. Yeah, over here you can see that this is the column which was having missing value in the salary. That is why that particular row has been dropped. So that's how you can remove any row or column. Another concept is the sorting. This we have discussed as well, I think. So this is very easy, just like in the Excel we do, or directly with the filter one, we can do it over here using the code also. So I've created a variable, then I'm— I have written df.sort_values. I want my salaries to be in the descending order so that I can check whose salary is highest one and what's the name of that person. So I have done ascending is equal to false. If you want it in the ascending order, you will keep ascending is equal to true. If you want in the descending order, you will say ascending is equal to false. And this is how the output is generated, where we have this is the highest salary and this is the person's name. Now I can easily observe it, right? Now let's see how to do the filtering. Filtering we have already discussed based on the criteria using the Boolean indexing. So I have created a variable. What I am trying to do is I am trying to check whosever age is below 25. I want all those youngest players in the list. So here is the younger player list where I am expecting df, df.age. Condition is applying using the— condition is applied using this symbol, and then I am writing 25. So here we have the output where the age are less than 25. Now let's see how to handle the missing value. If there is any missing value in the data, there are multiple ways to do it. One of the way was to drop it or to, uh, directly delete that particular row or the column. So now if you have the missing values, there are multiple ways to do it with. So suppose you have the missing value, you can use the .mean method to replace that particular cell with the mean value of that column. So in the case of salary, there are multiple missing values. So how I am dealing it is I am saying df_salary.fill_any df_salary with mean. So there I am writing a function.mean. So what will it do? It will take all the values from the salary, it will count out all the values, the number of values, and the sum of values. So sum of values divided by number of values, it will calculate the mean accordingly and will replace the missing values with that mean value. Another is that you can drop the rows with any missing value. So I have done two things. First, I have updated my salary column, and once the salary column is updated, what I have done is I am expecting it to drop all the rows which has the missing value in any of the column except salary column. So that is what is done. That's how you can do the data manipulation. And there are many more things that you have— you should learn in data manipulation. I will be uploading more videos in this, so stay tuned. You can subscribe to my channel if you like this video, and we'll meet in the next video. Bye-bye.