#4 Learn Pandas in One Shot: Stop Struggling with Data!

So hello everyone, welcome back to another video of Machine Learning from Scratch series. And in this video particularly, we are going to discuss about Pandas library. Now what is Pandas? Why do we need Pandas? Can't we do our work without Pandas? There can be so many questions in your mind, so let's first discuss them. The very first question is, what is Pandas? Pandas is the library which we use for preprocessing the data. Yes, our models need data, our artificial intelligence tools need data basically, right? So everywhere we have to work on data. Either we are doing the data collection, and after doing the data collection, we have to give it a proper format. We have to clean that data. We have to make sure that data contains only accurate value so that our models get trained on the correct data only. So therefore, Pandas is the library which helps us. Question number 2 is: on which technology or on which sector Pandas is built on? So Pandas is majorly built on NumPy library. Yes, that is why we have covered NumPy in the last video where you got to know how use NumPy, how mathematical computation happens, and how numerical computation happens within the NumPy, right? So this is the advanced version of NumPy, and Pandas is purely used for manipulating your data and doing data analysis. Both are the things which you can do within Pandas. Now Pandas is widely used in data analysis, data science, artificial intelligence, machine learning, and at many more places. So now what we will do is we will experience pandas from scratch, how to create projects, what are the things within pandas on our Google Colab. So let us continue on Google Colab. So if you all are excited about pandas and this wonderful video, then do let me know in the comments. Write down, I am excited, and let us begin on Google Colab. So first we have to import the library pandas. So when we are importing the library pandas, we have to give it a nickname. Yes, just like we all have nickname at our home, right? So same way, we have to give a name to the pandas library. Why do we need to give that? The very first reason to give nickname is so that whenever I have to call pandas, I do not have to write down a long name pandas. I just can write pd and it will automatically come wherever I need with all the resources. So pd is the nickname now. Now later on, whenever we will need this pd, we will— pandas, we will just use pd. Now you have to understand, in pandas we have different data structures. We have basically the different names of data structure. So just like in Python, when you learned about the basic, you learned that, okay, there is a numeric data type and we also have the text data type, in which whenever we want to store text like alphabets or numbers, we have to store it in string format. Same way, in pandas also we have a different type of Data structures. Yes, if you want to store something in pandas and you wanted to basically create the data structure, then there are two options: Series and DataFrame. Now, what is Series and what is DataFrame? So as I told you, in NumPy, right, we create arrays, right? And arrays are what? Arrays are similar to lists. Same way, Series are also similar to lists. You can also consider lists— that Series basically as a one-dimensional array. Yes, that one-dimensional array is series. Now when we talk about writing the series or creating the series, it is similar to the list that we create. It means in list what can we have? We can have the values, we have to store it in the square bracket. But there is a difference when we go via NumPy array and when we create series in pandas. There is a difference. Pandas is something that is column major. Pandas is column major, but NumPy is row major. I will show you the difference practically, but for now, remember these two statements: Pandas is the column major and the Series is the row major. Now, the second data structure that we have is the DataFrame. So DataFrame is nothing— it is just similar to the Excel sheet that we have. In Excel sheet, we have rows and columns. You can consider it as 2D array, two-dimensional array, in which we have rows and columns. So rows are horizontal, columns are vertical. You can create rows and columns with the help of DataFrame. And DataFrame is used especially— like, mostly we use DataFrame only within pandas. Now see, within pandas, as I told you, if you want to create multiple data frames, then we call it panel. This is the third data structure in pandas. Why I have not mentioned it earlier is we rarely use panel. So panel is nothing but a dictionary of data frames. So when we combine multiple tables or multiple data frames, we call it a dictionary. In short, in pandas we call it a panel. So we won't be discussing a lot about panel, but we will be just discussing about the two data structures Series and DataFrame. Now Series and DataFrame both are important because sometimes we have to extract the data, and whenever we are extracting the data, we can go with the one-dimensional data storage unit so that we can extract it quickly, right? So it is a column major, as I said. Pandas is column major. Row is, uh, sorry, NumPy is row major. So two different libraries, two different use cases. So there is a difference. Still there is a difference between the NumPy array and the Pandas series. Column major means one next to the other, but in the horizontal line. It means everything that you are going to create will create— basically, if you are going to create a series in Pandas, it will create in the format of column. And when you are creating an array in NumPy, it goes as a row major. Yes, it means it will print the output horizontally. Pandas will print it vertically. Now, for creating any Pandas series, what do you need? For creating a Pandas series, we don't need much. We need to use the pd.Series function. Yes, in Pandas we have a Series function. So this is Series where you have to make sure you write S in capital. But before Series What do you have to mention is you have to mention that library name from which you are taking. So pd.series, and then bracket. Obviously, pd.series is a function, so within the bracket you are going to pass the value. Now you see, if you look at it carefully, it printed the output vertically. It printed the output vertically. So this is a series where you are getting the output vertically. And what is the data type? Data type is integer64. Now if I write down the same code, if you remember the last video, import numpy as np, and then I'm creating a NumPy array with the same value 1, 2, 3, very basic value we are taking. But for now you have to notice what is the difference between an array and series. In NumPy the values get stored as in row column or row ways, row major basically, and in Pandas it goes as column major. So that's a major difference between NumPy array and Pandas series. I hope you understood. And still you have any doubt, drop it in the comments. I will take it up from there and will surely help you. Now, so when we talk about writing the series, so what I will do now is I will go back and create the series again. So whichever series we have created above, I will take it up and I will show you how expression works. Okay, so this is the series which I have created, 1, 2, 3. Now I will add a NumPy expression into it. NumPy expression means I'm trying to evaluate it. Basically, I want to perform some mathematical expression on top of it, right? So when I wanted to do that, what do I have to do is I have to write down np because I have already imported the NumPy library in it. So I will directly go and say np.expression. Now np.expression means now the series has a NumPy mathematical operation activated, and I can add or subtract or multiply whatever I want to do. Now let's say I want to go element-wise and I want to perform addition. So in the s series, I want to add 1000, 2000, and 3000. Now see, 1000 1 is 1001, 2000 2 is 2002, 3000 3 is 3003. Now what I did is I did element-wise addition. What I did, element-wise addition. But that element-wise addition is with the help of NumPy expression. So this NumPy expression can take any argument, any keyword argument, and can let you perform anything on the array that you have created. You can create or calculate the exponential of all the elements. Like you wanted to perform element-wise, you wanted to perform, you know, at one go with one value you wanted to perform any multiplication, division, or something, you can basically do that. So it is nothing but you are trying to perform element-wise, or at one go you want to perform some operation, then you have to use this np.expression so that you can apply any mathematical expression on top of it. Understood? So now for this, what I did is I did element-wise. Now we will try how we can use one value and perform Something on series. Something on series means obviously the operation, mathematical operation. I hope you are getting it. We have to calculate sometimes mean, median, mode, or sometimes we have a fixed value given to us. Let's say a company is giving bonus and that bonus is fixed, that okay, we are going to give $1,000 to everyone. So $1,000 means 1,000 is a fixed value, so that has to be added in everyone's salary no matter if anyone's salary is $5,000 or $10,000, 1,000 has to be added, right? So there we are going to use it. Now instead of giving 3 different value, what I will do, I will remove 2 of them, which is 3,000 and 2,000, and only 1,000 I'm going to add. So now see, this way I am able to add the 1,000 on each value. This is what we call as broadcasting. Broadcasting means sending. Just like on WhatsApp broadcast, right, we have a feature we can send the same message to thousands of people, right? Maybe lesser than that, but basically more than one, right? So broadcasting means when you are trying to perform any operation with single value on multiple different elements, then you are going to use the broadcasting method. Instead of plus, you can use any other operation as well, so that is fine. Now let us understand next thing. Now what is next is I have copied the same code from above and I am going to show you some other operations. So same code I have taken. Now you can filter out as well. Now let's say I wanted to generate some Boolean expression. Boolean means true and false. So now what is my condition? My condition says if less— if s is less than 0, then do let me know. Is s less than 0? Any element in s if any element in s is lesser than 0, then it will generate true for me. Otherwise, false will be generated. So now you can see I'm just flipping the operation, means s greater than 0. Now it is true, true, true because all of the values are greater than 0. So basically you can use this type of filter out stuff so that you can filter the series and can basically check what type of values exist in the series. Now notice one thing, we are creating one-dimensional array, and with the one-dimensional array we are getting by default value 0, 1, 2. If you remember the basic Python, this concept is known as indexing, right? 1, 2, 3, it means the serial numbers added to, uh, the positioning is indexing. So in list also we have the concept of indexing. But can I change index in series. Although in list we cannot do that, right? But in series we can. So just like how we created the basic series pd.Series and giving the value, in the same bracket, in the same round bracket parenthesis, add comma, write down index, and then write down whatever indexes you want to give. Now I have 3 values, so I will give 3 index: Akansha Yadav, Python, and let's say Pandas. So 3 different values I have. Initialize. Now if I will run this, I will get the values stored. Now you can see the values are here. It means now my indexes are replaced with the string values which I have given. You can give it any numeric value, you can give it any string value as well. So both are fine. Only thing that you have to take care is that you have to make sure that you are giving the same number of elements. So this is indexes that we are going to apply. And we also call it index label. Index label means— label means I'm giving some text, right? So this is index label. Now, does it mean that when I changed the index, my initial or by default 0, 1, 2, 3 are gone? Let's check it. So when I wanted to check whether it exists or not, or what is the use case of basically index label So whenever I want to check what is the value at index pandas, I can instantly get it by using the square bracket. How we do it in basic Python, we use square bracket to mention that, okay, this is a pandas library— oh sorry, this is a pandas index. And from the pandas index, what is the value stored at this particular index? It is 3. Now if I wanted to access the same 3 value via by default integer index which is 0, 1, 2, 3. Can I get it? So let us try. So I'm writing s1 square bracket 3. 3 is obviously— it doesn't exist. So obviously it will generate the error because we only have 3 values and the indexing starts from 0. So now I will write 2. You can see 2 is giving us the correct value 3. It means whenever we are giving the indexes, it doesn't mean that we are replacing the by default integers. No. Why, why is this feature in Pandas? Are— do we have like extra space to do that? No, there is a reason for that also. We will talk about it. In Pandas, we create series. Series have the indexes, which is the by default indexes, like integer indexes, 0, 1, 2, 3, 4, so on. Still, for your convenience, you can replace it with any label, any other integer, or any other alphabet. That is fine. But in pandas, we have two functions, okay? When we work with millions of data, these two functions helps us a lot to access the value. loc is one function, and then we have iloc also. So let's first see what is loc. Loc is a short form of location, you can say, okay? So when I'm writing loc and in square bracket I'm writing quotes, it means I am trying to mention any label, okay? Loc is for location. If someone asks me where do I live, I can mention my location, right? In which country specifically I live, or in which state specifically I live, right? So location is label-based. Location is label-based, right? So I'm writing location is pandas. So where do I have to apply this location? My location has to be applied on S1 data which I have created. So S1.loc square bracket pandas, it means wherever in S1 there is a pandas check out what is the value stored at this particular index and give it to me. And we are getting 3. Now, why do we have integer value? Why by default we have, you know, values like 2, 3, 1, 2, whatever the basic by default values get initialized with, right? So because we have another function called iloc. Iloc is indexed-based locationing, right? Index-based. I stands for index and loc means location. Now we can run this and get still the same output. So this becomes more convenient. So at any point of time you wanted to access something with your own labels, maybe the cust— company has a specific criteria or specific, you know, uh, rules and regulation. According to that also you can add some text and can access the data quickly. And second, by default indexes also works. So make sure you use both, like whenever, whichever is required, go jump, use it. LOC and ILOC. LOC in short is for labels. ILOC in short is for indexes, which is integer-based indexing. Label means any text, any alphabets which you are trying to write down as an index. You can use it for your convenience. Got it. Now see, when we are creating loc and iloc, do not get confused, right? Iloc also helps us in slicing. So whenever I want to slice, let's say I want 2 data— I was having 3 data, but now I just wanted to print 2. So it means I want to slice it, just like how we do it in Python, right? Normal list, we use first value, last value, and we enclose it in the square bracket. And the last value has to be 1 more than what I am expecting. So I want from 0 to 2. So I can simply print it and you can see the data type is integer64. It means in pandas, whichever value you are trying to store, it gets stored as an integer, single value. It doesn't combine it, right? So when we create list and we check the data type, it says list, not integer, right? So that is why when we perform any mathematical operation on top of list, it gets treated as a string. Now I am creating, let's say, a surprise as a series which is with some random numbers. Now I wanted to slice the surprise, so what I will do is I will write down the variable surprise now. Now within the surprise, first look at the values what is getting stored. We are getting the values in the column major and all the values are listed like this. Indexes also exist. Now I wanted to slice it. To slice it, it is easy. You write down surprise_slice and then surprise is obviously a variable in which we have, uh, series, and we can use it directly. And then I will use the slicing method just like I did above. 2 colon, it means starting from 2, whatever exists, do that. Now let us just run it again. Starting from 0, I want every single thing. Starting from 0 means all the data I want. How I'm trying to basically show you all the methods that we applied on list are still valid for, you know, NumPy series, NumPy arrays, and this Pandas series. But there is more convenience, there's more easy way to get the data, to access the data, and so on. Okay, so this 5, 10, 20, whatever you want. Now earlier in list case, what happens like when we are giving more value? It means if my data is just limited and I'm trying to access something out of the range, I will get the error list index out of range, right? But now in pandas it doesn't happen like that. Now let's say I am trying to, you know, work with— let's say I am using the same variable but I have mistakenly— obviously intentionally I'm writing it, but I am trying to generate error, okay? So let's say I'm writing wrong error for now, which is— let's say obviously this will generate the name error, but now I'm writing KeyError. So now KeyError as e. Now I will write print, and I'm just going to smoothly handle this error. I hope you know about try and except. Try and except is exception handling, and exception handling means we are trying to handle the errors smoothly. So these red color error which comes up, right, these red color flashes basically crashes your code, right? So now I know that, okay, this code is going to generate name error. So when I will write down the correct error, it won't give me that red color flag. Red flag, I'm going to avoid. So now I can just smoothly handle the error and smoothly jump on to the next code cell without crashing my code. So try and except is equally important for all of, you know, coding journey. If you are trying to create models, try and except we will be using a lot because there are some cases where maybe your system is not able to handle the large data set. So try and except can truly help you at that point of time. So try and except, exception handling, you must know. You should know the methods which are applicable on lists. So basic Python is literally very important. You must check out my Python course, which is absolutely good, I would say, if you wanted to learn Python fully with all the methods, all the basic stuff, and even the logic building. So this is try and except, and you can use it for the majorly, you know, calculation and handling. Okay, so now let's see how to create a series with the help of a dictionary. Yes, you can create dictionary and can create series out of it as well. Let's say I'm creating name colon, and I can give it a name, and I can add more key-value pairs in it. Let's say I'm adding Python, and let's say I'm adding an integer value. Let's say 45. Then let's say I'm adding Bowen, and let's say I'm adding one more integer value. Now if I will run this, then what will happen is I will create series out of this dictionary. So first I have created a dictionary. I will have to run all cells so that Okay, so let us go to that particular cell in which we were writing the code. Okay, so now I have created a dictionary. Now using this dictionary, what I will do is I will write down pd.Series and then can pass the weight, which is my variable for the series creation. Now this is how you can create it with the help of key-value pairs. So in this case, what happened? In this case, we were able to store the key-value pairs. In this case we were able to store key-value pairs. It means the values are the actual value which will be converted as in the column format. And when you are writing the key, key will be considered as index. He will be considered as index. And if you wanted to give any other new index, or you wanted, you know, you want to control basically, let's say you want to control which element you want to include in the series, and in what order, let's say. And you wanted to mention the order, let's say. So when you wanted to mention the order, let's say you don't want it in this order, you want Python first, then Darwin, and then name. So you can rearrange it as well. So in the same, you write comma and then give it an index. Index is equals to, and then you can give them the value. So what are the values? Let's say first I want Python, then I want Darwin, and then Run it. So now see, the order will change, the order will change, the value will remain the same. So like for Python I have written 45, so it will remain the same. For Darwin it is 78, so it will remain the same. And then you can create this series. Now next is when you are creating the series, you can do automatic alignment as well. So whenever the operation involves like, let's say, multiple series, you have, you have multiple series objects, so pandas will automatically align the items. By their matching index label. Let me show you how. So let's say I have— okay, I already have one series which is this. So let us store it in some variable. Let's say this is my S1 variable in which I have the series. Okay, now what I will do is I will write down— so let us say I want to create S2 and I am writing pd. Series brackets, and I will use the square bracket and mention some values. Okay, I've written 4 values, or let's say 3, right? So I will use 3 of them only, and I will write comma, index is equals to, and I will give it them index. So whatever the index I want, I can give it. So let's say my index is name Python, and then comma, Darwin. And then I will just say s3 is equals to pd.Series of— wait, then I will just print s3. Is, and we can see the series have these three names and it is considered as object. And if I will just add both of them, let's say I have two series, so I will say S2 S3, and now I will run this. Wait, S is in the caps for us, so I would like to have caps. So what I did wrong? Okay, let us just change this value and let's say 58 is the value. Now we'll just run these cells again. I just wanted to show you can perform the addition as well when you have stored the values. So earlier it was 85, 68, 45, 95, and then 78 and 85. 5. I created one series with the new values, which is name Python, Darwin, with this different values. And the second index, second time when I am creating, which is the second time over here, S3 and S2. In S2, I have initialized both of the things, values and the labels, in the same line, right? And the S3 is something which is using the weight, and in weight also I have mentioned in the key value key pairs. So there is a difference. I am using a list format, I am using a dictionary format. I have created two different series, but what is common in these two series? The common thing is that I am using the same index. You can see same index I am using. Only three indexed I have used: name, Python, and Darwin. These are the three names which I am using, and using these three names I can do it. Right, I'm creating series and then wait. Now s3.keys, I can simply do the addition. Now you can see 68 85 153. Then all the addition happens. Basically I added up two different series and this is how the count shows up. This is the s3 which is a series integer, 64 values are shown. How many values do I have in this series? 3 value. What is the minimum? Value in this, I have the 45 as a minimum value. What is the 25% value? It is 61. So that is how we can work on series. There is an automatic alignment. If you have the common name, common numbers, you can automatically work on it. And later on, if you want, you can make this. Okay, now let's say I'm going to pick one series which is this S2. Okay, in the next cell, and I will show you, let's say I'm creating S4 as a new series, but in this series what I will do is I will just write series bracket, and what do I want to store in this? So let's say I'm just storing some random numbers. Okay, so 3 numbers I'm taking because 3 values only I have. And now if I will run this and I will say S2 S 4. So what will happen in this case? In this case, it will again do the same thing. It will try to merge both of the series, but this time it will automatically give the by default values, by default indexes. And these default indexes helps us to track the values. So this time pandas cannot align the series, so because the labels were not matching. But as the labels were not matching we use the new values. So it is initialized with the new value. Okay, now this happens when there are no matching labels, right? Now the next thing is, let's say I am creating a series which is pd.Series, but this time I'm not giving it the sufficient value. Let's say I'm just giving How many values? That's only one value, 52. I'm getting, okay, so 52 is my value. And now I am creating the index, okay? This is my index. So I'm writing life, let's say world, just giving some random names, okay? And let's say anything, okay? 3 labels I have given and 52 is the value. Now if I will run this, you can see it will pick up the value. So basically, now you can initialize the Series object with the scalar value. What is the scalar value? Scalar value means single value. And the list of index labels— it means I do not have to explicitly mention that it is for the index. I just gave one value and separated it by a comma. So my Python will automatically understand this is the scalar value which is given to us, and we have to store it in these 3 keys or these 3 indexes. So series can have name, series can have anything, right? So if I will run this, let's say this is a data type integer. For now it is just showing data type integer. Now I can give it a name also. So now let's say this is just random thing, so I can give it a name. So series can have name. So let's say the name is, uh, randomness, random. Okay, so this series will have name now, you can see. You, instead of the 0 over here, we were getting 0 earlier, right? Now I changed it to random. It means series can have name and you can give it any name. Got it? So in pandas, when you are creating series, this is how it works. You can work with the datetime as well within pandas. So for working within datetime, this is very important. That is why we are going to discuss it now. So you just have to write pd. DATE_RANGE. So there are different date function— DATE, DATE_OFFSET, DATE_TIME_INDEX. So I'm using DATE_RANGE for now. So what is the range and what do I want to get? Let's say I am— I want to just create a time series data using this function, and I want, uh, to return basically the date-time index, uh, where I will contain— I will basically try to have one date-time per hour. Okay, for next continuous 12 hours, let's say. Okay, so what I will do, I will just use the bracket and mention the date. So let's say, what is the current date right now? Or let's say the previous date, let's say 2020, and it means October, and let's say 28th, 28th October. And then let's say the timing is 5:30 PM. And how long do I want? What should be the period? Period is equals to, let's say, I want 12 hours. Within 12 hours, I want continuous data. Okay, so I do not have to mention the hours, but I'm just saying, telling you that, okay, 12 is the value. And then what is the frequency that you are looking forward to? I'm looking forward to hours. I'm looking forward to hours. It means H. Got it. Now if I will say dates, it will generate something like this. Now you can see 2020, 10th, which is 28th October 2020. And this is after 1 hour, whatever comes up, it gives us 5:30, 7:30, 9:30, 11:30, and then midnight, midnight, right? 5:30, 6:30, 7:30, 8:30, 9:30. 10:30. So basically 1-hour gap. It means I am trying to generate a data which is in the datetime. DateTime means I have the date also and time also. So date is in the format of year, month, and date, and then hour is hour, minute, and second. So this is the correct format of datetime index. If you are creating something like this, it will be initialized, okay? That's how you can create or generate any data, time series data basically. Okay, there is one function called timestamp. There is a function called timestamp. Okay, so let us just utilize that. So for timestamp, you write it like pd.Series. Let's say I'm creating the timestamp series. Series is, let's say, pd. Timestamp, and within this timestamp I will give it certain time. Let's say I'm giving the same one, 2020, uh, 10, and let's say 28 again. And then let's say I'm giving it 5:30 PM. Frequency, let's say again an hour I want. 4 point— let's say I want to use it as a— let's say I'm using it key-value pair. Okay, so this is my key and the value I want is, let's say, 4.4 comma Let's say I'm going to use this again. And this time, what do I want is I don't want 5:30, but I want 6:30 and R. And let's say the timestamp is 4.5. So basically, this is how you can create it. And then I will just run it. I have not closed it. Okay, let's just close it. Frequency, I'm using h, and then I'm using the timestamp. Okay, so that's how timestamp works. Okay, time This is how date range has to be created. Now the next thing is I will create some temperatures, let's say. Okay, so I'm creating the temperature variable, and within this temperature variable, what I will store is I will store some values like random float values. And then I will create temperature series over here using pd.Series, and within this pd.Series I will write temperature, which is my variable, and comma, I will use dates which we have defined above. Okay, so both I'm using, and then I will just print this temp series and see what is in the series. Now you can see in the series I have the time and the values are over here. Now what I will do is I will write down— I will do resampling. Okay, so temp series I'm going to use, and within this temp series is equal to pd.series again. This time I will use a new variable, which a new function, which is timestamp. Now within this timestamp, I will start a bracket. I will write down the values in, uh, date and then time. 10/28, and let's say the time is 5:30 PM again, comma. Let's say frequency is equals to H2O. Okay, and I will just close the bracket, comma again, and I will use the same set of value, but we'll just change the frequency. Okay, so let's say for this one it is 2. 2.3, and for this one let's say 4.5. And let's say the time is also changed. So now what I will do next is I will say my temperature series is equals to temperature series again, dot resample. Now I want to resample it on 2 RVs, okay? And then I will say temp series. But before we run this, what I will do is I will add more values in it, okay? So I will just copy this, add comma, paste. This time I will make some changes. And then one or two more data I will add. 5.2, let's say, and the time is 9:30. We've done, say, 5:30, 6:30, 7:30, and then 8:30, let's say, and then run it. So that's how you can create a series with the temperature and the date as well. You can plot it in matplotlib library, Seaborn libraries, and other libraries that we have. Okay, so I hope you have understood the series completely. Now let's jump on to the DataFrame. Now in DataFrame, if you wanted to create DataFrame, DataFrame objects can be created using the dictionary variable because we need two values for the dictionary, right? So if you want to create a DataFrame now, for DataFrame we need— we— you just visualize this as a spreadsheet, as an Excel sheet that we have. In Excel sheet, what do we have? We have cell, we have cell values, it means what we are storing. We have column, column names, row, row index. We have and you can define whatever expression you want on column. So it is like totally like spreadsheet management using the DataFrame. So that is what we will create now. So DataFrame is very important because whenever we will load the data from Kaggle or anywhere, we are going to use the same thing. You can even use the Series to create the DataFrame as well. So for now, what I will do is I will pick up all the Series that I have created above and will show you how to create one. Okay, so let's say this is the one I'm copying and I will just use it over here. Okay, so first what I will do is I'm creating, let's say, po_dict is equals to curly brackets. This is my first. Okay, so as I told you, it is— it has to be in the key-value pair. So key-value pair means dictionary like this, okay? And this S1 is obviously the key for the dictionary, so I will reuse this. Then the second one, comma, paste it again, comma. I will just name it S2. Instead of equals to, we need colon. And then following up with the next, same thing I'm doing. And then I will give it another name. Let's say S3 is the name given, and then comma, colon again. And yeah, that's it. Now I will just run this. So this is, um, tuple object is not callable. Where are we creating tuple? To just restart the session. And in the meantime, let's add the code. Import pandas as pd. Now let us run this. And now it worked. The code was correct. Actually, above I have declared so many series and other stuff. That is why it was happening. Now if I will just print po_dict, right? So what will happen? We will get table with columns, and we are also getting the rows. So these are the rows, these are the columns. These columns have the names which we have decided, and this is the prior value that we were having. And same data I added, that is why 3 different rows and 3 different columns are created. Got it? So it is super easy to understand that this works like a spreadsheet and There are few things to note when we are creating the series and when we are using series basically. So series will automatically align the value, it means automatic alignment will be there. Second, if there is any missing value, NaN will be added. Let me show you how NaN will be added. Let's say I am creating one more which is comma dance and colon pd. Let's say series, and I'm adding square bracket. Let's— I'm writing biking comma jumping is equal to index jumping, comma, index is equals to name, and name Python pandas. Now what I have done is I have created 4 series, and within those 4 series I'm trying to run the code. Now this is my second one, first one, and then fourth one also we have, right? So within these 4, what is common is 3 are same. It means 3 of them are having the same data, are having the same data. Hey, what happened? Now look at the table. Now in this table we have NaN. Why do we have NaN? Because I have not given anything for the Dow in, in the fourth one. So whenever there is any missing data, Python series or Pandas series, you can say, will automatically pick up the NaN value. So that is DataFrame. So whenever we work with the DataFrame, we are going to use these concepts. Okay, any doubt so far? Now what I will do is I'm going to use one method which is transpose method. I'm writing po.t. What will it do? It will transpose column to rows, rows to column. You can see if you want to convert it column to rows, rows to column, you can do it within just one line of 4, which is changing one form to another form. Okay, there is one more, which is, let's say I'm creating dict2, which is equals to— we have po_dict, po_dict.stack function. Now what this po_dict will do, I'm going to print the dict2. Now you see What happens? This is how it stacks. We call it the stacking method when you wanted to push the lowest column level after the lowest index. Okay, so you can see for name we have 4, for Python same, for Darwin same. So whichever exists the data and then it will automatically take it up. So there is no NaN value. NaN value are not there. So this is stacking and unstacking the label. Now the NaN values will only appear if I'm going to add more, right? This is how you can use it. Now you can use the same thing, like let's say I created dict4 is equals to po.unstack. So unstack is also the function, and if I will run this, you can see NaN X. So if we call this unstack one, we will do the reverse of it. It means if you are creating many NaN values, you will get it in the output. Okay, if you are going to again call the same thing, like in the same file, if you are going to again call this unstack, what will happen? With this unstack, you will be able to get the series object again. You can see this is object right now. If I will just change the variable to 5 and I will just write dict5 here and I will just run it, you can see you are getting the same values again. That's how it works. Okay, so these two things, the, are very important. Now you have so many different functions within Python Pandas, but I believe that those functions should be learned when we are creating the project because that time it makes more sense. So we will pick up now a dataset and we will work on that dataset so that it becomes more meaningful. And to save time, I am going to open the file in which I have the data.. I have the code cells and we will just focus on understanding. So I believe this is how you are going to save your time and learn efficiently as well. So make sure you open your Notepad or like coding code, this collab, Google Colab, and write down the code at the same time. If you are using mobile phone for watching the video, use your laptop to write down the code. So let us jump on to the code file. So, okay, so what we have done so far is we have done rows and columns. We understood that this is how we have to create the data structure. So this is a new notebook which I said that I am opening. So this is the one where you can see it is similar to what we have done so far. We created series, we created, uh, you know, uh, if you wanted to create a DataFrame using a dictionary, this is how you can create one. Now once you have created the dictionary I have my data loaded on my drive. I will show you from where I have got the data. So if you will search on Kaggle about IMDb top movies, which is Bollywood edition, and you can get the dataset from here. Now, what is Kaggle? Kaggle is a source for open source data, and if you wanted to get the data, simply go search whichever domain you are interested in, you can get it. But for now, I'm going to add this link also in the description, and you can get this particular data. You can download it from here. One CSV file will get downloaded, and we are going to work on this data. Now what I have done is I have uploaded my data on Drive. So you go open your Drive and upload it. That's it. After uploading the Drive, how do we load the data in Google Colab? Let us understand. So very first thing is you have to write down from Google Colab import drive. What do we write? From Google Colab import drive. Why importing drive? Because our data is on Drive. So when the data is on Drive, we need to first import the drive in our Google Colab. Okay, second is we have to mount that folder. It means I want that whatever content exists on my drive, I want to access it in this particular notebook. So we will write down drive.mount and then this particular location. Now my notebook, my file is basically in the folder Colab Notebooks. Your may be in the My Drive. So if your file which is movies.csv is in My Drive, do not write this, okay? You can skip this part. And if it is directly, then you can write it like this. Okay, whatever the location is, you cannot copy path or something, then you write this. And if it doesn't work, it gives you some error like Unicode error or something, write R in front of the line. I repeat, whenever you are loading the data, first mount your drive. After mounting your drive, get the location. Where is the location? So first location will be content drive, then within drive I have my drive, within my drive I have one folder collab. No notebook, and in Colab notebook I have movies.csv uploaded. Maybe you have uploaded movies.csv in your Drive or My Drive directly, not in the folder Colab notebook, so accordingly write down the path. After loading it, you have to write down display movies.head. Now this will display the first 5 rows of the data. You can see this is the first 5 rows where we have title X, ID, then poster path, then wiki link, and so on. The basic information is there. So that's how we load it in the pandas. So in pandas, you just have the function read_csv. Depending upon what is the extension of your data type or data set, we are going to use that. So for me, it is CSV, so I have written read_csv and loading the CSV. Done. Now next, next is the very first thing that we do after importing the data is we make sure we check the number of rows and number of columns. I have loaded a data which has 1629 rows and 18 columns. How many? The first value which shape function gives is rows, and the second value is the column. So that's how you can check it with the help of shape function, right? The second that we have is the dtype. Just like we have types in basic Python, we have dtype. It means data types. Data types, why? Because we have multiple columns. So for each column, it will, it will mention which data type does it contain. So it has object. Object means text. Object means text. Integer, how long integer, and how long the float data does it contain. It will write everything about the data type. Next is I want to see what is the index, because we are going to work on it, right? So index range starts from 0 and goes till 1, 6, 2, 9, and step is 1. It means only one counting— 0, 1, 2, 3, 4, like this. If we have the numbers like 0, 2, 4, 6, and so on, then the step would be 2. So this is an INDEX function to check the index. Next is to check the column. What are the columns that exist? These are the column names. So every time when we access the data, when we access the cell value, we have to use the correct column name. So This is very important. Again, next is I want to check students.value. What is students.value? Now I will show you students. So basically, before creating this data structure or loading this, I created this student data frame. Within the student data frame, I was having the values like key-value pairs I added, and the same one I am trying to print the values key value will have value, and when we have the column that is different, okay? Now I wanted to just— I want to check first 5 rows of my data movies, okay? So I will write down movies.head. Instead, by default, whenever we write head, right, this head function, whenever we write, it takes 5 values by default. But if you wanted to change it, you can, just like I change it to 3. So that I can only see which type of values do I have in my data for further analysis. Now next is info. What is info function for? Info function will give you the complete information of your data, like what is the data type. So it is a DataFrame. How many values I have? 1629. How many in, in indexes do I have? 0 to 1628. How many total number of columns do I have? I have 18 columns. What are the column names? How many non-null values do I have? How many counts? And then data. See, what is non-null? It means these many number of columns, these many number of cells are filled with the value. If you see tagline, in few data the tagline is missing. Not just few, very, right? You can see in actor there are few. So we have to look at the null values also. And what is the memory usage? What is the amount of memory it is consuming? Next is the data type. Again, how many float data type, how many integer data type, or how many object data type? So whenever we work within pandas, we have only 3 data types. Series and DataFrame are data structure. I'm talking about data type. One is float, another one is integer, and then we have object. Object means string in simple words, okay? In this float and integer also we have further divisions like 32, 64. Like this we have the values to tell that what is the maximum memory it is going to consume. Now next is the describe function. What is the describe function for? This is— this gives you the statistical summary. What does it do? It gives you statistical summary. Statistical summary means I want to know statistically, like how— what are the total number of rows? What are— what is mean? What is minimum value? What is maximum value? What is the middle value, like median? Everything related to stats, it can give you at once with just help of describe. Got it? So describe is count, mean, standard deviation, min, 25%, 55%, 75%. And this describe function, by default works on numerical column. On numerical column, how many numerical column do we have? 3 integer and 1 float. It means total 4. So it will only take 4 values. Okay, it means only take 4 columns. Now I want to check the null value in all the columns. So I have— I'm using 2 functions. First I will show you the result. See, isnull.sum. When you will just write isnull, what will happen? When we are writing just null, this is how the output gets generated. True, false, true, false, true, false. And it becomes hard for us to count the true and count the false. So we are not able to do it directly, right? So that is why we use .sum, so that it will automatically count the number of null values. So it will show you how many null values are there. So in few columns we do not have null value, but in poster path we have 103 null values. Story have 20 null values. Tagline is more than 1000. Win nominations are below 1000. So basically it gives you how and which column basically have non-null values— sorry, null values, right? If you want to check if there is any duplicated, then 0. I will show you now. See, is there any duplicate row? If there is any duplicacy, then it will just mention the true. If there is no duplicacy, again, this is difficult for us to track on or count on. So we will write down the .sum function, and we can see that that sum function will show us how many duplicacy is there. I want to rename— let's say I want to rename marks to percent and package to LP. In student, I'm talking about. Okay, let me take you back to the student. Yeah, so in student dictionary, I have name, IQ, marks, and package. Name, IQ, marks, and package. I want to rename the column name. So to rename anything, what do we have is we have this, yeah, over here. Yeah, you can use students.rename and then write down the columns, and whichever column you want to rename, use it as a key-value pair, and then once you will run, it will update. Okay, now on same data, I will perform some maths functions. So I want to calculate the sum. So when I write students.sum, what I get? I will get only the numerical value sum. It means the percentage and the LPA kind of values will be used. Got it? It will work on X. Like sum is the X's argument, and you can see this is the sum of rows. Now next is the sum X is equal to 1. Now this goes as rows. This was as column. Then by default, when you are writing sum, it will go as column. But when you are writing X is equal to 1, it goes as horizontal one. Which is as a row, correct? If you wanted to calculate mean, minimum value, maximum value, we have a direct function: .mean, .min, and then we also have .max for the maximum value, .std, which is the short form of standard deviation, and the variance also, var. Mathematically, you should definitely understand what is standard deviation and what is var, variance. Okay, now how do we select the column from the dataset? So if you wanted to select the column, so you write down the dataset variable name in whichever variable you have, you know, loaded your data, and then in the bracket write down which column do you want to select. Now this is the single column selection, and I'm writing type before this. So what is this type for? It will tell you what is the type. If you want to check what is the type of single column, then it will be considered as string only. It will be considered as series only, right? So this will be string, column name will be string, the initialization within the DataFrame will be series. So series, multiple series makes our DataFrame. Now let's say I want to do the slicing, it means I want to select multiple columns. So we do not use the colon method over here, we write down the data and within the square bracket, you create one list of all the column names that you wanted to access. So I wanted to access 3 of them, so I have written 3 column, and it will only generate the output with the 3 columns. Now there are 2 methods, as I told you, selecting the rows. It is like iloc and loc. You can use for index-based and label-based. Suppose I want to select the single row, so I can go with iloc 0. iloc 0 means only single row will be printed in this particular table format, right? And I using ILOC, I can use even the multiple columns as well. So starting from 5 to 16, I want to go— I want to go from 5th row to 16th row, but I want to skip 2. Instead of going 5, 6, 7, 8, I want to say 5, 7, 9, and so on. So you can see 5, 7, 9, 11, 13, 15. It means skipping 2. You can even skip more numbers. So let's say I want to skip 5, so only 2 to 3 rows will be printed. So 5, 10, and 15 are the 3 rows which we are getting because I'm skipping 5. I want to just view the data. Now this is fancy indexing. Fancy indexing means starting 0, 4, and then basically I want 0th column, 4th column, and 5th column— sorry, row, not talking about column, it is row. Okay, so whenever we use this, right, whenever we use ILOC, in this case this is indexed position. If you have any indexing allotted to the rows, you can use this. Okay, and in student also you can apply, like if you are creating the normal table like this, you can access LOC with the column name as well. So in this case it was Gaurav as an index, right? So that is why you can access it via LOCs. Okay, ILOC is for number, LOC is for the values. Got it? If you want to do filtering, so this is very basic, experiment a lot with the column collection selection. Okay, when you want to select rows and columns both, then you are going to use comma in between. First you will mention about rows, second you will mention about column. So when I'm writing movies.iloc , 0 to 4, it means first 4 rows and first 4 columns. First 4 rows and first 4 columns. Okay, now I want to apply some filtering. If you wanted to filter those movies which are higher than 8, which have the rating higher than 8 and votes greater than 10,000, then you're going to get it. Why am I using .shape? I will show you. There I'm getting 43. How many movies have this rating? Total 43. How I got this value. I will just remove this for now and I will just run this. Now see, if I'm going to just run it, I will get the data like this. I can see where this movie was uploaded, what is the data, and some more details about what I extracted. So those movies where the ratings are greater than 8 and movies with the highest votings are printed. But when I want to know the count, so I'm printing the number of rows. shape.column gives us what? shape.column gives us two values, rows and column columns, right? So I just want row, I just want row. So that is why 0, that is why 0. If I am going to write 1, what will be the value which I will be getting? The total number of rows, because total number of columns, because I'm not mentioning anything about column in this. I have filtered out on the basis of row, right? So that is why I am printing shape , which is 43 movies are there with this rating. Now I want to split something. I want action movies with rating higher than 7.5. So what is my genre? My genre is basically action, right? So first what I'm doing is I'm taking the— I will take you that, show you the data. Uh, let's do this one. Okay, I will just run it again. If you see this data, we have comedy, crime, and drama. But I am using a special character from my keyboard basically to split these things, right? So there is a special character. So that is why I am using string.split. So the data is string only, object data only. So I am splitting the data on the basis of this because I am looking forward to only one word which action. And now I am applying a lambda function. That is why I say lambda functions are very, very important. Act— apply lambda x action in x. It means if there is action in this text, apply it on the a variable. Now I have the a variable ready. b is saying that wherever the rating is 7.5. Now I will say movies a and b, both conditions I want. First condition is the action movies. Second condition is that rating higher than 7.5. When you have two conditions, okay, this is again important. A and B are two conditions. First condition is action movie. To get the action movie, how am I getting the action movie? I'm using the lambda function. What is lambda function doing? Lambda function is taking the complete string and checking in that string whether do we have the word action in it or not. If it is action then I am storing it in variable A. Again, let us see, we are working for action movies. To get the action movies, we are working on lambda function. We are using lambda function. What lambda will do? Lambda will take complete string, check that string. If that string contains action, it will store it in variable A. What is my second condition? My second condition is 7.5, like rating has to be greater than 7.5, and I am doing it directly. When we write down the lambda function, just like in the list comprehension comprehension format or comprehensive way, then we use apply function on pandas. We basically use the apply function to apply any function on that data, okay? And then these are the built-in— string and split are built-in within the Python, we get it, right? And then this is how you can apply two condition by using the AND operator, okay? And I'm saying I just wanted to know how many movies are there. So how many will be from shape? Shape means shape will tell us how many. Shape is for how many rows and columns, but first value is row. I just want row, that is why 0. If I want to add new column, I can add it. So let's say I want to add movie country as row, okay? And then if I wanted to drop the existing one, movies drop, drop any. If there is any drop, if there is any, any values I wanted to drop them and I wanted to make the changes in the actual data, that is why in place is equals to true. Now movies lead actor is equals to actor and splitting it and applying some again function. Now you look at, I want to see those movies where there is a lead actor, right? So I'm splitting it again and putting it in the lead actor format. Okay, that's how we can add the columns. I hope you have understood how to use this Now there are more functions, so I won't be using the movies dataset now, but I am going to show you some new methods. Now first is I am creating a list, again a list. This is a list of numbers, this is a list of alphabets, this is an array, and this is a dictionary. 4 ways, and I have the data ready right now. What I want is I want pd.Series, data is equal to list. What is list? List is the variable which I have used over here. Now I will show you, this will create a series for me, series of list, which is 10, 20, 30, 40, print. Now within this I wanted to add the labels. Now what is labels? Label, if you will go above, another variable which have ABCD. So I can give the labels like this, and I can give the labels, and I can print the labels as well. You can see ABCD is added as a index, and then we have the list. You can write down the same thing like this also, where you do not have to mention data equals to and index equals to. By default it will understand that these two parameters are for data and index. So by default, we do not have to mention data. Now, when we are creating the series, you can even use the NumPy array. So you can see I have the NumPy array over here created, which is in the arr variable, and I have created a NumPy array. Now, if I want to use the NumPy array, I can use it and create the data frame— sorry, series, right? With the labels, you can do that. This is also super easy, which we have already discussed. Now next is we can do arithmetic operation, which I already told you— addition, subtraction, and so on. And if I wanted to create a dictionary and use the dictionary, and can I— let's say I have created a dictionary which is names, mark, and city. This is my dictionary. This is my dictionary. I am converting this dictionary into DataFrame, and my dictionary is ready. Now let's see, let's talk about it. Now first thing what we did is we got the data. What we got? We got data. We did preprocess, preprocessing, and we worked on data cleaning. Let's— what is data cleaning? There are multiple steps. We worked on, uh, let's say missing data very first, right? We either deleted it, calculated, uh, basic stuff, right? We get— we got information about the data with the describe, right? We cleaned our data, we removed unwanted columns, right? So there are multiple steps that we follow within the data cleaning, and this is not fixed, this is flexible. This depends upon what type of data you have got and what is an expected output. It totally depends upon that, right? So we accordingly, we prepare the data basically. Once the data is done, we have to get the perfect data. Which type of data? Perfect data. To get the perfect data, this is my input file, right? I have to also create output file. Within that output file, I want to generate CSV file, right? Specifically because CSV data I was taking— CSV is the ideal way for training the models as well. So we use the function called to csv. read_csv is to load the file, to_csv is to give you the final file. So if you'll go and check, you have— might have received friends.csv in your drive. And then describe function will help us check what we have received. 4 marks, mean value is this, standard deviation is this, head, tail, index. I can work on it. Okay, I can work on these data. I have name, mark, city ready. I can select first 2, last 2, because I have basically exported that data. You can even create a new DataFrame with the help of NumPy library. Within NumPy library, we have a random function, and within random function, we have rand as a new function, and you can mention what do you want the function to do. I can create np.range of 3, 3, 3, 3, 3, 5. So this will contain some random integer like this, and you can generate, you can even work on it later on, you can extract it and so on. Now next concept is data alignment. What is data alignment? In pandas, data will be automatically aligned. Okay, so let's say I have one series, second series with index, and I wanted to combine them. I can combine them directly. If there is any NaN value, it will— if you wanted to handle the NaN value, you have multiple ways. Either you can add NaN, automatically it will handle NaN. But sometimes in our data you will find NaN also. NaN means not a number. None means N-O-N-E, none, right? If you wanted to check if there is a NaN value, isNull function can help. Fillna will help you to fill all those missing values with a specific value. Let's say wherever there is a null value, I want to replace it with 0. So this will help us to fill it. We just have to write df.fillna and it will just fill it. That's it. And if I'm writing dropping rows into, uh, let's say I wanted to drop those rows wherever there is a missing value, I can drop it in. I can directly go and drop it wherever. So that particular row, so these were 0, 0, right? So it means row number 2 and 1 are deleted. Only 0 and 3 exist now. So what happens is when we are filling, we are basically preserving the data. We are preserving the data for future. But when we are dropping, it is so risky. So we generally do not recommend anyone to drop anything from the data because data is very important. We are working so hard to collect the data. Now there is one library that you might have heard, which is Seaborn library. Seaborn is generally used for what? So Pandas is for data manipulation. NumPy is something that is used to, you know, create— Pandas— NumPy is for numerical computation. This is for analysis and manipulation. We have two more libraries which I would recommend you to learn, which is Seaborn and Matplotlib. Now, this Seaborn and Matplotlib will do what for you? It will help you create data visualization. To understand your data better, to understand your, uh, business context better with respect to data, we have to create some visualizations. Like you can create line chart, you can create bar chart, you can create any other charts as well. So basically you have to create some charts as well to understand your data and to do the proper analysis. Okay, so Seaborn is that one. Now what Seaborn have is Seaborn have datasets. Just like for now what I did is I went to Kaggle to download the data and I downloaded it, then uploaded it on Drive, and then from Drive I uploaded it in my IPYNB file. This is what I did. But now you do not have to do that. You have the Seaborn and you just import Seaborn. And Seaborn has built-in data type. What do we have in Seaborn? Built-in data type. What is built-in data type? Built-in data type means for practice purpose, Seaborn has so many datasets. You just have to use load_dataset as a function and you can use whichever dataset you want to use. So for now I'm using iris. Iris is like flower, and within that flower dataset I can work on it. So for now I'm not jumping directly on pandas, I'm using the Seaborn to load the data in df variable. And sns.load_dataset Iris— Iris is the name, you can write down any other. You can Google how many free datasets do we have within SNS, and you can replace this Iris with that particular dataset name. I hope this is clear. Now I will use pandas to work on it. Now I have the df created. This is how the data looks like. This is for practice purpose. You cannot create, you know, models based on this data and add it in your resume. These are not resume-friendly, but this is absolutely perfect for basic understanding and core understanding. Now I want to select the simple column— single column I can extract it, which is sepal length. I think we need to add underscore. Yeah, you can see this is a sepal length, and that was the correct name of the column. And next is the sort index. Sort index means I want to do the sorting of the index. If you look at the data we have the data like this right now. What is the condition I want to work on? X is 0, ascending is equal to false. When we are writing ascending is equal to false, it means we want to see the last data. So there are 150 rows and 1 column in this data, right? I want to view the data from 149. I do not want to view it from the top, but I wanted to view it from the last. So what I'm doing is I'm applying the SORTINDEX function and saying X is equal to 0. Which means you work on horizontal level, work on row level. And I'm saying ascending is equal to false. When ascending is equal to true, it means it goes from lower value to the higher value. When we are saying false, it means it is opposite. Now you can see over here, this is how it works. It's split. So earlier our column name was sepal length, sepal width, petal length, petal width, and species, right? But now when I said axis is equal to 1, You can see it flipped. So that's how you can flip your data and make it perfect as per your view or as per your requirement. Now next is you can apply LOC and you can change the values. You can work on the values and you can see how it goes. So you want to change, let's say, the first value, first row, first value, first row, first column, and I want to replace it with 654. Now you can see I can replace it when I am writing new df, which is a variable, then loc function, and within the bracket I have to mention the perfect location. The perfect location includes row location and column location. Just like if you will ask me what is my location, how can I tell you my location on the globe? I have to give you longitude or latitude, right? Once I will give you longitude and latitude, you will be easily able to track me on globe. Right? So same way in LOC, we need R and C both. And once you will run this, you will get the data like this, and you will see the updated data. The updated data is 654 added. Okay, you can view the top views, you can apply some functions, drop those values, whichever values are not required. And that's how we work on pandas and DataFrame. Later on, we will be creating more meaningful projects I will not jump into data analysis, data visualization part because this is purely the model creation series. So we will jump on to model creation now. So happy learning! So I hope you have learned so many different things for your next project, and you have learned about pandas, how to do data manipulation, data analysis, and data cleaning, and much more, right? So if you still have any doubts, do let me know in the comments If you still have any question or query in your mind, do let me know in the comments. Or if you are planning to do any projects, do let me know in the comments.