Machine learning engineer Jern Buje introduces dimensionality reduction using a Pokemon dataset. He shows how to inspect data with Pandas, check variance across columns, and drop features like `generation` that have zero standard deviation. Understanding feature properties before applying advanced techniques is essential.
Chapters
Hi there. My name is Jern Buje and I'm a machine learning engineer at Faction. In this course, I'll be teaching you how to reduce dimensionality in your datasets. Before we get going, it's important to clarify some concepts. When I mention dimensionality, I mean the number of columns you have in your data set, assuming that you're working on a tidy data set, such as the Pokemon dataset shown here. Working with a tidy dataset makes your life as a data scientist much easier to do. So just make sure every column resembles a variable or in other words, feature such as the properties of Pokemon like Attack and Defense in this example. Similarly, each row holds an observation for each variable. The first row gives us all the properties of one specific Pokemon. You end up with a value in each cell of our dataset. You can learn the number of rows and columns in a Pandas dataframe using the shape attribute. For our small Pokemon dataset, shape will return a tuple, telling us there are five rows and seven columns. Throughout this course, we'll be using Pandas a lot and will always pre import it for you as pd. When you have many columns in your dataset, say more than 10, the data is considered high dimensional and if you're new to that data set, it could be hard to find the most important patterns because of the complexity that comes with high dimensionality. To overcome this, we can reduce the number of columns using dimensionality reduction techniques. However, these techniques can also be useful for relatively low dimensional datasets such as this one in the dataset shown here. You'll notice that all Pokemon come from the same generation. If we're interested in how Pokemon are different, this feature would not be very useful since it has no variance. We could reduce dimensionality by simply dropping that column. To get a quick overview of which columns contain little variance, we can use the Pandas dataframe describe method. By default, Describe gives us a number of summary statistics for each numeric column. You'll see that for our small data set, the standard deviation of the generation column is zero and that the maximum and minimum values are the same. By default, Describe ignores the non numeric columns in the dataset. We can tell Describe to do the opposite by passing number to the exclude argument. We'll then get summary statistics adapted to non numeric data. We can see that in the five observations there are only two unique types of the top occurring type is grass since it's in the type column three times. Getting comfortable with the shape of your dataset and the properties of the features within is a crucial step you should take before you make the decision to reduce dimensionality. Another important exploration step is to visualize your data. And we'll get to that in the next lesson. But first we'll practice. It's your turn to explore a larger sample.
