Learn how to convert string categories into pandas' category dtype for faster machine learning. The instructor shows astype('category') to create numeric representations, pd.getdummies to build dummy columns, and lambda functions to apply categorical conversion across multiple columns at once.
Chapters
We've seen we have some numeric values and some text values in our data set. It's common to have data where each value is from a known set of categories. For example, a column season of year may have the values winter, spring, summer, and fall. These kinds of data are not simply strings. Let's look at our sample data set again. If we look at the label column, we can see that it takes either the value A or the value B. We can treat these kinds of variables in a way that solves two problems for us. The first problem is that our machine learning algorithms work on numbers. We need a numeric representation of these strings before we can do any sort of model fitting. The second problem is that strings can be slow. We never know ahead of time how long a string is, so our computers have to take more time to process strings than numbers which have a precise number of bits. In Pandas, there's a special data type called category that encodes our categorical data numerically. And because of this numerical encoding, it can help speed up our code. In Pandas, we can call the AS type function with the string category. To change a column's type from object to to category, here are two rows from the label column in the sample data frame when the data is loaded. Pandas assumes that these variables are strings, so the dtype is object. By calling as type category, we are returned a categorical variable. As we can see, Pandas is already smarter about the values that appear in the column. In this case, the two values A and B. We have actually converted these strings into a numeric representation of the categories. To see this numeric representation, we can use the getdummies function. In Pandas, this is called getdummies because this process is called creating dummy variables. Our dummy Variables DataFrame has two columns. The first if the value is labeled A. The second if the value is labeled B. Each row contains a 1 if that row is of that category, and a 0 if not. This is also called a binary indicator representation. Note that the prefixsep parameter is useful to tell git dummies what character should separate the original column name and the column value for our dummy variable. Before we create categorical representations of our budget data, we want to mention lambda functions, a feature of the Python language. Instead of using the DEF syntax you may have seen before. Lambda functions, let you make simple one line functions. For example, we may want a function that squares a variable. We can define a lambda function that takes a parameter, the variable X. The function itself just multiplies x by x and returns the result. We can call this function just like any other Python function, and it returns whatever the one line of code evaluates to. In the sample data frame, we only have one relevant column called label, but you'll remember from looking at the budget data that there are multiple columns. We want to make categorical to make multiple columns into categories, we need to apply the function to each column separately. We will use a small lambda function to convert each column to a category. We then use the Apply method on a PANDAS dataframe to apply this function to each of the relevant columns separately by passing the parameter axis equals zero. Now it's your turn to use as type category to encode categorical variables in our actual data set to help speed up the models that you're going to use on your data.
