Python Tutorial: Why generate features?

Hello and welcome to Feature Engineering for Machine Learning in Python. My name is Robert o' Callaghan and I'm a data scientist. Feature engineering is the act of taking raw data and extracting features from machine learning. Most machine learning algorithms work with tabular data. When we talk about features, we're referring to the information stored in the columns of these tables. For example, if we were looking at information on houses, the features will be things like square foot, number of rooms, etc. This course is designed for data scientists who want to expand their knowledge of how to incorporate feature engineering into the data science workflow. Most machine learning algorithms require their input data to be represented as a vector or a matrix, and many assume that the data is distributed normally in the real world. More often than not you will receive data that is not in this format. You will also need to work with many different types of data. Some data types you will often encounter continuous variables, categorical data, ordinal data, Boolean values, and dates and times. Dealing with these is manageable but requires a well thought out approach. Feature engineering is often overlooked in the machine learning discussions, but any real world practitioner will confirm that data manipulation and feature engineering is the most important aspect of the project. Over the span of this course, we will be addressing how to deal with many different types of data and how to convert them into a format that can be easily used for machine learning. In the first chapter you will ingest and create basic features from tabular data. In the second chapter you will learn how to deal with data that has missing values. You will then move on to transforming your data so that it conforms to statistical assumptions often necessary for machine learning models. And finally, you will convert freeform text into tabular data so that it can be used with machine learning models. Now let's jump straight in with some examples. During this course we will be leveraging the Pandas package substantially as it is very useful when working with data in tabular form. It is common practice to import Pandas using the PD alias. You can use the readcsv function to import a CSV file and use the HEAD method to quickly look at the first few rows of the data frame. For the first three chapters of this course you will be working with a modified subset of the Stack Overflow survey response data. This data set records the details and preferences of hundreds of users of the Stack Overflow website. To see the features used in the subset, you can use the columns attribute to print the names of all the columns in the dataframe. To print the data type of each column, you can use the dtypes attribute. Here you can see three different data types integers, floats, and objects. In Pandas, objects are columns that contain strings. Knowing the types of each column can be very useful if you are performing analysis based on a subset of specific data types. To do this, you can use the selectdtypes method and pass a list of relevant data types to the include argument. For example, if you want to select only the integer columns, call the selectdtypes method on DF and set the include argument to int. Let's get right into it and start practicing.