Lisa Stewart shows how to handle missing data in machine learning using omission or imputation. She demonstrates scikit-learn's SimpleImputer and IterativeImputer with strategies like mean, median, and mode, and explains how each choice affects bias and variance. Improper handling introduces bias, so the goal is to minimize impact on the data's original distribution.
Chapters
Welcome to Preparing for Machine Learning Interview Questions My name is Lisa Stewart and I am a data scientist. In this course, I will cover most of the topics that will help you succeed in a machine learning interview using Python. Being an interview prep course, you will find the concepts and exercises more challenging than standard data camp courses, so please ensure that you're comfortable with the prerequisite courses, supervised learning with SciKit learn and unsupervised learning in Python. We're going to take what you learned so far and step it up so that by the end you'll set yourself apart from other potential candidates in a machine learning interview. In this course, we'll start off chapter one by covering data pre processing and visualization. The second and third chapters will be dedicated to supervised learning and unsupervised learning, and the fourth and final chapter will touch on model selection and evaluation from a high level the machine Learning pipeline using SciKit Learn looks something like this. You import the modules you need to use, instantiate an object which you then fit and predict. But there is more to the story, so the pipeline we are going to use incorporates other important steps. Don't worry about the details, we'll start slow and build as we go, continually orienting ourselves to where we are in the process. In the remainder of this video lesson we're going to discuss how to find missing values as well as the impact of different techniques designed to fill missing data as a pre processing step in the machine learning framework. This is an integral part of exploratory data analysis. You should always begin with the two most commonly used strategies are omission, involving removal of rows and or columns, and imputation, which includes filling missing observations. You'll use two functions from scikit Learn's impute module. Simpleimputer fills with zeros the mean, median or mode by supplying a value to the strategy Keyword argument and iterativeimputer imputes by modeling each feature with missing values as a function of other features. So why bother trying different imputation techniques when data is missing? Because how you handle missing values can introduce bias. Handling it appropriately will reduce that probability and perhaps most importantly, most machine learning algorithms require complete data or an error is generated. The exact effect an imputation technique will have on the distribution of a given feature depends on factors that include the missing values, the original variance, if there are any outliers, as well as the size and direction of skew. General guidelines are that removing rows and or columns can result in removing too much data. Filling with zeros tends to bias the results downward and median tends to be a better choice when there are outliers. Mode and iterative have varying degrees of helpfulness, so it's best to just give them a try. Here are a few more functions you'll use in the exercises. Combining isna with the sum function finds the number of missing values using bracket subsetting with an aggregate function such as mean returns the mean shape gives row and column dimensions, while fillna fills missing values with the argument passed to it by specifying include np number, select dtypes returns numeric columns or string columns for object. Finally, FitTransform fits and transforms the numeric columns passed to it before we move onto the exercises, here is a multiple choice question for you. What are the effects of missing values in a machine learning setting? If the answer is not immediately apparent, pause this video to read through the possible answers and give yourself a moment to think about it. If you still aren't sure, consider re watching the video lesson up to this point before revealing the answer in the next slide. The answer is that missing data introduces bias, so filling missing values by testing which impacts the variance of a given data set the least is the best approach. These are the reasons why the other answers are incorrect. Make sure you understand them. Time to handle missing data.
