Missing values in loan data can bias predictive models or cause training errors. Learn to detect them with pandas isnull and sum functions, then either replace with mean/median, drop rows, or keep them if they signal something real like processing delays.
Chapters
With the outliers now removed from our data set, we can focus on another problem with credit data, and that is when data is missing. Normally you might think of missing data as when an entire row is missing, but that is not the only way data can be missing. Data can be missing when there are null values in place of actual values. It can also be an empty string instead of a real string. For this course, we will refer to missing data as when specific values are not present, not when entire rows of data are missing. Any of the columns within our data set can contain missing values. If we see a row of data with missing values in a PANDAS dataframe, it will look something like this. Notice for employment length we see NAN or not a number instead of a value. One issue with missing data is similar to problems caused with outliers in that it negatively impacts predictive model performance. It can bias our model in unanticipated ways, which can affect how we predict defaults. This could result in us predicting a large number of defaults that are not actually defaults because the model is biased towards defaults. Also, many machine learning models in Python do not automatically ignore missing values and will often throw errors and cease training. Here are some examples of missing data and possible results. If there are null values in numeric or string columns, the model will throw an error. So how do we handle missing data? Most often it is handled in one of three ways. Sometimes we need to replace missing values. This could be replacing a null with the average value of that column. Other times we remove the row with missing data altogether. For example, if there are nulls in loan amount, we should drop those rows entirely. We sometimes keep missing values as well. This, however, is not the case with most loan data. Understanding the data will direct you towards one of these three actions. For example, if the loan status is null, it's possible that the loan was recently processed in our system. Sometimes there is a data delay and additional time is needed for processing. In this case, we should just remove the whole row. Another example is where the person's age is missing. Here we might be able to replace the missing age values with the median of everyone's age. But how do we find missing data? With pandas? We can find missing data like nulls using the Isnull function and the sum function to count the rows with data missing. By combining the function isnull sum and any, we count all the null values in each column. This produces a table of values that shows the count of records with nulls in the data. If we decide to replace missing data. We can call the FILNA method from Pandas along with aggregate functions. This will replace only missing values. In this example, we replace null interest rates with the average of all interest rates in the data. The result, as shown here, replaces a null interest rate with 11%. Dropping rows with missing data is just like dropping rows with outliers. Like in the previous video, we use the drop method from pandas. Here we find the rows with missing data using isnull and then drop the rows from the data set entirely. So.
