Python Tutorial: Handling errors and missing data

So far, we've imported flat files with minor tweaks to set column names and manage the amount of data loaded. This is enough if the data is already in great shape, but what if there are issues with the data or the import? Common issues include incorrect column data types, which can hinder analysis, missing values indicated with custom designators, or records that Pandas cannot read. Luckily, readcsv offers ways to address these issues during import, reducing the wrangling needed later on. When importing data, Pandas infers each column's data type. Sometimes it guesses wrong. Checking the data types in the tax data, we see that Pandas interpreted zip codes as integers. They're more accurately modeled as strings, though zip codes are not quantities and include meaningful leading zeros. Instead of letting Pandas guess, we can set the data type of any or all columns with Readcsv's DType keyword argument. DType takes a dictionary where each key is a column name and each value is the data type that column should be. Note that non standard data types like Pandas categories must be passed in quotations. Here we specify that the zip code column should contain strings, leaving Pandas to infer the other columns printing the new DataFrame's DTypes, we see that Zipcode's DType is object, which is the Pandas counterpart to Python strings. Missing data is another common issue. Pandas automatically recognizes some values like na or null as missing data, enabling the use of handy data cleaning functions. But sometimes missing values are represented in ways that Pandas won't catch, such as with dummy codes in the tax data. Records were sorted so that the first few have the zip code 0, which is not a valid code and should be treated as missing. We can tell Pandas to consider these missing data with the Navalues keyword argument. Navalues accepts either a single value, a list of values, or a dictionary of columns and values in that column to treat as missing. Let's pass a dictionary specifying that any zeros in zip code should be treated as missing data. Then we filter the data using the ISNA method on the zip code column to view rows with missing postal codes. One last issue you may face are lines that Pandas just can't parse. For example, a record could have more values than there are columns like the second record in this corrupted version of the tax data. Let's try reading it by default, trying to load this file results in a long error and no data is imported. Luckily, we can change this behavior with two error bad lines and warn bad lines. Both take boolean or true false values. Setting error badlines to false makes pandas skip bad lines and load the rest of the data. Instead of throwing an error warn, badlines tells pandas whether to display messages when unparsable lines are skipped. Let's try importing the corrupted file again, this time with error bad lines set to false and warn bad lines equal to true success. A word of caution. If lines were skipped, it's worth investigating what was left out to see if there are underlying issues that should be addressed. In this video, you've learned about common issues encountered importing data from flat files and how to handle those issues. Now it's your turn to practice.