Python Tutorial : Clean and Validate

In the previous lesson, we read data from the National Survey of Family Growth and selected a column from a data frame. In this lesson, we'll check for errors and prepare the data for analysis. We'll use the same data frame we used in the previous lesson, nsfg, which contains one row for each pregnancy in the survey. I'll select the variable birthweightlb1, which contains the pound part of birth weight, and assign it to pounds and birth weight. Oz 1 contains the ounce part of birth weight, so I'll assign that to ounces. Before we do anything with this data, we have to validate it. One part of validation is confirming that we are interpreting the data correctly. We can use value counts to see what values appear in pounds and how many times each value appears. By default, the results are sorted with the most frequent value first, so I use sort index to sort them by value instead, with the lightest babies first and heaviest babies last. As we'd expect, the most frequent values are 6 to 8 pounds. But there are some very light babies, a few very heavy babies, and two values, 98 and 99, that indicate missing data. We can validate the results by comparing them to the codebook which lists the values and their frequencies. The results here agree with the codebook, so we have some confidence that we are reading and interpreting the data correctly. Another way to validate the data is with describe, which computes summary statistics like the mean, standard deviation, min and max. Here are the results. For pounds count is the number of values. The minimum and maximum values are 0 and 99, and the 50th percentile, which is the median, is 7. The mean is about 8.05, but that doesn't mean much because it includes the special values 98 and 99. Before we can really compute the mean, we have to replace those values with NAN to represent missing data. The replace method does what we want. It takes a list of values we want to replace and the value we want to replace them with. NP NAN means we're getting the special value NAN from the NUMPY library, which is imported as np. The result from replace is a new series, which I assign back to pounds. Remember that the mean of the original series was about 8.05 pounds. The mean of the new series is about 6.7 pounds. It makes a big difference when you remove a few 99 pound babies. Instead of making a new series, you can call replace with inplace true, which modifies the existing series in place, that is, without making a copy. Here's what that looks like for ounces. Since we didn't make a new series, we don't have to assign it back to ounces. Now we want to combine pounds and ounces into a single series that contains total birth weight. Arithmetic operators work with series objects, so to convert from ounces to pounds, we can divide by 16 there are 16 ounces in a pound. Then we can add the 2 series objects to get the total. Here are the results. The mean is about 7.1, which is a little more than what we got before we added in the ounces part. Now, we're close to answering our original question, the average birth weight for babies in the US but as we'll see in the next lesson, we're not there yet. First, let's practice what we learned in this video.