Python Tutorial : Dataframes and Series

Welcome to Exploratory Data Analysis in Python. I'm Alan Downey and I'll be your instructor. The goal of exploratory data analysis is to use data to answer questions and guide decision making. As a first example, we'll start with a simple what is the average birth weight of babies in the United States? To answer a question like this, we have to find an appropriate data set or run an experiment to collect it. Then we have to get the data into our development environment and prepare it for analysis, which involves cleaning and validation. For this question, we'll use data from the National Survey of Family Growth, which is available from the national center for health statistics. The 2013-2015 data set includes information about a representative sample of women in the US and their children. The Python module we'll use to read and analyze data is Pandas, which we'll import as PD. Pandas can read data in most common formats, including CSV, Excel, and the format the NSFG data is in HDF5. The result from ReadHDF is a DataFrame, which is the primary data structure Pandas uses to store data. Head shows the first five rows of the DataFrame, which contains one row for each pregnancy for each of the women who participated in the survey, and one column for each variable. The data frame has an attribute called shape, which is the number of rows and columns. There are 9,358 rows in this data set, one for each pregnancy, and 10 columns, one for each variable. The dataframe also has an attribute called columns, which is an index. That's another Pandas data structure, similar to a list. In this case it's a list of variable names, which are strings. Based on the names, you might be able to guess what some of the variables are, but in general you have to read the documentation. This figure shows an entry from the codebook for birth weight LB1, which is the weight in pounds of the first baby from this pregnancy for cases of live birth. In many ways a dataframe is like a Python dictionary, where the variable names are the keys and the columns are the values. You can select a column from a dataframe using the bracket operator with a string as the key. The result is a series, which is another Pandas data structure. In this case, the series contains the birth weights in pounds of the live births or in the case of multiple births. The first baby head shows the first five values in the series. The name of the series and the data type float 64 means that these values are 64 bit floating point numbers. Notice that one of the values is NAN, which stands for not a number. NAN is a special value that can indicate invalid or missing data. In this example, the pregnancy did not end in live birth, so birth weight is inapplicable. Let's start exploring this data by working on some exercises.