In this instructional video, the presenter demonstrates how to import only a portion of a large flat file into a pandas DataFrame, using a U.S. tax dataset as a running example. The tutorial covers four main techniques: (1) selecting specific columns with the `usecols` argument (by name, index, or a custom function), (2) limiting the number of rows read with `nrows`, (3) processing a file in chunks by combining `skiprows` and `nrows` while handling the header row appropriately, and (4) assigning column names manually via the `names` parameter when the header is omitted. The speaker shows how to verify that different import methods yield identical results, how to construct a list of column names from an existing DataFrame, and how to combine these options to efficiently load manageable subsets of a large dataset. The lesson concludes by encouraging viewers to practice these workflows for better memory management and data handling.
Chapters
You now know how to load a flat file into a dataframe. Sometimes, though, you only want part of the file. In this lesson, we'll look at ways to limit the amount of data imported and how to make that data easier to work with by naming columns let's revisit the U.S. tax data. Checking the data's shape attribute, we see that it has almost 180,000 rows and 147 columns. Unless your computer has very little memory or is running several other programs, Pandas can easily handle data this size, but it is more than we need. We have some options to whittle it down. The first way is to choose columns to load with the Usecols keyword argument. Usecalls can take a list of either all column names or all column numbers to import. It can even accept a function to select columns so that you can say import all columns starting with the letter n. But this is a more advanced technique to focus on the number of tax returns. By incumband and zip code, we can set up a list of either the relevant column names or their numbers. We supply the file path as a string to read CSV as usual. This time we also pass Usecols the list of column names or numbers to load. We can check to make sure the two ways produce the same result. And they do. Another option is to reduce the number of rows imported with the NROWS argument. When reading the file, we pass an integer of the maximum number of rows we want 1000. Here, checking the dataframe's shape, we see we have exactly the number of rows we asked for. NROWS is especially handy when combined with the skiprows argument to process a file in chunks. Skiprows accepts a list of row numbers to skip a function to determine whether to skip a row or the number of rows to skip. Note that Pandas automatically makes the first row imported the header. So if you skip the row with column names, you should also specify that header equals none. Let's get rows 1000 to 1500. Like before, we use NROWS to specify how many rows we want, but add that skiprows equals 1000. We skipped the header row, so we also specify there isn't one. Here, checking the head of the dataframe, we see there are no column names. Let's fix this. To assign column names when there aren't any, we use another readcsv argument. Names as you probably guessed, names takes a list of column names to use. The list must include a name for every column in the data. If you only want to rename some columns. It should be done after import. For datasets with fewer columns, you might manually build the names list with the help of a data dictionary. We want to copy a long list of column names from an existing data frame, so we'll pass the first tax subset to the list function to get the column labels. Then we read the CSV using Nrows and skip rows to get the next 500 records, setting header to none and passing the list of column names to names. Now our dataframe has column names. Whew. You just learned workflows to manage the amount of data loaded from a flat file. Now you'll get a chance to practice those techniques.
