Python Tutorial: Thinking about Data in Chunks

We've seen that available memory and storage restricts data sets that can be analyzed. A common strategy is to subdivide data sets into smaller parts. We'll use a 200,000 line file summarizing new York City cab rides for the first two weeks of 2013. Then, using readcsv with the parameter chunksize50000, the function returns an object we can iterate over. The loop variable chunk takes on the values of four successive data frames, each having 50,000 lines except the last, because the first line in the file is the header line. The loop variable chunk has standard dataframe attributes like shape, so the last chunk has almost 50,000 rows and 14 columns. Calling the info method shows the column names like trip time in sex and trip distance. We can construct a logical series islongtrip that is true whenever the trip time exceeds 1200 seconds or 20 minutes. Recall, we can use the LOC accessor with the Boolean series islongtrip to filter rows where this condition holds. The shape attribute reveals about 5,500 taxi rides longer than 20 minutes in duration for from this chunk of 50,000 trips. Let's embed this filtering logic within a function filterislongtrip that accepts a dataframe as input and returns a dataframe whose rows correspond to the trips over 20 minutes. Next, we make a list of data frames called chunks, by iterating over the output of readcsv, this time using chunks of 1000 lines. Rather than initializing an empty list chunks and appending elements within a loop, we can also use a list comprehension to build the list. Remember, this list comprehension is equivalent to the preceding for loop. In both cases, each chunk is filtered as it is read from disk. We can use another comprehension called lengths, to see that the data frames in the list chunks each have around 100 to 200 rows rather than a thousand rows in the unfiltered chunks. The PANDAS function concat accepts this list of dataframes with common column labels and stacks them vertically. The resulting data frame, LongTripsDF, has almost 22,000 rows, much fewer than the original 200,000. Finally, we can visualize these trips. The result looks something like this. We generate the last plot with this code. We start by importing matplotlib pyplot and constructing a scatterplot using plot scatter, we apply labels and display the plot with plot show. Remember, we used filtering or logical indexing to extract the small subset of relevant data in manageable chunks. The entire dataset was never in memory at one time. Take some time now to practice reading and filtering files in chunks in the exercises.