In this tutorial, the instructor explains how detailed visualizations of data subsets can reveal insights that summary visualizations may hide. He first illustrates the concept with stock data, showing that while annual returns appear similar across four stocks, daily price plots expose distinct yearly patterns. He then turns to a taxi dataset to investigate why tip amounts are zero for cash payments. By extracting a focused subset of trips on a popular Upper East Side route and limiting to cash and credit card transactions, he demonstrates techniques for handling overplotting (using alpha transparency) and compares fare distributions using a quantile (QQ) plot via ggplot2's geom_qq. The analysis shows that cash fare distributions are shifted and more discrete, leading to the conclusion that cash payments do not include tips in the reported total fare. Throughout, the tutorial emphasizes subsetting, visual clarity, and appropriate plot types for distribution comparison.
Chapters
We have seen that visualizing summaries can help us discover and describe overall trends and relationships in the data. In this section, we will explore some examples of how we can complement summary visualizations with detailed visualizations of smaller subsets of our data. While summary visualizations can be very revealing, sometimes important insights are covered up in the summarization and we need to look at the data in more detail to discover them. For example, here we have a summary visualization of the annual return for four stocks. From the summary, it appears all four stocks had a similar year, a return of around 13%. However, looking at detailed plots of the daily prices, we see a very different year for each stock. We'll have more fun with this stock data in chapter 3. Visualizing large data in detail is challenging because there's too much data to look at. A useful technique in this case is to take a manageable subset of the data that has some natural meaning, such as, for example, all data for one stock and visualize and explore as we saw in one of our previous exercises, the distribution of the Tip amount is 0 for all payment types but credit card. This is an interesting phenomenon that we want to get to the bottom of with cash payments. Does the taxi payment system not distinguish between TIPS and fare, or does the total fare amount just not include the amount that was tipped? To investigate this question, we turn to detailed visualization of a subset of our data. We expect rides of the same nature to have similar fare and tip amounts. Therefore, if we can pull out a subset of our data for similar routes, we can compare the distributions of fare and tip amount. To investigate our question, we expect the distributions of total fare for rides paid with cash and card to look similar if both cases include tips. Here we have extracted a subset of the data for the most popular route from the Upper east side south to the Upper east side north of Manhattan. Looking only at these trips and only at cash and credit transactions, we have about 5,000 observations. Let's do a check to ensure that this subset is well behaved. Looking at the relationship between total fare versus trip duration, we expect the relationship to be cleaner since we are focusing on one simple route. Even with data this small, we are still over plotting many points and we can alleviate this to a degree using the alpha parameter to add transparency to the points. This looks much cleaner than what we saw for all routes. To compare the distribution of payments using card versus cash, we can use a quantile plot. This displays the ordered values of the data against the quantiles of a uniform distribution and is often more useful than a histogram for comparing distributions. We create a quantile plot using ggplot2's geomqq, specifying that the data should be plotted against the uniform distribution. In this plot, we see that the card and cash distributions have a similar shape but are shifted. We also see that the cash payments are made up of several discrete values, while card payments are more continuous, which we wouldn't be able to see in a histogram due to binning. From this, we can reasonably conclude that that tips are not included in the total reported fare amount for cash payments in the exercise. We will see if the two distributions are similar if we remove tips from both. Let's go.
