Learn how density plots use kernel density estimation to visualize data distribution. Bandwidth controls smoothness: 0.69 shows a smooth curve, while 0.4 reveals more peaks. ggplot2's geom_density automatically truncates at data limits, changing the area under the curve.
Chapters
Density plots are an excellent way of visualizing the distribution of univariate data sets. They are less common than histograms and somewhat poorly understood, but actually they're quite intuitive. If you've ever looked at a statistics book, you've already seen plenty of density plots in the form of probability density functions. So called PDFs. Things like the normal, the T, the chi squared and F distributions, among many others, are simply theoretical PDFs. Here we're going to make an empirical PDF. So instead of making a density plot based on a formula, it's going to be based on our data. But the purpose is basically the same. To do this we need to use a kernel density estimator, the so called kde. Everett and Hawthorne give a definition which explains this pretty well. They describe the KDE as a sum of bumps placed at the observations. The kernel function determines the shape of the bumps, while the window width H determines their width. To understand this, we'll focus on a simple data set that Everett and Hawthorn used as an example. The data set consists of eight values which are shown here as green vertical lines. This was done using the RUG geome, which produces a strip chart in the margins of two dimensional plots. The KDE calculates a normal distribution for each value. These are the bumps placed at each observation that Everett and Hawthorne refer to. To obtain the true density curve, we simply add up all the Y values for each bump along our X axis. Many overlapping lines will result in a higher value, ergo a higher density. The plot shows us exactly what it describes. Regions of high and low density. Thus the empirical probability density function. It does exactly the same thing that a theoretical PDF does. For example, recall that in a normal distribution, the area under the curve equals one and intervals represent the probability of finding values in that specific range. So here we can also get an idea of where our data cluster on the scale. Just as an aside, this is how we determine the mode of a continuous variable, or indeed how many modes we have. So that's what we mean by the shape of the bumps. But what about the bandwidth H? There are several ways of choosing the bandwidth, and a good general rule of thumb is simply use the default. We won't get into the math, since there are actually many variations on this. And we'll get back to it into the exercises. For the most part, you won't need to change the bandwidth, but if you want to see more subtle features in your density plot, you can decrease it. For example, this plot has a default bandwidth of 0.69. If we reduce this to 0.4, we get a density function with more features. The reason is that the individual bumps are narrower and so they have less overlap and our curve is not as smooth. As a consequence, the peaks are higher, which we can understand as operating just like a normal curve with a small sigma. It's not to say that one version is correct and the other is false. Remember, there are many ways to calculate the bandwidth, and not only that, but the normal KD is just one of many kernels. Density plots are just representations of the underlying distribution, so it's valuable to know how the bandwidth was chosen, if it was manually set and what it is. So far we've drawn density plots with the underlying intermediate steps. Notice that this means that the red density curve extended beyond the limits of our data. That's completely expected given the underlying mathematics, but it's also somewhat off putting and confusing, especially if you imagine the density plot extending beyond the natural limits of the variable under consideration. When we only use the GEOM density function in ggplot2, our plot will look slightly different. The geomdensity function cuts off the curve at the minimum and the maximum values of our data set marked here with blue lines. The shape of the distribution is the same, but because the limits have changed, the area under the curve is no longer one. This is true for any bandwidth that you set, and although it avoids confusion about the range of the density plot, the area under the curve is no longer one. Alright, let's head over to the exercises and see all of this in action.
