In this instructional video, the presenter walks through a complete workflow for dealing with missing values in R. The narrative begins by emphasizing the importance of exploring and visualizing missingness before consulting stakeholders about business rationale. Three overarching strategies are presented: discarding data with high missingness (used sparingly), imputing missing values, and using methods that can handle missingness directly. The speaker introduces the naniar package, demonstrating functions such as anyna, vismiss, ggmisscase, and the creation of a shadow matrix to identify and visualize missing patterns, including clustering rows by missingness. The video explains the three classic missingness mechanisms—MCAR, MAR, and MNAR—and notes their implications for bias when choosing a strategy. Evaluation of imputation quality is covered via two approaches: external evaluation (building a machine‑learning model on the imputed data and comparing performance) and internal evaluation (comparing pre‑ and post‑imputation distributions of means, variances, and scales). The simputation package’s imputelm function is showcased for linear‑model based imputation, and bindrows is used to combine results from multiple imputation models into a single tidy data frame. Throughout, the presenter stresses practical considerations such as minimal loss of data, checking for bias, and visual cues that can hint at the missingness type.
Chapters
In this video, we're going to answer two fundamental questions around dealing with missing values in your data. An important question is what to do with your missing values. First, explore your data set, identify its missingness and visualize it. Then talk to your client about these missing values and see if there's any business rationale that can be applied to deal with them. In general, there are three main ignore them by discarding samples or variables with high levels of missingness. Use this option sparingly as this often leads to loss of valuable data. Impute them. This means replace them with other, hopefully more meaningful values and third, accept them and proceed to choose methods that naturally deal with these missing values. Unfortunately, not many methods can do that. This strategy depends on the type of missingness you have. The Naniar package provides many usable functions to identify, visualize and deal with missing data. Any NA will tell you if there are missing values in your data frame. Sometimes you need to manually replace one or more missing data symbols with an ace. As done here, you can easily summarize the level of missingness across variables and instances in your data set. Here we see that five out of six variables have missing values. Visualizing the missing values in a data frame is very easy. Just invoke the VISMISS function. You can optionally arrange the rows according to their missingness with cluster equals true. The ggmisscase function displays the missingness at the row level or cases. In this example, only a very small fraction of the observations have two or more missing values. There exist three types of missing data missing completely at random or MCAR missing at random or MAR and missing not at random or M NAR. To understand these patterns of missingness better, check chapter 2 of this data camp course. Each missingness type has its own implications when it comes to performing imputation or deletion, as bias could be introduced in the data by doing so. There are also some visual cues related to the missingness clusters that may help identify the type of missingness in the data. Although this is not a bulletproof resource, evaluating the quality of an imputation is also an important issue. You can do that in two different ways. An external evaluation relies on building a machine learning model from the imputed data set, then assessing its performance as a function of the imputation method alone. All else being equal, this should give us a good indication of how beneficial that imputation was to our entire machine learning pipeline. An internal evaluation compares the distributions of the variables before and after the imputation in terms of their mean, variance and scale. Ideally, you want an imputation model that does not drastically change the distribution of the imputed variables or or their relationships. Big changes in these indicators could signal a problem with imputation. NANIAR allows easily constructing a shadowed matrix. This is a data structure with new columns labeled after the original column names, but with NA appended to them. These extra columns indicate whether a variable value was missing or not. This feature makes it very easy to track imputed values. The simputation package provides the IMPUTELM function to impute the values of a dependent variable as a linear function of the values of the independent variables. We use bind rows to aggregate the information from multiple imputation models into a single data frame that looks like this. Okay, we're almost ready to visualize.
