R Tutorial: Detecting anomalies in data

Dealing with anomalous data points, also known as outliers, is an important step in the data preparation phase. If not properly handled, outliers could skew your analysis and produce misleading conclusions. In this video, you will learn how to detect and treat outliers in your data. To detect outliers in the distribution of a single numerical variable, you can resort to two simple the three sigma rule for normally distributed data and the more general 1.5 IQR rule. In a multivariate setting, there's a plethora of methods to choose from. We will highlight two of the k nearest neighbor's distance and the local outlier factor. If your variable is normally distributed, the three sigma rule can be applied. In this case, you can label any observation three standard deviations above or below the mean as an outlier, given that its probability will be less than 0.1%. Another popular method is the 1.5 IQR rule. This method can be applied regardless of your underlying data distribution. The Interquartile range, or IQR, is the difference between the third and the first quantiles of your variable. Any observation below Q1 1.5 IQR or above Q3 1.5 IQR is considered an outlier. Multivariate methods can be broadly categorized as either distance based or or density based. They both rely on the reasonable assumption that anomalous observations frequently lie far from their neighbors. Distance based methods often use the average distance to their k nearest neighbors as an indicator of the extremeness of the observation. Density based methods operate by counting the number of neighboring points within a predefined reachability distance. This plot shows some potential outliers along the fuel consumption and distance axes. We can expect they will have a larger average distance to their neighbors compared to those points that appear clumped together. We will use the get knn function from the FNN package to efficiently compute the distance matrix for the knee. Distance based methods fail to properly detect outliers in regions exhibiting different spatial densities. These are called local outliers. For example, while the data point in the middle top of the figure could be regarded as a global outlier, there might exist multiple local outliers in different regions. The local outlier factor, or lof, is an algorithm that measures the local deviation of a data point relative to its neighbors. Outliers are defined as data points with substantially lower density than their neighbors. Each observation receives an LOF score that indicates whether it is deemed to be a regular data point, an inlier, or an outlier. We will use the LOF function from the DBSCAN package to compute the LOF score. After detecting the outliers, we can proceed in several ways. First, try to retain them in your data by choosing algorithms that are robust to them. If this is not possible, you could try applying data imputation methods such as linear or KNN imputation. Hopefully, they will replace the outliers with less extreme observations. Another option is capping the outliers by replacing the low values with the fifth percentile of the variable distribution and the high values with the 95th percentile. Excluding the outlier observations from your data analysis pipeline is also possible, although this is often frowned upon and definitely not recommended for small data sets or those whose distribution is not Gaussian. In any case, use your domain knowledge to understand why these extreme observations are happening in your data set. Okay, let's.