Clustering (part 4/4: characterizing clusters)

Now that we have seen how to construct classes of individuals, let's take a closer look at detecting and describing common features of individuals put in the same class. To start with, we can look for the model individual, that is, the one closest to the center of the class. Why are we particularly interested in this one? Well, because the actual center of gravity of the class is likely to be a point in space rather than exactly on top of one individual. It's clearly a better idea to use a real individual to understand the average features of a class. Here, for example, we see that in class 1, it's Oslo that's the closest to the center. It's at a distance of 0.34 from that class's center of gravity. So, to look at how the class is on average, we can take a closer look at Oslo. In class 2, it's Berlin that's the closest to the center, and for class 3, Rome. People who have done a lot of data analysis already know that these model individuals will be quite useful to us. Here we show how we can put these model individuals on the factor plane. Here is the center of gravity of class 3, and here the model individual Rome. This characterization of each class using the closest individuals to the center is helpful but not really enough to say a lot. We'd also like to describe class— classes using the variables. The goal is to find the variables that most influenced overall how the partition was formed, or the variables that best represent a given class. In terms of questions, we therefore have: what variables most influence the partition? What variables best describe or represent individuals in class 1. The partition, that is, the division of the individuals into the various classes, can be seen as a qualitative variable with as many categories as there are classes. Thus, looking for the variables that best characterize a partition really just means looking for the variables that best characterize a qualitative variable. For each quantitative variable, we can build an analysis of variance model for it, which will play the role of response variable, as a function of the class variable, which will play the role of covariate. In summary, we do an analysis of variance of the quantitative variable as a function of the class variable, and run an F-test to see if the class variable has an effect on the quantitative one. One way to proceed is then to keep variables with p-values below 5%, and sort them by increasing p-value. In our example, we see that the variable that best characterizes the classes is October. It has the smallest p-value and best separates the classes. But be careful, remember that, like when we described the dimensions in PCA, we have to use these tests wisely, because the October variable plays an active role in the construction of the classes, as it's part of the distance calculations between individuals. To be rigorous, only the p-values associated with the supplementary variables of north, south, east, west can be legally interpreted in the usual way. But the p-values for the internal variables are still useful, Even if, if they can't be used like in classical tests. An F-test just helps us find variables that more or less influence the partition. Next, let's try to understand what each class can be defined or described by. Which variables are most important for each class? This plot helps us to visualize the data. Each row corresponds to a variable, each point to a city. The black points represent Helsinki to Stockholm, the red points Amsterdam to Sofia, and the green ones Athens to Rome. Now, how can we see if there are certain variables that describe effectively a certain class? Are the values of a certain variable particularly outlying for the individuals from one of the classes? For example, we have that for January, The black cities take smaller values than all the others. So it's clear that the January variable can be used to precisely define the black class. Similarly, we have the impression that the green cities take larger values than the others. Obviously, it's too much work to go through each variable and each class one by one with a plot like this. What we really need is is an automated procedure. The basic idea we are going to use to decide whether a variable describes a class well is the following: if the values of a quantitative variable X for the individuals of class q seem to be randomly drawn from all possible values of X, then X does not characterize that class well, because those individuals don't take any particular precise set of values of X. Inversely, if the values of X for the individuals of class q are found in a particular region of the possible values of X, then it's doubtful that they were drawn randomly from across X, and we can state that the variable X, in some way, describes that class. Here, the values of 2 variables are shown. One variable with random values, the other showing the values for January variable. We can see that for the randomly drawn variable, we have black, green, and red points mixed up everywhere, whereas for the January variable, there seems to be much more structure in the location of points. For instance, January seems to characterize well the green class, by the fact that its points take all the most extreme values to the right. We can also consider that the more our random draw appears unlikely, the more X characterizes the class Q. So, how do we go from this intuitive idea to a real statistical test? The underlying strategy is to compare the actual data with N_Q randomly drawn values from N. The class q has Nq values, and we want to know if we can reject the possibility that these seem to be randomly drawn from the N values in the overall population. To study this, we need to know what values X bar q can take, the mean value of the— of that class. So, is there any way to know what the distribution of X bar q looks like? Well, the expected value of X bar Q under the hypothesis that these class values were randomly drawn is simply the population mean value, that is X bar. In order to calculate the variance of X bar Q, we have to draw without replacement NQ values from within a population of size N. Therefore, it's the standard error of the population. Here S squared, divided by the number of individuals drawn, N_Q. So, because we are dealing with a finite population, there is a correction factor of N_Q over N minus 1. As for the distribution of X bar Q, since it's a mean, we can suppose that it follows the normal distribution, thanks to the central limit theorem. We can therefore now calculate the standardized and centered x bar q minus x bar over the standard error of x bar q, that is the square root of the variance of x bar q. This quantity, under the hypothesis of random draws, follows a standardized Gaussian distribution. If the test statistic is between minus 1.96 and and 1.96, we can reject the hypothesis that the data could have come from a standardized Gaussian distribution. However, if the absolute value of the test statistic is above 1.96, then this test statistic is uncommon, so it is unlikely that the data came from a standardized Gaussian distribution. This would lead us to question the hypothesis that these N cubed points were randomly drawn from N points. That is, the values of X for class q show some structure, so, to some degree, X characterizes class q. Taken to extremes, the more the absolute value of the test statistic is large, the more the random draw hypothesis is doubtful, and the more X can be thought of as characterizing class q. So, what we can do is rank the variables in terms of increasing absolute value of the test statistic. Here are the results for the black class. The test statistic has the largest absolute value for the month of March. We need to read this table from the bottom up because the actual test statistic values are negative here, so the most extreme ones are at the bottom. The mean for the individuals in the black class for March is -1.14. Degrees, whereas the mean for all individuals, including this class, is 4.06. We see that the standard error in this class is 1.1, and the population standard error is 4.39. The last column shows the p-value associated with the test for whether the data follows a Gaussian distribution. We see that there are lots of very small p-values, so this class of cities is characterized by many of the month variables. All of the test statistics are negative, signaling that the values taken by individuals in this class are smaller than those of individuals in general. And obviously this is true, these are cities where it's cold most of the year. As for the second, red class, none of the variables helps us describe it well, in the sense that these cities don't really take extreme values at one end of the spectrum or the other. They are all in the middle, spread out perhaps, but not far out to the sides. And for the 3rd green class, we have positive test statistics. The variable that best describes this class is September, with the most extreme test statistic of 3.81. The mean temperature of the 3rd class is 21.2 degrees for September, whereas it's 14.7 for all individuals. This class of cities are where it's hot most of the year. Can we also describe the classes using qualitative variables? To do so, we have to again consider that a given partition is a qualitative variable and look for the links between pairs of qualitative variables, between the partitions of the class variables and other qualitative variables. For each qualitative variable, we can do a chi-square test between it and the class variable. Then, we can sort the qualitative variables by increasing test statistics. In our example, we only have one qualitative variable dealing with the location in Europe of each city. We see indeed that this variable is connected with the partition of the cities. Can we also characterize each class in terms of the categories of our qualitative geographic variable? Take, for example, the 3rd class, and let's see if the category South describes it well. The basic idea is to compare the proportion of South in the 3rd class with the proportion of South in the whole set of individuals, that is cities. We can fill in a simple table with just the category South, and all others joined together in another category called Other. In the columns, we have the 3rd class, and then all the other classes joined together. Let's now construct a test to compare the proportion of South in the 3rd class with its proportion in the whole population. The null hypothesis, H0, is that the 2 proportions are the same, and the alternative hypothesis is that South is overrepresented or underrepresented in the 3rd class. Under the null hypothesis, the random variable NMC, which represents the number of individuals with category M in class C, follows an hypergeometric distribution with parameters n, NM over N, and NC. We can then calculate the probability to have an even more extreme value than the observed one, that is 4. We obtain this table. In the rows, we have the categories. Here, we have just one row corresponding to South. The first column of the table gives the proportion of towns from the South that are in the third class. The calculation to be done is therefore NMC over NM giving 4 over 5 or 80%. The 2nd column gives the percentage of towns from the 3rd class that are in the South. Here, 4 towns out of 4 in the 3rd class are in the South, 100%. The 3rd column gives the proportion of towns that are in the South out of all towns, which is NM over N, giving 5 over 23, or 21.74%. Now, we can calculate the p-value, which will indicate to us whether this category, South, can be said to significantly characterize the 3rd class. We get a value of 0.000564, which is less than 5%. Therefore, we can reject the null hypothesis and consider that South indeed does characterize this 3rd class. So in summary, it is useful to comment on p-values and compare proportions in a given class with the general population to see if categories are over or underrepresented. The test statistic itself, which is just a way of converting the p-value into a quantile of a normal distribution, also provides this information. If its absolute value is greater than 1.96, We can say that the category of interest characterizes the class of interest. Furthermore, if the sign is negative, the category is underrepresented, and if it's positive, the category is overrepresented. Here, for example, the South category is overrepresented in the 3rd class. And again, like for quantitative variables, all of the categories of qualitative variables used to characterize the classes can be sorted with respect to increasing p-value. We can also try to characterize classes using dimensions. These dimensions are quantitative variables, so we can use exactly the same approach that we used a little bit earlier for quantitative variables. We can see that the first class is well characterized by the first dimension. As the test value is negative, this means that the individuals in this class have quite extreme and negative values in the first dimension. The mean in this class is -3.37 while the overall mean is 0. For the dimensions, the overall mean is always 0 because the coordinates on the principal components are centered. We see that the 2nd class is characterized by the 3rd dimension with quite negative values for the individuals in this 2nd class. While the 3rd class is characterized by the 1st dimension, with large and positive values for the individuals in this class. To conclude, let's quickly recall all that we have seen in this part of the course. First, clustering methods can be used on data which pairs individuals with sets of quantitative descriptive variables. However, if variables are qualitative, We can instead use MCA to transform them into dimensions, which are quantitative variables. Hierarchical clustering gives us hierarchical trees, which show distances between individuals and groups of individuals. Hierarchical clustering also gives us an idea of the number of classes there are in the data. Another strategy to build classes is to use partitioning methods, like k-means, to support the conclusions obtained from the hierarchical tree or to improve the class assignment. This can lead to more stable classes, but also to a loss of the hierarchical information from the tree. Remember also that we can use partitioning methods as a pretreatment, a simplification step, when we have high-dimensional data. And, as we have just seen, we can try to describe and characterize the classes we get using quantitative and qualitative variables. These can be variables that were already used to calculate the distance between individuals, or supplementary ones. In other words, active variables or illustrative ones. Remember that a course on clustering with the same methodology of our video can be found in this book. Also, don't hesitate to try and plot the same graphs and get the same outputs seen in the videos using R functions from the FactoMineR package. That's it for the clustering videos. We recommend going and looking at the FactoMineR video now to see how to put clustering into practice and then characterize the classes you've found. Also, don't forget to do the quiz and try the course exercises.