Clustering with R (FactoMineR & Factoshiny)

We will see how to make a clustering with FactoMineR and Factoshiny. In FactoMineR, the clustering is performed either on a raw data table, or on an object resulting from a factor analysis, whether it is a PCA, a correspondence analysis, a multiple correspondence analysis, a factorial analysis on mixed data, or a multiple factor analysis. When you have a dataset with quantitative variables, you can therefore either perform the clustering directly on the data, or you can perform a PCA beforehand and use the PCA coordinates of the individuals to perform the clustering, keeping only the first dimensions of the PCA. When the values are qualitative, mixed, or when it is a contingency table, MCA, or factorial analysis on mixed data, or a correspondence analysis will be be used. Then, the clustering will be carried from the Factoshiny menu of the principal component method, after exiting the principal component method. We will run the clustering on the decathlon dataset. A little reminder about this dataset: we have 41 athletes and 13 variables. The first 10 variables relate to the performances of the 41 athletes in the 10 decathlon events. Therefore, for example, 100 meters, long jump, short put, foot, etc. Then there are 2 quantitative variables used as additional variables: the rank of the athlete in the event and the number of points obtained, and finally a last qualitative variable which is the name of the competition: Decastar or Olympic Games. The clustering is built on the basis of the first 10 variables only, so the distances between individuals are calculated on the basis of their performance in the events. Without the variables number of points, rank, and competition. Let's open RStudio and load the Factoshiny package and then the Decathlon dataset, which is available in FactoMineR. By doing Factoshiny Decathlon, we run the Factoshiny function on the Decathlon dataset, which opens the Factoshiny interface. If you click on clustering, the clustering is started from the raw quantitative variables directly. If we previously do a PCA, this allows us to choose the variables that are used to calculate the distances between individuals. We can also standardize the variables, which modifies the calculation of the distances between individuals by giving the same importance to each variable in this calculation. And finally, we will choose the number of dimensions that we want to keep when doing the clustering. Here, we are going to do a PCA and standardize the variables to give the same importance to each variable in the calculation of distances. And we are going to eliminate the last dimensions, keeping only the first ones. We are therefore going to add the variables number of points and rank, which is the same as constructing the clustering from all the other quantitative variables. We standardize the variables, which is done by default, to give the same importance to each variable. Then, we choose the number of components to keep for the clustering. Indeed, the distances used to construct the clustering will be calculated only from the first dimensions. The last dimensions are left out because they are considered as noise. There is thus a choice on the number of dimensions that is important to realize. By clicking on Values, we will see the table with the percentages of inertia. Here, we will keep 8 dimensions because we can see that with 8 dimensions we keep 96% of the information. We could be satisfied with a little less dimensions and keep 75% of the information, for example, or we could choose to keep all the information and keep all the dimensions. After clicking on Quit the app, the PCA interface closes and a new interface opens. The clustering can now be set. First, we specify the number of clusters that will be used later to create a partition. A number of clusters is suggested. This number maximizes the difference in inertia loss between 2 successive partitions. In other words, the partition in Q clusters will be retained if the loss is important between a partition in Q minus 1 clusters and a partition in Q clusters. And that this loss is smaller between the partition in q clusters and the partition in q+1 clusters. We will keep the default number, which is 4 clusters. One can choose to do a preprocessing by k-means before doing the clustering. This is useful when there are many elements to classify. The idea is then to build a coarse partition with many clusters, for example 100, and then build a hierarchical tree from the 100 cluster centers, Weighted by the cluster size. The top of the hierarchical tree is stable compared to a clustering built from all individuals, and the clustering is much faster. However, be careful, if there is a huge number of individuals, several hundreds of thousands, we advise to work without the Shiny interface and to launch the following lines of code. One line to do the PCA, or any other principal component method, the next to do the clustering without describing the clusters using the kk argument to do a k-means preprocessing with kk= 100 clusters. Then we can build the graphs and characterize the clusters. Consolidation allows to improve the partition obtained by simply cutting the hierarchical tree. Without consolidation, the clusters are those obtained by simply cutting the tree. If the clusters are consolidated, the individuals are reassigned to the nearest cluster barycenter, with a k-means algorithm. It leads to more homogeneous clusters, but the correspondence between the clusters and the hierarchical tree is lost, which is a disadvantage. Here, we consolidate the clusters. Finally, we can choose the Euclidean or the Manhattan distance. If we work on the outputs of a principal component method, we always use the Euclidean distance. On the raw data, we generally work with the Euclidean distance, but in some cases the Manhattan distance is useful. Now that we have set up the clustering, we can comment on the results. Some graphical options allow you to modify the graphs. For example, in the 2D representation, we can project the tree on the graph. Or in the 3D graph, we can add the names of the individuals or represent the cluster barycenters. In this graph, we have a diagram with the inertia gains. We see here a bar that indicates that there is a strong loss of inertia if we merge 2 clusters in 1 cluster. So we do not have to group 2 clusters to make 1 cluster. Similarly, there is a loss of inertia if we go from 3 to 2 clusters and if we go from 4 to 3 clusters. On the other hand, if we go from 5 to 4 clusters, The loss of inertia is much smaller. So we can keep 4 clusters here. Another criterion to choose the number of clusters is the shape of the tree. Here, the shape of the tree suggests that the proposed cutoff level is suitable and we keep the partition into 4 clusters. The second graph corresponds to the individuals on the main plan of the PCA, thus on dimensions 1 and 2. The individuals are colored according to the cluster to which they belong. There is also a 3D graph with the individuals on the main plane of the PCA with, in perspective, the hierarchical tree. This hierarchical tree shows the proximities between the individuals. We see for example here, as on the previous graph, that the blue and black clusters are well separated, but that the green and red clusters are much less well separated. One may ask the question, why are these clusters tangled? We can build the graph on dimensions 3 and 4 and see that at this time, the red and green clusters are well separated. So in fact, the first plan separates the blue and black clusters well, while the 3-4 plan separates the red and green clusters well. So we need 4 dimensions here to separate the 4 clusters. We can now see the numerical results. The tabs Clusters, Characterization, and Link with the Partition allow you to view the tables while the tab Values details the results. If you click on Clusters, Characterization, you have in column each cluster and in line the quantitative variables and the categories of the qualitative variables. The present variables or categories characterize at least one cluster at the chosen threshold, default 0.05. The lines are sorted in alphabetical order, but you can click on a cluster to sort the variables according to their discriminating power for the cluster. For example, by clicking on cluster 1, we see that the variable number of points is very characteristic. The value in the table is a test value. A v-test lower than -2 means that the individuals of this cluster obtain a number of points significantly lower than the number of points obtained by an individual in general. Here, no category characterizes the clusters, but if we increase the probability by setting a value of 0.5, Then we see appearing the 2 categories Decastar and Olympic Games. The tab Linked with the partition gives the variables which allow to separate the clusters as well as possible, that is to say which allow to characterize the partition. A table is provided with the quantitative variables and another one for the qualitative variables. For example, here, we can see that the number of points variable is the best to separate the clusters. The tab Values details the results obtained in the 2 previous tabs. It also allows describing the clusters according to the individuals. We have several result objects: a description of the clusters by the, by the variables, a description by the dimensions, and a description by the individuals. The description by the variables first describes the clusters by the quantitative variables, We find the information in the tab Links with the partition. The variables are sorted from those that best describe the partition to those that describe the partition a little less well but still describe it in a significant way. Here, only the variables that have a significant link with the cluster variable are kept. The strength of the relationship is measured by the correlation ratio, between the quantitative variable and the cluster variable. We see whether this correlation ratio is significantly different from 0. For example, we see that number of points variable is the variable most related to the cluster variable. Then we find the description of each cluster with more detailed results. For example, for the first cluster, the most characteristic variables of the cluster are the 100 meters, the 400 meters, but also the number of points and the shot put. For the 100 meters, the individuals in cluster 1 take values significantly different from 0 and higher than the average. Significantly different from 0 because the V-test is higher than 2 in absolute value, and higher than the average because the V-test is positive. In detail, individuals in cluster 1 run the 100 meters in 11.27 seconds on average, while individuals in all clusters, including cluster 1, run the 100 meters in 10.99 seconds. Individuals in cluster 1 do not run very fast since they take longer to run the 100 meters. From the number of points, we can say that there are individuals who have a significantly lower number of points than the others, lower because the v-test is negative. On average, individuals in cluster 1 scored 7,596 points when the overall average for all athletes is 8,005 points. We then have a description of cluster 2, cluster 3, and cluster 4. Here, there are no qualitative variables that describe the clusters, otherwise we would have a description of the clusters by the qualitative variables as well. We can characterize the clusters by the dimensions, thus the dimensions of the PCA in our case. As the dimensions are quantitative variables, we use the same methodology as above, and we can see that dimensions 1 and 3 allow us to best characterize the clusters. In fact, the individuals in cluster 1 have significantly weaker coordinates than the others on dimension 1. For individuals in cluster 2, they have significantly weaker coordinates on dimension 3. Individuals in cluster 3 have significantly stronger coordinates on dimension 3 and slightly weaker coordinates on dimension 2. And individuals in cluster 4 have significantly stronger coordinates on dimension 1. When we see this, we can represent the hierarchical tree on plan 1-3, which is what we are doing here. Indeed, with the representation of the hierarchical tree on the 1-3 plane, we can clearly see the separation of the clusters. A small remark on the representation with the 1-2 plan. This representation gives the best view of the distances between individuals. But by adding the colors on the different clusters, we can see that there is a difference between the individuals of the green cluster and the red cluster, therefore a strong distance between the individuals of the green cluster and the red cluster. The clustering allows to see in addition what happens on dimensions 3 and 4, so we have an optimal vision of the distances by the PCA plan completed by the partition. We can now see the results on the individuals. First, we find the paragons of each cluster, that is to say, the individuals closest to the barycenter of the cluster. For cluster 1, the paragon is Uldal, which is at a distance of 1.43 from the barycenter of the individuals of cluster 1. It is the closest to the center of the cluster. Then, the second closest is Barras, then Carleven, etc. For cluster 2, the closest individual is Ernu. We have another measure from the individuals which is called specificity. To do this, we calculate the distance of the individuals to the barycenter of the other For the individuals of cluster 1, Kazarsa is the individual who is farthest from the other clusters, farthest from the barycenters of the other clusters. The distance to the nearest barycenter is 5.08. From this point of view, Kazarsa is really very specific of cluster 1. He could not be in the other clusters. The most specific individual in cluster 2 is Smith, since he is very far from the centroids of the other clusters. The interface offers an automatic report on the results of the clustering in English or French. This automatic report retrieves the graphs and the characterization of the clusters, and can be generated in R Markdown, HTML, or Word format. Finally, the button Get the HCPC code of the clustering gives the lines of code allowing to implement the method identically. By clicking on Get DH CPC Code, the lines of code appear to implement the principal component method, PCA, MCA, or other, then the clustering, and finally to draw the graphs. You have seen all the results on the clustering. You can therefore describe clusters from individuals, from variables— quantitative, qualitative, active, or additional— from the dimensions. It is now up to you to implement clustering.