Clustering algorithms can give wrong results when your variables are on different scales. This tutorial shows how to normalize data in Python using SciPy's whiten function, which rescales each variable so its standard deviation equals one, then visualizes the before and after with Matplotlib.
Chapters
Hello everyone. Now that you're familiar with the two basic clustering techniques, let us discuss an important step in processing data that we should apply before performing clustering. Why do we need to prepare data for clustering? Imagine a situation where you have a set of variables within comparable units, such as the dimensions of a product and its price. Even if variables have the same unit, they may be significantly different in terms of their scales and variances. For instance, the amount that one may spend on an inexpensive item like cereals is low as compared to traveling expenses. If we use data in this raw form, the results of clustering may be biased. The clusters formed may be dependent on one variable significantly more than the other. How do we account for these issues? Then? We use a process called normalization. What is normalization of data? It is a process by which we rescale the values of a variable with respect to standard deviation of the data. The resultant standard deviation post normalization is one. The process of normalization is simple and achieved through dividing a value by its standard deviation. Let us look at its implementation in Python. A normalization library is available on the SCIPY package using the WYTHEN method of the VQ class. First we initiate the data to be normalized in a list and then use the WYTEN method to transform. The array may be one or multidimensional. Note that in the example we use a one dimensional list. In case of a multidimensional array or list of lists, WYTHEN divides each value by the standard deviation of the column. The result of the WYTEN method is an array of the same dimensions. Let us look at a plot using the matplotlib library in Python to compare the results of the normalization process. First you need to import the pyplot class of the matplotlib library. In Python we initialize the two lists which contain the original and scaled data points. By default, Pyplot plots line graphs Next, we enable the legend in the graph and then display the plot. Let us see how the plot looks like. The original and scaled lists have been plotted in blue and orange colors respectively. The serial number of the points is in the x axis, whereas the value of the original and scaled data points is in the Y axis. As you can see, the variation in the scaled data has been toned down from the original data. Interestingly, the trends remain similar to the original data though. With this we come to the end of this demonstration. It is time for you to get familiar with the normalization process by writing some.
