Python Tutorial: t-SNE visualization of high-dimensional data

In this video you'll learn to apply T Distributed Stochastic Neighbor Embedding or T sne. While this may sound scary, it's just a powerful technique to visualize high dimensional data. Using feature extraction, T SNE will maximize the distance in two dimensional space between observations that are most different in a high dimensional space. Because of this, observations that are similar will be close to one another and may become clustered. This is what happens when we apply T SNE to the IRIS dataset. We can see how the Setosa species forms a separate cluster while the other two are closer together and therefore more similar. However, the IRIS dataset only has four dimensions to start with, so let's try this on a more challenging dataset. Our answer Female Body Measurements dataset has 99 dimensions. Before we apply T SNE, we're going to remove all non numeric columns from the dataset and by passing a list with the unwanted column names to the Pandas dataframe drop method. T SNE doesn't work with non numeric data as such. We could use a little trick like one hot encoding to get around this, but we're using a different approach here. We'll create a T SNE model with learning rate 50 while fitting to the dataset, T SNE will try different configurations and evaluate these with an internal cost function. High learning rates will cause the algorithm to be more adventurous and the configuration it tries out, while low learning rates will cause it to be more conservative. Usually learning rates fall in the 10 to 1000 range. Next, we'll fit and transform the T SNE model to our numeric dataset. This will project our high dimensional dataset onto a numpy array with two dimensions. We'll assign these two dimensions back to our original dataset, naming them X and Y. We can now start plotting this data using Seaborn's scatterplot method on the X and Y columns we just added, the resulting plot shows one big cluster and in a sense this could have been expected. There are no distinct groups of female body shapes with little in between. Instead, there is a more continuous distribution of body shapes and thus one big cluster. However, using the categorical features we excluded from the analysis, we can check if there are interesting structural patterns within this cluster. The body mass index, or bmi is a method to categorize people into weight groups regardless of their height. I added the column BMI class to the dataset with the BMI category for every person. If we use this column name for the hue, which is the color of the Seaborn scatterplot we'll be able to see that the weight class indeed shows an interesting pattern from the 90 features in the dataset. T SNE picked up that weight explains a lot of variance in the dataset, and used that to spread out points along the x axis, with underweight people on the left and overweight people on the right. We've also added a column with the height categories to the dataset. If we use this height class to control the hue of the points, we'll be able to see that in the vertical direction, variance is explained by a person's height. Tall people are at the top of the plot and shorter people at the bottom. In conclusion, DSCNE helps to visually explore our dataset and identify the most important drivers of variance in body shapes. Now it's your turn to use TSNE on the combined male and female.