Tabular Data Visualization

Hello everyone. Welcome. My name is Nirmal Malapaka. I'm a kernel developer here at Wolfram. Today I'll be talking to you about tabular data visualization in the Wolfram language. In version 14 of the Wolfram language, we are introducing new experimental functions to visualize tabular data. In this talk, I'll be giving you a high level overview of these functions. We will start by looking at the structure of tabular data. Then we will look at some current visualizations of these data. Then I'll introduce you to the new visualization functions and what they bring to the table. And finally, we'll look at some options for these new functions and I'll talk about future directions for these functions. So let's get started. This is the general structure of tabular data and tab data. Here at the bottom is an example of such a data set. It has a length of 150 and every sub list has four elements in it. What makes tabular data tabular is how the information we are interested in is arranged inside it. Typically in tabular arrangements, information is grouped into columns. So this is what the illustration shows. If we consider the columns, we have n columns starting with y1 up to yn. And in the example data set, y1 contains the values 5.1, 4.9, 4.7 and so on. And y2 contains 3.5, 3, 3.2 and so on, etc. I'll be using this data set that you see on screen throughout this talk to keep things simple and consistent. In the Wolfram language, we have a couple of functions that will help us display this data in a spreadsheet like view. And they are data set and table view. Both of these functions show us four columns. In this example data set. Table view also shows the indices of the columns and rows. So we can see it has four columns and 150 rows rows. It's slightly easier to scroll through table view, so let me do that here. So we see 150 rows in that data set. Similarly, we can have a grouped structure for tabular data that looks like this, where each element is a table. So here we see our example Data set of 150 rows grouped into three subsets of 50 rows each. And we will look at what this means from a visualization point of view later. So here if we switch to table view, you can see you have the three groups and each one has 50 rows. Now let's look at this data set using some existing plotting functions and see.

What we can learn.

I'll Be dividing this into two broad categories. Those plots that give us information about what the data values are, and those that give us information about how the data are distributed. So first let's plot the data set using three LISP list plot, list point plot 3D and listline plot 3D. And all three give us a good idea of where the values in the columns are in general and in comparison to each other. List plot shows us the row index along the x axis and the values along the y axis, but we lose the separation. Whereas the 3D list plots show the row indices along the x axis, the column indices along the y axis, and the values along the z axis. Just a note here that we have to do a transpose to plot the data set this way. Alternately, we can use something like bubble chart 3D to plot our data set directly. So here each bubble is a row, we have 150 bubbles, and this gives us a better sense of the relationship between data in the columns in 3D space. However, as you can imagine, we have run out of dimensions. So if we add, let's say, another column to our data set, we'll need to employ other techniques to encode that information. Note that here we are plotting our group data set, which shows up as three colors. So what other plotting techniques can we use to visualize tabular data? More generally, we can use layouts. We can lay out plots in a grid. So here we have the four columns laid out in a two by two grid. This way we can arrange the plots to make comparisons easier. We can also use multiaxis arrangement which superimposes the values but preserves the ranges and gives us dedicated axes for each column. As you can see, such plots quickly get hard to read and interpret. Now let's switch to some statistical plots and see what they can tell us about our data. Again, we are plotting our example data set transposed using box whisker chart, quantile plot and probability plot. Box whisker chart gives us a quick visual representation of the five number summary of data in the four columns. Quantileplot plots the quantiles of data in our columns against quantiles of a normal distribution, so we can compare where the values deviate. Probability plot plots CDF of values in columns against CDF of a normal distribution. So we can compare the distributions of data in columns, albeit in an indirect way. We will see how we can do a more direct comparison later. Finally, let's quickly run through a set of plots that let us visualize multiple dimensions together. Here we're doing a Radial axis plot of our transpose data. So we get four columns plotted radially. Then we have the parallaxis plot where we get a dedicated axis for each column, much like the multiaxis arrangement, but without the row index axis, so the points collapse onto the axes. We are also plotting our group data set here. So we get the three colors. And for completeness, I'm going to include feature space plots which plot the features extracted from the vectors, one directly from the images and the other from our data. So with that background, let's dive into these five new functions. They're called pairwise plots. We have five variations. Pairwise lisplot, pairwise density histogram, pairwise smooth density histogram, pairwise quantile plot, and pairwise probability plot. These plots combine some of the techniques we have seen so far to create n by n arrays of plots that give us insights into high dimensional data. So if we go back to our tabular data, here is an illustration of the transformation of data into pairwise list plot. We take the columns of the data and plot every column against every other column. This creates the array of list plots that you see on the right. And this plot has a signature look with the diagonal, which is made up of plots of every column against itself. The illustration also shows how a single point in one panel corresponds to its location in other panels. One of the advantages to plotting our data pairwise, as opposed to every column individually, as we saw earlier, is that we don't have to do the comparisons in our head here. Every panel shows a comparison visually and every combination is covered by the entire array of the panels. Also, when we scan across a row or across a column, we may pick up patterns of dependence visually, which any individual panel might not clearly show. In that sense, this really is a case of the sum of the parts adding up to more than the parts. And here is a brief visual summary of some of the features we may be able to identify in each panel. Broadly speaking, correlation density and clustering are shown. And these are specific to the 2D pairwise plots, I.e. pairwise lisplot, pairwise density histogram and pairwise smooth density histogram. So in the lisp plot case, if we see points laid out in this way, they generally tend to be positively correlated, whereas a negative correlation shows up with the direction of the points changing. Highly correlated are pretty tightly packed together and loosely correlated or spread out. Same with smooth density histogram, we are able to pick up the modes of the underlying distribution, bivariate distributions. And so we have can have uniformly distributed unimodal or bimodal, and clustering and outliers can also be seen clearly in some of these plots. We have a slightly different kettle of fish with pairwise quantile plot and pairwise probability plot. Here we are not plotting the data directly. In the case of pairwise quantile plot, we are plotting the quantiles of data in the columns against each other. For probability, pairwise probability plot, we are plotting CDFs against each other. That means there is an extra step involved in reaching conclusions about our data. We are visually identifying the patterns and the distribution of data and then interpreting what that pattern tells us about the data themselves. I'm not going to describe these patterns and connections here, but just point you to our visual guide that we have in the documentation for these plots. It shows what the pairwise panel looks like on the left and the corresponding distribution of data on the right. With that, let's look at how simple it is to produce these plots and what they look like with a real world data set. So here we are getting the famous Fisher's iris data from the Wolfram data repository. This is the same data set we've been using all along in this talk, only difference being that we have labels for the variables in this data set. It has measurements of different visual features of the flower. The goal in analyzing the iris data is to develop a rule based on widths and lengths of petals and sepals that would assist in determining the variety of an iris. We can directly plot this data set using pairwise list plot. Note that pairwise list plot gets the header information from the data set and labels the columns and rows in the plot. So that gives us information about the variables we are plotting in each column. And then we can start identifying some of the features we talked about. We can see how some of the pairs of variables may be correlated. We can clearly see clustering and also spot some outliers in some of the panels. To focus more on the density of points, we can use pairwise density histogram and depending on the data, one of these plots may work better than the others. Similarly, pairwise smooth density histogram gives us a better sense of the underlying bivariate distribution of data and its modes. On the other hand, if we want to compare the univariate distributions of data in the columns, we can turn to pairwise quantile plot and pairwise probability plot. Again, I'm not going to go into the details of interpreting these plots as they tend to be somewhat more complicated, but you can follow the guidelines I showed earlier to investigate the relations between these variables. Finally, I am going to highlight a few options for pairwise plots. Standard visualization options like plot legends and plot markers work. So here we are representing each species of iris with a different marker and constructing a legend. We are also specifying custom headers with the headers option. Often it is difficult to accommodate headers in the limited space available, so we introduce the header display function option to control the display of headers. So for example, here the first headers are somewhat long, so we're doing a couple of things here. Because the headers are centered by default, we are fading them out at both the ends. As you can see here. This prevents the headers from running into each other, but also is a visual cue to indicate that there is more header text. And if we hover over the header then we can see the full text in a tooltip. And this happens with both the row header and column header as well as the row header here. Another option worth mentioning is plot layout. Here we are using a layout that changes the orientation of the diagonal to run from the bottom left to top right. By default it runs the other way.

Also, since.

Pairwise plots are symmetric, we can use triangular layouts if we wanted to avoid repetition. So that is what we are doing here with these two triangular layouts. As you can see, what we can do with these plots is in some ways only limited by the richness of the underlying data models. And that is the future direction these plots will be taking and will be part of the larger data story in the upcoming releases of the Wolfram language. If you want to know more, I would like to point you to a couple of talks at the conference this year. One on formats by Peter and Sean and one on Table View by Jason. We also have a visualization Q and A if you want to interact with the visualization developers. And that brings me to the end of my talk. If you have any questions, I will take them now and I'll also be available at the visualization Q and A. Thank you.

So hey guys.

So yeah, if you have any questions, I'll take them now.

Deliberately left plenty of time here to discuss because these are experimental functions.

So we are interested in learning more.

From what you guys want to see.

In these plots or generally in the areas that are covered by this talk. So if you have any questions, comments I've opened in the comments section chat section for this talk, so just put.

Them in there and can have a conversation. Okay, so we have a couple of comments. Any suggestions on display of TensorFlow animations? Not sure.

I guess so.

Right. So here we are looking mainly at like multi dimensional, like pairwise and table view kinds of things. So yeah, not really in this area. Jane, comments? Excellent addition. I've used data for pairwise plots.

These plots are prettier. Yes.

So yeah, there are pairwise plots are produced by lots of software out there and there are generally some problems that are kind of difficult to solve around how to make them look pretty. So one of the things like I mentioned is with the headers and how they typically tend to run into each other. And so that's one of the problems that we've solved and the other is with how do you really optimize the.

Look of these plots for different sizes.

Of the matrices that you're plotting as.

Well as the size of the graphic itself? And so yeah, by no means it's.

A solved problem, but we're trying to make them look as good as possible in most circumstances. So thank you for that comment. And I have added the notebook that I used for the presentation, the handouts tab. So yeah, I welcome you to, you know, play with that notebook if you have access to the conference beta build. If not, these will be experimental functions in the released 14.0 and feedback is always welcome. So you can reach out to me or to Wolfram support and those questions will get to the visualization team. Right. So Thomas is elaborating on his previous question about tensorfields.

I guess so.

Right. So if you want to have. Right, so these are, we call them pairwise plots. And these can be like N wise.

Plots if you wish. So in that sense.

We can expand them as long as the data, if.

You have the data to plot and you're.

I guess, reducing the dimensions, in this case we are reducing to two dimensions.

Projecting down to two dimensions.

If you want to project down to.

Three dimensions, then yes, you would get a 3D cube of plots. Something that can conceptually be done, whether.

We can do it in a nice way that is intelligible. These are all questions that we need to still research into and part of the reason why we we have them as pairwise plots, I mean, sorry, experimental plots.

So.

We may expand them to nwise plots if there is a need for that.

But yes, please do reach out with.

The kinds of data and the applications that you have for these kinds of plots and then we'll be glad to take a look at that. Okay. Kurt has a common thinking towards publications. Are there grayscale styles that are easily displayed when printed?

Yes, so we have support for.

Plot themes. And so, yes, so that's going to be something that we can optimize for.

So the plot themes that I used.

Are the dark backdrop in this case, but we can set them to any of the ones. So if you want something that's publisher friendly, then I guess we can make plot themes for those. Okay, so I'm being told that this webinar is going to end automatically in a couple of minutes. So thank you everyone for attending. And I look forward to interacting with you in tomorrow's visualization office hours. And you can come there with your questions and we can delve more into this or other visualization topics. And yeah, otherwise you can hit me up one on one at the wtc.wolfram.com website or you can reach out to me by email. And thank you again. And I'm going to be ending this webinar now.

Bye.

Bye.