Live CEOing Ep 751: Language Design Review of Visualization Features for 14.0

Okay. Hello everyone. We are continuing the large collection of design Reviews for version 14 today. It's one about visualization.

So we have a family of pairwise plotting functions. They will put up an N by N grid of panels and they allow you to plot high dimensional data sets and get insights of various kinds. So the first three allow you, for instance, to detect correlations.

You know, so I mean, if I say things like pairwise list, plot of random real 10 comma, let's say something like 10, 10, 10.

No, it's more like 100 comma 10.

Yes.

Oh, so it doesn't deal with. Wait a minute, why is it hanging?

It's probably doing multi data sets, but it's not going to be.

Okay, well, so explain this to me. So I mean, the scales look pretty funky down here.

So I mean, the data you gave is the least interesting of all possible data.

I understand that. I understand it, but sure.

Right. Okay. So, so do 20 comma 10.

It's very, very slow. What's going on here?

Let's list plot and you know, needs, opt, you know, need some optimization.

Well, okay, sure does.

Yes. So you have 20 points that are 10 dimensional. Right. And, and so scales look like.

I mean, look at this stuff here. What the hell is going on there? Some of them are 0.0, some of them are.024. What the hell's going on here?

Well, it probably depends a little bit on what values happen to be in each position.

Well, this looks terrible. I don't think it's a good idea to have that the scales change. See what I'm saying?

Well, I mean, the alternative is, I mean, you end up with, you know, lots of stuff colliding.

Okay, let me understand what this picture is of.

So in panel ij, row I, column J. Yep. You project down to the coordinates I.

Comma j and It's a list 10 squared things here.

Yep.

Right.

Yes, but this is, this is the identity. Why isn't it on the leading diagonal? Why is it on the backwards diagonal? It is on the leading diagonal.

No.

What do you mean it's going backwards here? The leading diagonal will be down here. But that's a thing correlated with itself.

Yeah. You know, if you, if you plot X, if you plot X comma x for different values of X, it's going to be a diagonal.

Okay, fine. Yep. Okay. All right, so let me plot something more interesting. What? I mean, if I, if I say random real.

Go, go to the documentation for pairwise list plot. It has.

Oh my gosh. Sorry. So that really Needs optimization. And this really doesn't look good. I mean, I think you need a way to lock these scales to be the same. If we look.

But they aren't necessarily the same.

No, I understand, but they could be made the same. I mean, you could make them, you know, we can make them all plot range know. We could throw all the points in and say, what's the plot range automatic version of all those points? Because that way this thing would be meaningful. Otherwise this thing is meaningless.

No, that would not be meaningful. I mean, look at the examples in the documentation for pairwise list.

But I don't understand here's.

I mean, so Stephen, you are generating points with 10 dimensions, all coming from the same distribution.

I know.

You know, in reality this is going to be like your height and your weight and.

Well, sometimes there's no.

Sometimes it will be uniform. Column is going to be.

Sometimes it will be uniform, sometimes it will not be. Sometimes the dimensions will all be, you know, meaning the same kind of thing. And sometimes they won't be. That's what I'm suggesting. This is a case where they do happen to mean the same thing and where you should lock the, the scales together. Even though it might be that the default should not be to do that, but you should have a way to do it.

Right? I mean, the way to do that would be with plot range.

Okay, so if I say here, plot range arrow automatic, what will that do? Will that throw everything?

That is probably what it's doing right now. Okay, if you said plot range goes to minus 20, minus 20, you know, minus 20, 20, minus 20, 20 or whatever, minus 10. I guess we're getting down to 20 in some cases.

That will do that, won't it?

That should. I don't remember if that's 0 to 20 or, you know, minus 20 to 20.

Okay, that's more. Much more informative than the previous spot.

But that's a very special case.

Well, it's not that special a case. It's a case where the data happens to be. You have a high dimensional function, not something where every piece of data and every dimension comes from a different thing. There are two cases, right? This is more of the kind of pure case. The other case is more of the applied case. I don't think this case is so totally irrelevant. I certainly would have had applications for this over the years.

Okay, we can come up with a syntax for plot range to sync everything.

I think full does it now.

Does it again, plot range full.

I really doubt that. That seems very. Seems totally crazy.

I Mean, that is what I was going to suggest for that, you know.

Okay.

Really? That this is the right thing. Well, I'm surprised by that. It doesn't seem quite right to me. Okay.

I mean, I don't think full normally has, you know.

No.

A meaning for this case. I mean, for this type of, you know, list plot.

Okay, so can I take a data set and plot it this way?

I think so. It should. I. I don't. I'm not 100% confident that's been hooked up, but it's certainly, you know, on the list. If it isn't, That's going to be really boring.

Well, let's just see it. Is this new, this color, which the.

Is that a pinkish?

Yeah. Okay. That was boring. Okay, how do we see something which is more multidimensional with numbers rather than categorical data?

Go ahead and scroll down to the basic examples in the docs.

Okay. I was going to ask about Mr. Fisher's irises.

Yes. They are a classic example for this type of plot.

Okay. Now here, each panel is chosen with its scales different, right?

Correct.

Now, is there any way that I can see where which data point can. Corresponds to which data point in another panel?

Not yet.

Not at the moment.

That's something we would like to have as part of the interaction. I'm not sure we're going to make it for this release because it might require some underlying data structures to back it. But, yeah, you're right, Stephen, that. That is a key, key interaction point to know which ones are, you know, connected. And often you do. You select a group of points and the group of points light up in all the other panels.

A bit messy. And the rectangular pixels are kind of unpleasant.

Yeah. I think we should try to get them square if we can.

I think so.

This will look nicer.

Yeah, looks pretty good. I mean, hard to know what. Now, why isn't it labeled with, you know, petals and sepals, you know, and whatever they are?

Because that's not in that data set.

I see.

Normal, is there. What's the option setting for that? Or is there an example? Comment in chat.

Okay. All right. Now let's try a pairwise smooth histogram that. Let's watch something exciting happen here a little bit. It's exciting. That's nice. That's nice. Now, for this categorical data, can we get different colors or something for that?

I mean, at the moment it would be a manual thing and you wouldn't do it with the histograms. What was that? Oh, that was. You Reevaluating.

One day. That should work, right?

Yeah. So if you did like.

Okay, why is this gray and this not gray?

Oh, is this different?

What?

What?

Stephen?

The label colors here and they're black here.

I think it's very subtle over zoom.

Yeah, it's very subtle over zoom, but the intent is not to vary it out.

Okay.

Right. So if you did like.

I'm just a design question. Have we had design look at this?

Not yet. Other than Tim.

Because I'm. I'm wondering, is Tim here?

I don't remember if Tim is here.

So we have a long. Actually, it would be useful if Tim joined and we could showcase some of the designs that we're aiming for, which is baked into some of the themes.

Fine. Ask to be competitive because, I mean, here I. I'm wondering whether these should be lightened. These borders. I don't know. Doesn't look bad right now, but I just wonder.

We have lightened them a little bit, but I don't know if we.

How well is plot theme section filled out normal in the docs?

Some, but.

Try web. I think that's nice, by the way.

And it's not obvious to me that we need the gutters in general. Are the gutters really doing anything for us? I. I think they just look kind of, you know, confusing.

I mean, I think they help on the density cases for sure.

I see.

Although it looks like the density still has a little bit of plot range padding, so maybe they aren't necessary even there, but.

Well, in this case, if we know these are going to be dark, we could have these. These lines be. Be white.

Yeah.

I don't know if that will work, but at least 50% gray. Try.

Could you try pairwise this plot with web? Yes.

I want to know the following thing. How do I change the color of the. Of the, you know, crisscross lines? What's the option to do that? Should have to be an appearance option with a. With an association of things or something.

See?

So that would not be grid line. Possibly. We overlooked that one somehow. Okay, try frame style goes to white.

Yeah, it should be frame style with probably not hooked up.

It worked. It's very nice.

Okay.

That's really nice. Seems to me we need design involved in this. Okay. Because that looks so. So we have.

Tim has done a lot of design and we can bring in the other design, too.

Yeah. Okay. But this looks good to me and I would. I would just go, you know, tighten.

We, you know, pick frame colors that are contrasty by default.

Yeah. I suspect if we say gray level 0.5 doesn't look fairly reasonable. A little bit light, I mean, but. But if it's thicker, what would happen? That's absolutely fine. So there's a way to do this.

So some of the things that we have had to work on to fix is that often these labels, the variable labels basically belong.

I see.

You know, and I think we have a good repertoire of solutions for that.

Well, can you show me a gallery? Of what? Of things that happen. I mean, I also, I'm not convinced that this is the right default color scheme, by the way. Looks very, you know, Victorian ghost oriented.

Yeah.

You know, like those auras and things, you know.

You try for. Could you try the list plot instead of that one? I haven't even seen this one myself.

What is that?

That's Web. That's.

No, but, yeah.

Right. I think the background on the headers is too dark on this one.

But yeah, it is.

I don't know. This doesn't look too terrible to me.

But there's some way. There's some things that are a little bit heavyweight there and stuff.

Well, but that's. But I set the frame style, so.

Oh, you did? Okay. I didn't see that.

That'll help.

Yeah, that looks pretty good. Okay. All right. What else should we look at here? I mean, I don't know. My input would be, you know, some graphic design work, removing those gutters, things like this probably.

Is Tim coming or not coming? Can we show the design? No, it's fine. Did someone ask him?

Jeremy, please.

I'm trying to.

Yeah, I'm trying to reach him. I. I mean, I'm pinging him. Well, also ping Jeremy. Okay. All right. What else would you like to highlight with me? And then we're also going to look at these univariate functions.

Yes.

You haven't given an example of that. Where's the example?

It just opened a page. Because we hadn't done the illustrations at the time I wrote this, we're still to some extent working on them. So these two are a little bit different in that what they do, the way you go from input data and dimensional is that you look at. You compare every pair of columns basically to see how their distribution of values relate to each other. It's basically what they do.

About our visualization directions. Okay. So there are still some plots that we don't have. Like this. Gosh, what is it called? The, the. The Sany diagram. Things like that.

Sure.

That are for much simpler kinds of data than these. We really need those?

Sure.

I mean, these things that deal with, you know, I have not used radial. Whatever it's called. Plot. What is it called?

Radial Radio access.

Plot.

Comparable access. I haven't used that yet. And which is a bit surprising. Andrew on a live stream is commenting. Will there be improvements to graphics aesthetics when using the multi axis functionality? For example, things like grid lines, frame, label do not work easily when using the says.

I'm not sure if any of those fixes are going for 14, but.

Things here. Okay, all right. But I don't understand this time plot. So maybe I should just see. What the heck is this? So this is a quantile plot a variable A against variable B.

Right.

By the way, see why you don't use. I think you should use in the basic example something which does have labels because I think otherwise people just don't know that there are possibility of labels there.

That's a good point.

I mean, since you're hiding the input data anyway, right?

We can hide it. We can hide the labeled version.

I really hate the way that those scales just look so messy. I guess there's no choice, but it just. They seem awfully messy.

I also think, I mean, those are one of the areas, you know, that pose difficulties, you know. You know, not just for us, but, you know, for everybody who creates these types of. Yeah, plots of, you know, it's. It's interesting to see, you know, sort of how different, you know, people do it and where it breaks down and.

Yeah. Okay, well, this question whether we have gutters between the things or whether they are coincident is something we should figure out. How would I set an option to make those things coincident? Spacing zero.

Yep, spacing zero.

I don't know. I mean, the points there are so you know that you can barely see the points. You're mostly seeing the, you know, the, the crisscross in the window. You're not seeing anything in the view. I don't know why. Maybe the points are too small.

I think the points are too small.

There's also another style of points.

Yep.

It's coming from the edge form of the point. So if you set a plot theme, you'll probably be able to.

No, I don't want to set a plot theme. I want to set plot style.

Sure.

And how do I set that?

Yeah, you can just set it to any color and.

No, I'm not talking about the color. I'm talking about. I'm talking about making these points bigger.

Absolute point size of three should be plenty big. It's not as big as I was expecting.

Well, now when I expand this, it gets bigger, which is very odd because it's an absolute point size.

Yes.

Let's try an absolute point size of 6 to see if it's even doing anything.

There we go.

Yeah.

See, it's not doing anything.

Okay, so. Yeah, just was doing something last week, I thought. Or maybe it was just colors somehow.

Okay, so this is a quantile plot. You've got one thing against the other and you're sorting. I don't understand. Normally in a quantile plot.

So this is. The bottom left corner is going to be the minimum and the minimum.

Yeah. Okay. And then to maximum. Yep, a maximum. Okay. All right, I understand what this.

That's a unit framework regression.

Does iris data have units on it?

Well, maybe it does.

Probably not.

Well, then why does it say this? Let's try.

I mean, so the fact that the probability and quantile plots generally share a lot of code.

Okay. So something went wrong.

Yes, something went weird.

Okay. All right. What else should we look at? I mean, you were going to show me a bunch of aesthetic choices.

Yeah. So if. If maybe Tim can share or he can send you. The notebook would be great.

Why don't you send it to me, please, Tim?

Here, just. I don't have. Here, let me put one together real quick of what I did before.

Where's the one that you have for general multi panel stuff?

Okay.

Yeah. With all the components. Well, who's doing the.

So one scheme that we learned, by the way, Stephen, which may be applicable in a bunch of other cases of labeling, is, is to have the panel labels fade out when they get too long.

That's interesting idea.

So then we learn if you resize one of those plots and make it, you know, like half width.

I think they're going to resize with the.

Oh, that's boring.

Well, that is a bug that you've got to fix. Okay. That's the same bug that I was reporting to you on graphics grid about the fonts being broken off. Right. You understand that bug?

Yep, I understand that bug.

Okay, so at that size, you should do something about these. Right. You should probably either remove those or something. I mean, the only reason that the fonts look okay is because they went out of range. So there's more to do here. You should make the scales change in size as well, I would assume. Are these changing in size?

I don't think there's changing in size. I think there's a measurement that's in.

The wrong coordinate system in there.

Normal. Is header display function fade working?

Yep, that should work.

So what should I do here?

You can do column labels.

Pick one of the ones that's doing the data set. Yep. And then header display function goes to the string fade. Ah.

And they picked.

Is that pinking?

Well.

Oh, okay.

So not working. But. But I like the idea of faded light.

So it turned out that that is in fact the same scheme. We didn't know about that, but then we learned it's the same scheme that's web browsers use and etc.

For long URLs in the URL field.

As opposed to dot, dot, dot. Yeah, it's an interesting idea.

Yeah.

Okay. Okay.

All right.

What else should we see here?

I think that's about it. There's maybe a question that you have an opinion about, which is that we co opted a bunch of options that are basically for.

For. For individual plots. And you're using them for the whole gaggle of plots, right?

Well, that too.

But we also are using like a lot of data set options and you know, like the spacings and like header display function is a data set option originally.

I don't think that's terrible.

Okay.

I think we should leave them experimental until we have a little bit more of these in, you know.

And what does this mean once you.

Widen the notebook a little bit?

Not sure if.

There we go.

Reading across.

Yeah.

How is that specified plot layout.

Right. And some of the stuff in Tim's notebook is for functionality that's not in this. Not for this version that is related.

So the aspect ratio could be the individual panel aspect ratio. It could be the aspect ratio of the whole thing. Right.

And we're doing the individual panel.

Okay.

But for some other things you're not correct image size, you're presumably doing the whole thing.

Yep.

Oh boy, oh boy, oh boy. That's kind of annoying.

It is. But I think for most of the options at least there's a pretty obvious which way you want it to behave.

I think the aspect ratio is not so obvious if you have a collection of these.

Well, I mean. Well, okay. So you know, aspect ratio for a normal plot is not, you know, the ratio for the image size.

Yeah, I understand that.

So.

What is this?

This is for missing data.

No, I mean that's not for the pairwise stuff.

No, it's not for the pairwise. All these were done before we even started on pairwise. It was just something to get us started. But one of the things, for instance, so we rotate labels and so now you don't really want to cock your head at two different directions. Right. Cock it left and then cock it right. So we consistently rotate. There's like. If you're going to cock your head, at least it's only one direction.

You'Re cocking for this one because you're reading it like you would ordinarily read it.

We'll put the label on the right edge.

What about the x axis? Eu, us, Canada and the right edge?

Well, that's a single recocking, so to speak.

Exactly. We explicitly made it only one.

We could have could have flipped it. So if there was like an ostrich reading these, it would just on emu or something. It will just never mind. Exactly.

Reminds me of a comic I saw. It's a doctor Sorry, it's a dog teaching other dogs and all the dogs are cocking their heads left and right. Like from all the cocking of heads. I think I gotta re. Explain this topic.

Huh. Okay, so what's this? How did you get the. Is that gettable in. In the current setup or is that something.

This whole every. Everything in here is faked.

Oh.

So just to get an idea across.

Okay, how are we doing this? Label, inside label. You know, how is that working.

So at the moment? I mean, so this is, you know, for a thing for where individual panels make sense to be labeled, which is not the pairwise case. Right. In pairwise, you want to label the rows, you want to label the columns.

Right. Okay, but I could in principle do this with place somehow.

In pairwise. No, I mean, I mean, what does.

Do if I try to use it in pairwise? So I'm just. It should give a message.

Say that again.

If you try to use placed for a plot label inside pairwise, it shouldn't just silently fail. It should give a message.

Okay.

And that's a general principle for these kinds of, you know, if you don't recognize some primitive give a message, don't just silently fail. That should be true across lots of graphics. Am I making sense?

I think graphics does that. It even thinks it.

No, it doesn't do it in many cases. I. It just. It just ignores it in many cases. I've tried to get different effects. You know, plot legends in the right place, blah, blah, blah. It just does nothing. Right? It just does nothing. And that's wrong.

Okay, so you want more messages out of the vis and plotting functions.

I do. In other words, if you can't figure out what to do, give a message. Don't just silently fail.

I mean, there are cases Where? Yeah, sure. I mean, I think for the most part we can do that. You know, if you're. I mean, if you specify something that, you know, involves dynamic, you know, we.

Can'T inspect that, fine, but that's, That's a different thing, Right. A lot of these are individual labeling, which we don't have right now for the power functions, right?

Yes, correct. But the general, you know, appearance of, you know, what it looks like at the top and what it looks like on the right is applicable.

Yeah. Any ideas for what to do with these? I have to tell you, if you ran some kind of color or something behind those things, I think it would look better than just these random little bits, you know, left over, like little pieces of. Of sort of confetti or something.

Are you talking about the ticks?

I'm talking about the ticks, yeah. I'm talking about these ticks. If, if you were to, for example, run a background color behind all of these or something or some way.

Or the tick labels.

Yeah, yeah, yeah, yeah.

I mean, do you want it to match the stuff that. The labels that are on the top and right, or do you think it should be.

I don't know. I. That's Jeremy or somebody. I mean, I, I think, I think, I think that will help make it more integrated.

I think what happened with too much emphasis on the tick labels.

No, we.

Because you want them on the heads, can we, you know, right now they're just kind of secondary.

Can we find Jeremy? Somebody looked for him when I asked before.

Yeah.

So he's findable or he's not findable?

I'm, I'm reaching, I'm trying to look.

Him, but we also talked about that 10 minutes ago. But. Okay.

All right.

We're not going to find him. Okay. So I'm telling you, you should talk to him about. I'm telling you something that involves a light color behind these or something like that will help make them seem more integrated. Am I making sense? Unless, like random things at different places.

Right.

And the collisions are pretty ugly. I mean, it's very unclear what those tick labels do for us. Am I making sense?

Well, I think that we have another shot when we go interactive. Let's say that you get a drop line both to the, to the X and Y.

Then you can, Then you can give what the value is on the.

Yeah, you can light it up with the value then. So that might be, in the end, the more useful. Not sure yet. We need.

I don't think this is useful here. I mean, when you Have a large number.

Well, I think that there's sort of, you know, I think that there's. Same thing that we talked highlighting. There's sort of the static versions for a static publication like PDF, and there's a dynamic version and sometimes when you publish it, you have to have something there.

I understand that, but I think it's going to be hard for people. I mean, that is just ugly, what you have there right now.

Yeah, it is.

Right. So I claim if they were all equal bins, which can happen from time to time, then you could label the first one and you'd be happy.

Another thing that's a bit odd, which is. Which comes out here, is that sometimes it decided to do integers and sometimes decimals.

Yes, well, that's because sometimes it has only integers. It has only introduced. It's using just integers, which is the right behavior.

No, I mean the range is a little bit different and so it's switching between.

I see, so you're saying. You're saying. Wait a minute. So if it's only going to show integers, there's no reason to have the decimals.

Right.

If it's going to show a decimal, then.

I see what you mean. Roger, that is an interesting observation. I wonder why. Okay, so drag that plot bigger. So we see more of the ticks.

So that's the 2, 4, 6, 8 there in the bottom.

Right. And the others are 2.5s.

Yeah, that's.

That's something that we're working on on the axis object side of things. So no issue.

Okay, but. But even at this scale, these ticks are really ugly.

Mm.

Yeah, we kind of are.

So I think Header display function automatic is hooked up to do the fading, if you want to try that.

But what's it called now?

Header Display Function Automatic, which you want to do with one with the labels.

Well, this is header. I mean, the headers or this or the axes.

No, this is a header.

Header display function is not doing anything with the ticks. Is it normal?

No, it's not.

Okay.

I mean, I know men was experimenting with.

Okay, I don't know what that is.

That's.

That's weird.

That's wrong.

That's an interesting case.

Yeah, well, it's wrong. Okay. Why is this taking forever to resize? Why is it taking forever? Forever. Forever to resize? My hand's been off the mouse for ages.

Where ages mean seconds.

Yeah, but I mean, it's been off the mouse.

Yeah, I know what you mean.

Why is it taking so long to resize?

I'm not sure. I don't think we're doing anything, so. Okay, we'll look into it.

Something came up here. Well, you've got something going wrong. Well, take a look at it.

Graphics is not super happy.

All right. Is there anything else we should cover here?

No, that's it. I think that.

I think that's it.

That's nice, but I want to see some of these other diagrams that are on simpler to specify data.

Sure.

I'm not sure Sankey is simpler to.

Specify, but I don't understand why this background is white and this background is blue.

It's just the sparsity. When there's zero in any bin, it's left white.

Okay, what is this here? Isn't that zero?

Well, but those are, you know, that's continuous, continuous function. So there's no, you know, it's not zero anywhere.

Okay.

Whereas with the discrete one, you know, parts of it really are zero.

I see.

And the zero parts aren't.

But it's very ugly that blue is the epsilon and white is the zero.

Yeah, I mean, we could. Well, okay, so if we look at like density histogram in general, should that fill in everything?

Why is this. I'm totally confuzzled about what's happening here. This is. So that's density histogram, but what about smooth density histogram? One went, you know, positioned itself for the seance.

Well, how important is it to know that something is empty?

I think it's important. Yeah, but the fact is that. That the fact that the color map here is so different in those two cases just doesn't seem right.

Well, I mean, the color map is exactly the same, of course, but.

Well, but okay, but the color map.

But the effect is different.

0 is white and epsilon is dark. Blue just seems wrong. Just seems like epsilon should be close to white.

So often a high value.

I mean, the other thing I was thinking is, you know, we could make, you know, like zero.

A shade of.

Gray or something, possibly. I just think you need something consistent here. Minor comment. I. I just don't. I think these. I know people often make plots like this, but I think we should rethink those colors a bit in view of the other one.

Right.

All right.

I mean, because, I mean, to some extent, arguably, you know, the, you know, empty squares should be blue. You know, I mean, it's. I mean, they're white because they aren't having the color function applied to them.

Yeah, well, that's just wrong if those are zeros in the top one. Yeah. It just seems wrong, but I don't think that the effect. I mean, again, I think you need to look at some graphic design stuff and see what. You know, it just. It doesn't seem right. That, in a sense, the. The most dramatic color is the color that corresponds to zero. Well, anyway, I should go.

Okay.

Okay.

Good stuff. Nice.

Okay, thank you.

Thank you.

See ya.