Live CEOing Ep 831: Language Design in Wolfram Language [Tabular]

Okay. Hello everyone. Welcome to another Wolfram language design review again with our friend tabula. Very needy design target. Today we are going to be looking at tabular visualization, if I'm not mistaken.

Maybe we should start the transformation just to be wrapped at and then go on from there.

Okay, So I mean we talked about transform columns, construct columns at great length. I don't know what happened with row indices and column indices. Did we not talk about the supplied type syntax or not supplied a nearest like syntax for getting those things? Or am I confused?

We have a. Yes, the nearest type syntax for select and discard.

I'm sorry, remind me what row indices does again?

I mean row indices is more general than select. So it can. It translates. Actually maybe the easiest is to open up the source so it translates, you know, index based things. We have span and other things. But you can also see the, the beginning here is.

I think the more relevant one is column indices.

We can't hear you, Rob. Rob, we can't hear you.

Sorry, I had my microphone.

So.

So the, the more relevant one is column indices which in addition, you know, it translates to part type indices but it, it translates keys to indices, which is sometimes quite useful.

Well, okay, but which way around is this? Is this. I would expect given the narrow indices. What's that Select.

Select can replace one case of row indices. The new design for select can can replace that.

But I don't understand. This is indices. The word indices is the last word in the function name. It means indices come out. Yeah, so they do. This is saying given a column, do I have a sample tabular somewhere? What's a sample? Where do I have a sample of tabular? Where's an example tabular.

Tabular for instance, the reference page tabular.

Do we not have samples and example data and things yet?

No, do we have that? We don't. We have them in example tabular or tabular example.

Yes, but not in example data.

Okay, but we're going to put them in example data. Presumably we need to. Yeah, yeah, just like we did for data set. Just like we have. Look, we have.

But Rob, don't we have the this tabular example or example tabular.

I thought we had something you can do needs of example tabular data and then example tabular data of say Titanic would work.

Okay, but if I did this, I don't know why this is so slow. But.

But that doesn't look like a well formed this. It looks like a.

It doesn't look. It's two levels down, right? Yeah, it looks. Yeah.

It looks tree, like nested like.

Right.

Titanic would work if, if you want to construct a tablet from that, I recall.

Yes.

Okay. Okay. So we're not quite there yet on the visual design. Right. We're close, but we're not quite there. Correct, correct. So for example, these, these. The cutoff of the framing isn't really right. Et cetera.

Yeah, I don't know. That's. That's. That's another thing. But like there's a. There's an obvious bug today which is like the column types are overprinting the name of the.

Oh, that's what that thing is. Okay. And the missing is going to be a dash. Right. And it's usual formatting.

Yes.

Okay, so here. I say column indices of line seven. Right. Comma, sex, for example.

Yeah.

Why is it called column indices when it will return?

So if I say typically there's more than one. Right.

Okay, just. Should that work? Should. Shouldn't it be a pattern specification?

No.

Well, no.

I mean. Well, I don't think that's a design. Right, right.

You can.

I think they're right. There is probably a list. Right. A list of columns which maybe could also support alternatives. Although it's. It's probably not doing real pattern matching. Right? It's probably.

Why shouldn't it do pattern matching? I mean, the number of columns that we're going to have is not going so large. It seems like pattern matching will be useful. What about string matching?

Yeah, I think. I think some. Some amount of string matching actually works. I didn't do pattern matching explicitly.

Okay.

Right.

What the heck is index? You do a regular. Wait, what's.

I think.

I think index Slack is something I'd love to get rid of. It was an idea I had like over a year ago and it was. It. It's basically a way of specifying parts with a select, which I just don't think is very useful. And I think we should get rid of it.

Let's futurize it. I mean, let's not. Okay, so.

So what's the span spec?

It translates it to indices.

So like you could do class semicolon.

No, no. Span. Span only works with numbers.

Okay, so these messages, how is that.

Different than from just the numbers? Won't it.

It.

Won't that be the identity operator or you're saying it translates it in. Yeah, but.

That is the identity operator for that. Okay, but. But what it does is it. It cannot. If you, if you put the type of output.

It.

And in particular when somebody specifies a whole bunch of Keys. Something something you may want to know. Are those really columns in the, in the tabular or not?

Well, so if I say hitting. Right. Well, that's not very useful. I think there's a lot of work needs to be done on the error messages here. I wasn't getting a single message and there were lots of things that were clearly.

Yes, we, we know that a lot of work needs to be done on the, the messages. We're, we're working through that but you know, it's going to take time.

Okay, but in case where I asked for one thing that doesn't isn't a column. Shouldn't that.

Right.

That return a list with one with two numbers and one missing?

Yeah.

Okay fine.

So is there an additional argument then you're saying or something that makes the span case not stupid?

Yes, that form argument.

What do you mean by not stupid?

Well, it was the identity. Right. When we, when you did the 2 semicolon 44 or whatever it was. Right.

None of these formats here have anything other than basically packaging of those numbers.

Yeah. So if you give list, if you add a span and give list, it'll.

Get that woobly doobly. Okay. Not the most exciting operation in the world.

No.

Right. All right, what are we looking at next? So, so we're looking. There's some tightening up needed in column indices. Row indices.

Does the same similar things for rows. If you open that page.

We don't have keys for rows.

So I don't understand the point.

Whole part is irrelevant.

I really don't know why we're doing row indices, why we need this.

Are there application examples on the page or something to motivate this. Because I agree that as it stands it doesn't seem.

No, when we designed it, we used to have, we used to have a. We used to have row labels and row keys, but we, we only have the key columns anymore. So, so I think, I think this one could.

Let's just drop it for now. Let's leave it as futurized on the page. Let's just plan to drop it.

And the column indices. You know, it's, it's. I. It's used a lot in our, you know, packlet code for doing tabular operations. But, but maybe, maybe users don't need that. It allows canonicalize things.

Which one?

Column indices.

Oh, I think we should support column indices. I don't see any reason not to.

Fine.

Fine.

I don't think it's super confusing. I don't think it's super important. But I don't Think it's super confusing either. Okay. We were looking at tabular. Okay. So this, this categorization stuff, we didn't get to that yet. Right, right, right. And the importance of that is. It's this. It's this. I mean, I don't understand why we're not doing that by adding a column that is the category name.

Yeah, exactly.

Is that what we should be doing?

Yeah. So either you replace or you add. And most often I think you want to add as an additional thing.

So I don't understand this name because, I mean, this is basically a call up. It's basically an. All right. Construct. Columns. No, no, no, it's a transform. Columns.

Yes. You could say that. If you create them all sort of by hand instead of getting a menu of ways of doing it, then. Yes.

Is this operation what I, as a physicist would call binning?

Yeah, for numbers, it's definitely binning. And it's sort of binning. You know, it's just a. Feels weird to have string bins, but I guess.

Right, but then the name of the bin, I mean, what this thing is doing is it's creating another column. Yeah. Whose entries are the names of bins. Yeah, Right.

Yep.

Right.

And.

Go ahead, Roger.

No, I mean, so part of the discussion I wanted to have, wanted to show why, you know, why it's really important to have these categories, because that's the basis for a lot of that. One is the least, you know, fleshed out of anything.

Okay. But I. I get it. I mean, you want to be able to do grouping by these things, et cetera, et cetera, et cetera.

Right. Many things you want to visualize by them because they represent subsets, partitioning of your data set.

Yeah.

You want to do, you know, regression modeling based on those quotes, factors or those categories. You want to do pivot tables. You want to do, well, a whole slew of things.

Okay, I understand, but so now I understand how this is characterizing up because, I mean, we have the notion of bin counts and so on. Bin. What is our main function? I mean, the. Say bin counts.

In lists is the function you're thinking about.

Oh, yeah, yeah.

And in this number categorize. It's actually, you know, so I've set it up to work on a tabular column, which is what you suggested months and months ago. And then it's from the. It fits and transform columns and it creates this new. Categorized, this column of categories. And it's. It's actually because I use the compiler to implement it, it's quite A bit faster than bin lists.

Okay, that's. Don't, don't tell me you can have a new function because you could. Didn't.

No, no, no, no, no. But it, it does more than bin list because it's.

What does bin.

Listen, label to each of the lists.

So that's taking it in bins of size two, right? Yes, yes. Yeah. Look, this one goes up to 1.79, etc. Etc. These are bins of size 2.

Yeah, yeah.

Okay, so let me understand what, what your number categorized thing is supposed to be doing. What is it doing?

So it, it can do binning. You know, it's, it's the usual things that you described there, which is sort of arithmetically spaced is what you kind of just described. Would you.

So this design is the same as the histogram design.

Yes. There's also an important style down there which is based on frequency.

Wait a minute. Does histogram support this case?

Yes.

Yeah.

Okay.

Yeah. Histogram supports a lot more than you think it does.

Oh, yeah.

Okay. And down there is the quantile case where it's.

Yeah. Equal probability spacing. Or you could give it yourself, you know, like the last case is. So if you wanted quartiles, for instance, you could just give that.

So this is basically bin list. Is that true? Other than the fact that it has column handling.

It'S very much, it's very much like bin lists.

Okay, so why aren't we extending bin lists to deal with these extra cases? I mean, certainly these things here we could absolutely put into bin lists.

Oh, yeah, that direction is easy. But how do we. I mean.

How do you, how do you. So, so what bin list is missing is a, is a set of, of the, the categories for each, for each case. So you, you deduced that it was, you know, between 0 and 2 and 2 and 4.

But.

So here, those categories explicit when we're categorizing something.

Okay, so let's give this as an example. Okay, so that's a case where we have category names, effectively, and category counts. But so what you're saying is what you want, you want to take a column and it's kind of like an image processing. You're giving a label.

Yeah.

Making sense.

Yeah, effectively.

So what are our names for things which label things?

I mean, a. You might want to do it yourself, like.

Yeah, I understand that, but, but in a. Okay, let's imagine I have a picture. What the heck am I doing here? Oh, this is the prototype build. Hey, somebody should report this bug. It shouldn't be doing this. It should not need to do.

And there are issues with downloads in today's prototype build, so somebody put in a branch prematurely. And so this may be. Well, it does seem to be progressing. So.

What the heck is it doing? It's downloading a million species or something to get one octopus. It should be.

Who doesn't need a million species?

Well, I don't know. Our planet has like 10 million, one would guess right now. Unknown how many it had before. Unknown why? That number is the number it is.

Well, Maybe there are 13 fields in the metadata.

Okay, there's an octopus. I don't know why I was making an octopus, but. But let me do. What's an example of something that does. Labeling morphological components of the octopus, I think that does.

Labeling.

Oh, right. Okay.

Yes, the.

Okay, so this is the kind of thing we're looking for, right? That every. In the. In the extra column, everything is given a label, correct?

Yeah, yeah.

And sometimes those labels will be sequential integers. Sometimes they might be a symbolic representation of the bin. Am I making sense? Is the default. Should the default be a symbolic representation of the bin?

I think I set it up to be a symbolic representation of the bin. So you can actually look at them and see what they.

Yeah, that's what I would say think too, which is interesting, because things like morphological components don't have that possibility, even though they perfectly well could. I mean, morphological components perfectly well could have a little copy of the image in every one of these entries. A little copy of that part of the image. I'm making sense. Yeah, it doesn't right now, but. So this thing, then what is its return? Oh, it doesn't say here. Can I do this? Can I actually run it?

No.

Okay. All right, so let's just understand that the. I have to say that I'm not a fan of the way this is set up. I mean, I would have thought it was a transform.

Yep.

Columns of tabular, comma, you know. You know, animal, no weight, arrow. I see. And now it's a global operation on that column. Is that correct?

Global thing on the column.

So that's.

That's.

That's.

And we're coming closer, so.

Right. So that's where we had that whole com. Conversation about the extra arguments. Remember? The two extra arguments.

Yes. To the. To this function.

Yes.

But that's pretty wild. I mean, if that. Basically, what this is ends up being, you know, Number categorize of/2. Well, no, what it really is is Number categories of hash1, comma, hash2 where hash1 is this element and hash2 is the, is all the elements. Is that correct? I mean, that's terribly inefficient. It's going to redo the categorization every time. But, but that's the concept, isn't it?

That's one concept. I mean, I'm, I've, I've been working through, I've been talking to Tom and I'll be talking to Jose and I've talked to Carlo about different ways of handling that very issue. And, and I'd like to put together something clean to, to bring that up. But I think that's a fundamental issue in tabular transform columns is what. How do we differentiate between things that work on a whole column and things that work element wise. But I don't think this is the time to sort of dive into that.

Okay, but, but I mean roughly one way to think about this function or ones like it is you're given the whole data set and you're giving a particular entry and. Yeah, I'm a little confused.

Right? I mean, so I mean, if we did it this way, right, Then presumably we could just stick bin lists in there with, you know, an extended third argument. I think the point with number categorize is, right? You just give it the tabular and the spec and it years out and maybe there, maybe there's no value added in there. But on the other hand, maybe just expressing it succinctly as the value added.

Yes, I, I understand that, but I mean, another possibility is that like nearest or something that there's a 1 argument number categorize that basically makes a number categorization function to which you feed any number. I mean, just like I can say, you know, nearest, Right? And now I feed a number and it gives me the nearest number. But I could equally well do nearest arrow index here, could I not? And then if I take this and I say something like that, I'm very confused. Why didn't that give me the nearest by index? You see what I'm asking?

Well, it's hard to, to check because you're you.

If you had range, that would make sense. But random integers, you know, who knows? There are probably three random integers at the same distance.

I get it, guys. Yeah, but that doesn't help me either, does it?

No, it's not very interesting.

I mean, you could probably have given it a decimal or something.

Okay, fine. So that's giving me the index.

Yeah.

So what I'm suggesting is that the analogous thing with bins, right? There's an analog of nearest with bins. See what I'm saying? Right.

Although we already have a. A rule syntax for bin lists.

I understand. I'm not telling you the syntax. I'm just saying, functionality wise, if we had a function that took, you know, a column or took some, you know, because you might want to standardize columns, you know, taking the bins from the naturally occurring whatever might not be what you want. You might have 10 tabulars that you all want to, you know, bin every. Every time you have this thing, you bin it from in, you know, blocks of 10 and 0 to 100 type thing. Right?

Yeah.

So it wouldn't be useless, I think, to have a, You know, categorizer function that's not a good name, but you get what I'm saying?

Yeah, I guess what you're saying. Yeah.

That then returns a category name. I. I kind of like this idea.

Yeah.

I'll. So for the next meeting, I'll. I'll bring up how this number categories actually works right now, and I'll look into this idea of the categorizer function, because I think I actually have something a bit like that already, but I just, you know, it's been six months since I worked on it, so I just don't remember that.

Okay, fine.

So some things I know for these categories and need to look globally on things like quantile stuff. Needs to look at the whole thing to.

Yeah, it has to look at the whole column.

No, no, but the whole point. Roger, Roger, the whole point here is once and for all, you run the categorizer function on the whole data. Okay, okay. Or on some faux Data that's always 0 to 100, let's say.

Yeah.

See what I'm saying?

Yeah.

You get back a categorizer function and then, then every individual piece of data now immediately gets told, well, what category is it in?

I mean, I guess they all could return. Could have a version that returns one of those functions. I mean, but the, the recipes, the recipes for how to describe categories. I think it's, you know, like for strings and, and dates.

I know it's different. Right, right. But nearest, for example, we basically.

We basically already have that sort of.

Right.

We have classifier function. Right. Learns on some stuff and then you feed it an arbitrary thing and it.

You're absolutely right. Yeah. Although classify. Yeah, Right. Classifier function goes to a group. But I'd like to point out that nearest works on strings and numbers and images and all kinds of things. Right. So I think that this categorizer could work the same way. And the, the issue is this is for things. Well, I think how does this relate to clusters? Because we also have clustering functions of various kinds.

Yeah. So I think that those can also be used to sort of discover categories, if you like.

But like as an example, you might.

Pick some number of columns and feed it sort of the data set that that represents all the way down and you get clusters.

But the point that I'm making is what comes out from nearest, from classifier function, from clustering and so on. All these things must be usable as group indices or group specifiers.

Yeah.

Okay.

I think, I think that the ones that we have here were sort of. They're kind of aimed at being author friendly. You know, you can express a desire.

These ones that we have here, the ones, the ones like.

Number. Right. I, I think the point is that sort of. Right. Number categorized that.

Right.

You can easily specify what categories you want. Right. Clustering is. You're asking some algorithm to go and decide some that have grouping for you. But essentially they're the same operation. Right. You're taking a bunch of points and you're splitting them into groups.

Yes, yes.

But in some sense I think people will want something simple for number like.

Oh yeah, yeah, yeah, I'm right. I'm not, I'm not, I'm not. That wasn't an argument not to have dysfunctions. That was just explaining. Yes, these are both the same thing. But how you express your desires is differently in, you know, in various clustering things. Either you let it do it or you give it some sort of machine learning algorithm to decide how to group things. And with, you know, bin list, you tell it. Well, I want, you know, 1 through 5 and 7 through 23 as my groups. Right. Or I suppose 1 through 3 and 4 through 9. If since it's GCSE day in England today.

What?

It's GCSE day that people got their test results. So 1 through 3 are not passing and 4 through 9 are passing grades.

Wow. They've changed that in the last 40 years.

Yes, they have.

That's interesting.

What used to be called the O. Well, I guess they're combination of O levels and some other thing that was even lower.

No, no, no, there was never anything below O levels.

Yes, there was, but never mind.

Okay, well, sometime in the last half century it got changed. Okay then. All right. I think we were supposed to go on here. We had been looking at. Okay, I'm understanding better this categorization thing and I think there's more thinking that has to go on here about how you, you know, to me, okay, I'm going to give my, my immediate con. You know, thing is that there's almost a categorize columns or something function that where, because it, it can see perfectly well, oh, those are a bunch of numbers. See what I'm saying? For a given column.

And I mean the type will tell you that.

Yeah, yeah, right. So then it's really a question of, of then what is the binning, right? Or even, you know, bin columns.

I just think it's going to be an unwieldy function to describe if it's.

Describing all differently because you might, you know, you might like in the Titanic thing, you have ages and you might want to bend those in a particular way and, but you might have some other column numbers which you want to bend in quite a different way. So I think it's kind of a column by column type of thing, really.

Yes, I, I agree with that. I agree with that. But I'm, I'm saying, look, it's a question how people do this. For example, it could very well be the case that I'm going to think about this one column at a time, okay? So that I say I've got a tabular, I bin columns on column, on the age column for young, young, medium old type thing, right? Then I get back another tabular which has that added. Then I say, oh, now I'm going to bin by sex or something, right? So I'm saying I do it again rather than putting them all in some big complicated specification in a single function. I mean, you could, you could say the same thing. It's not like, it's not like we don't get back tabulars that are then reusable. Right, Right, right, right.

So, so, right. So, and, and you know, maybe we, we should postpone this until we actually dig into the details of the function. But one thing that's not entirely obvious from the spec of if you're giving sort of multiple columns, is it binning independently in each column or is it binning?

Maybe it shouldn't, maybe it should just be doing, you know, one, one shot. Just like, I mean I could, I could imagine a gazillion functions where we could be doing three different things at the same time. And sometimes that's necessary, but often it is not. See what I'm saying? I think this might be a case where, and I really like the term binning, bin rather than categorize, I think it, you know, you can bin Strings, you can bin numbers, you can Bing dates. It's all good. And it fits in with our bin lists and bin counts functions. Make sense.

Yep.

Yeah. That really helps me to even understand what the heck those functions are, which I didn't understand before. Okay, should we go on to visualization, or do you want to.

No, no, that's fine. I think that. I think that we can. Yeah, I think it's appropriate. So for tabular visualization, that guide page is not complete. I mean, neither of them are complete, actually. And I think for visualization, we want to kind of turn over to Brett and Normal to kind of just give. Give an overview of where we're at and.

Okay. I mean, so if I give this tabular taboo, I need to look at what it has in it now. And I say histogram. How. How might this work? Well, that's dead in the water. Yep. Okay. Comments, guys?

Yes. Do you want them to show you.

Or they want to share or are they going to tell me something? Can you speak, please?

Yeah, there's no documentation or. There's very little documentation already. So.

Brett, but can you please say something? Brett?

What.

What should this have done? Brett, I think you're on mute. Can you take yourself off mute, please?

Well, it's not obvious to me that that should have done anything.

Is Brett here?

Okay, he's having some. That apparently his headset is not working.

Oh, I see. So. So with histogram, you can give it a rule, taboo rule, and then H as a string.

Yeah, right.

I.

Right.

I was going to say the one argument formula wasn't that it should do. Do anything, because I don't know how you. Histogram, five columns.

No, I. I understand that. I understand. I got it. I got it. Right.

I mean, also, I mean, you presumably just do taboo part all comma, age, which is kind of the. I mean, seems like the clearest way to do it.

Okay, so, by the way, error messages needed for a case like this.

I was actually wondering, though, whether I actually encountered this the other day, whether histogram could support sort of nominal scale stuff and basically be the same as, like.

I think it would be nice.

Bar chart account, essentially.

Yeah. Well, I think ordinal scale. Well, okay, there's nominal scale and there's ordinal scale. Right?

Yeah.

I mean, this is. I don't even know whether this is an ordinal scale here. Class.

I mean, class could be an ordinal scale, but I mean, but. But they're also. I mean, we already do.

I mean, when you're using the. The word first. You're by definition in or in ordinal land.

Right, I. I know that, but it's a question for the. For Roger and so on. In our ordinal scale technology, do we have something for the words first, second, third, and so on?

No.

We probably should.

Yeah. So it's another meeting. I'll bring up a lot of things that we could do to curate things.

Oh, we should do it. We have a whole giant pipeline for doing curation.

I know. That's why I'm bringing it up. So in some sense.

Almost heard you, I think.

Almost.

Say it again.

Okay. Now we can hear you, Brett. Or at least I can.

Okay. I have no idea what was going on.

So. Do you want to share? Is that the idea?

I can share.

That's okay. Why don't you go ahead and share? Can you initiate. Can you start sharing, please?

Yep, I'm sharing.

I don't think you are. I don't see as I'm sharing.

You need to go up to the top and. And choose into the view options and switch from seeing watching Steven scream to watching Brett's screen. Or. Or Stephen can, at this point, stop sharing.

It's not seeing your screen at all.

Is anybody else tab at the top of the zoom? Yeah, I can see you have to click it.

I. I can switch over to Brett when I. When I'm watching.

Fine. All right. Okay, great. No, it didn't work for me. Oh, yes, it did. Okay, there we go.

Okay, so you see the screen now?

Yep.

Okay. Excellent. So we have, you know, the visualization.

Start using that toolbar, by the way, that you're using that says X kernel and so on. Is that in wires?

No, it's very specific to my machine.

Is there a version version of it.

That you put in stash somewhere for people?

I mean, it lives in stash, and various people have, you know, taken wires.

Can we put it in the internal function repository?

Maybe. I'll think about that. I mean, the other thing is that, you know, like I said, you know, everybody who has, you know, done a version of it, you know, has put their own little things up here that are useful to them.

That's nice. But we can have five of them in wires. It's fine. Yeah, but having. Yeah, okay.

But I'm glad you like my toolbar.

Well, I only like it in appearance. I don't really know what it does, but go ahead. Right, okay. So now, what's this LV one. What does that mean?

So we have level one Is, you know, basically what you saw with the histogram. You know, so we have a basic support for tabular and then an arrow syntax.

Okay, can I see that? With list plot with two different arrows. You understand what I'm saying? Two different errors, that's what. Back up, back up, back up, back up. Okay, back up to the first example there. Okay, that's, that's nice. Okay, so that's what I was asking for. The X thing and the Y thing.

Exactly.

Okay.

Right. And so this is X. This is Y. I know, I can see that.

That's Y. And X is just the sequential thing.

Yes. This is still just Y.

Yes. Okay.

Is there a difference between those two inputs?

No, no.

Okay, it's just, you know, this is easy. This is.

Oh, one has a list. Okay, I, I, I missed.

Yes, there is a difference in the inputs, but it's just that list.

Okay, so, so if you had double lists. If you had double lists, the. So I'm trying to understand if I want separately plotted the body mass, the, the length and so on on the same scale. I'm confused now.

Okay, so let's look at this example now. Right. Okay, so this is body mass on X. Built depth is one is the blue on Y and bill length is the yellow on Y.

Okay, so if I just had to try this next, next cell, please just get rid of the body mass. Delete the body mass. Just delete the body mass argument piece and leave the rest.

Yep.

Okay, now run it. Okay, so those are the two. Why are they not two different data set colors?

Because this is one data set. This is.

No, no.

Build depth on X.

No, it should be the X coordinate. No, that seems right.

No, it's right.

No, no. What if you want multiple pairs, right.

You know, you could have X1, Y1 and then.

Yeah. Okay, so what happens? So then, then, then. Okay, so something like none and then. Or automatic comma. No. What? Okay, what say automatic comma and then a different attribute like body mass underscore G. Second, second one automatic comma, body mass underscore G.

And that's going to fail, I'm sure.

Okay. Do you think that that should work?

What, what is automatic? Oh, the one.

Current syntax of.

Right, yeah. To mix an XY thing and a Y thing.

Yeah, go ahead.

I mean, I would.

We're going to have something to make that work. That seems like as harmless a way to do it as any, because we haven't needed that. What's that?

If you delete the automatic. Will it work?

I don't think so. I mean, it did something, but I don't think it did what, you know, he wanted.

Right.

Yeah.

So wait, here's a question. I mean, why so, okay, first of all, if you want to do multiple data sets, can you put. Can you wrap some of these rule things in a list? Like, could I have a list of two of these? Does that make sense?

No, I mean, that's.

Right, that's getting into, like, these sorts of things. Right? I mean, this is.

No, no, no, no. Christopher is saying. Christopher is saying, put the penguins in a list. Okay. In other words, you've got.

No, no, no penguins in a. Just exactly that example there where you got the blue and the orange. Could you have written that by having a list where you say penguin's arrow, bill, depth and bill length, comma, penguin's arrow, body.

That's what I meant by putting the penguins on a list. Oh, yeah, right. So. Yes, try it. No, no, no, you're not doing the right thing. You're not doing the right thing. It's an outside list, a list around the whole thing. And you repeat example, tabular data of penguins. Do you understand what we're saying? Oh, oh, at the beginning. Put a list at the beginning of the list.

Yes, yes, yes. So you're talking.

Right?

Yes, now, now get rid of the first piece there. Whatever you want to do. Right. Try that.

I've still got an extra.

Yeah.

Okay, so we've got length and depth and.

Yeah, okay.

Yeah.

Okay, so that second one is a little miswarmed, but. Yeah.

Right.

But. But it does get a little complicated if we support the above syntax.

Right.

Because if we support both of those, then that means that we have, like, you could do two layers of data set lists, if that makes sense.

Yep.

Yeah. I mean, it gets, you know, potentially.

Way, you know, but so, so why not just do that? So in other words, we haven't had that issue before. Probably. We've got, you know, we've got a flatten and color. Does that make sense?

Hold on.

Sorry, I just don't understand. I mean, what is the. What is the. What can you express with the, like, above syntax you can't express with the below one? Like, why. Why have that above one for doing well?

Because it's convenient. I like that one. I think that's a good one.

I mean, one of the reasons is that, you know this, right? I mean, so this is level one, right? That table at the top at level one, two and three, you know, and we're looking at level one and level two for the 14, two, you know, timeline.

Expect to do the things that we're.

Going to be getting into in level three, but which gets more complicated, especially for things that aren't just list plots, is you might, you know, having something that is some sort of syntax like, you know, plot style goes to island or something, you know, something where you're then saying, go, do go. Use this other column to pick out, you know, pick out a color or the plot markers or whatever.

I understand.

So because categorical variables effectively pick out subsets. Right?

Yeah, Groups.

Okay. So that's often how you embed it in sort of one tabular. Right.

I think the thing you've just defined there as an awkward is going to be a mess. And I would argue that. Okay, I don't necessarily believe this yet, but something like, you know, visualize tabular, and then it's a pure list plot where, and this may be crazy, where essentially the strings are like data. See what I'm saying? So that. Because this idea that you say plot style, arrow, island without reiterating the tabular there, I think that's going to get very messy.

And when you say style, comma, Times New Roman. The Times New Roman might be a column in your.

Yeah, right.

Right. I mean, so.

Right.

I don't think island would be the liberal.

Right. Originally.

Right.

I mean, the actual syntax, you know, we were prototyping, you know, just to have it not interfere with anything else. And I am sure this doesn't work, you know, in this prototype build, but something like.

Right. And I. Yeah, this is pretty.

Thai. Hates encoding.

Not thrilled with, with using either.

No, these are both crazy. This is both crazy. But, but I get the point. That. And so then I don't think it would be a bad thing.

4, 4 different such dimensions that we can use.

Yeah, we've got size of blobs, we've got this, we got that. Yeah, right.

I mean, ultimately, what you would, what you kind of want to emulate, like the most explicit way of doing this would be if you just map over the tabular and then say, you know, list, here's my X coordinate, here's my Y coordinate, here's a style around it, whatever. Is the point. Just that that's really inefficient or.

No, I think this is part of the exploration phase of a tabular, you know. No, let's see if this, this, this factor has an effect.

Is.

No, sure. But I'm just saying, like, I mean, it seems like kind of, I mean, at least to me, it Seems like the simplest way of doing this. The way it's like the clearest. You see, you know, there aren't like rules and lists and whatever is just to map over it and make a bunch of styled or otherwise.

Yeah, right, right.

You create a new tabular that has an X column and a Y column and.

No, but, but Christopher is making a different point, I think, in the data, one of the things that we've done that's pretty nice is to have the possibility of symbolic wrappers in the data. You see what I'm saying?

Yes.

So, for example, one syntax I can imagine on the right hand side there, what we've got is just like you can say, if you say list plot, just type in list plot. Okay. List plot of, list, 1, 2, 3, comma, style, 4, comma, red. Okay?

And this one is red.

Right. So. But my point would be maybe what could be done is on the right hand side, where you have bill depth and things that could say something like style of bill depth, comma, you know, island or whatever it is.

No, well, I mean, no, that's not what I was proposing.

Literally map, you know, I know what crystal style of pound column A, you know, color, you know, color, color data of, you know, 104, comma, you know, pound column B. And, you know, explicitly construct the things, you know, that you're going to plot. And I think maybe that just doesn't scale very well. Right. You know, this automatic behavior, you know, so you only specify the minimal thing that you need, and then it can figure out everything else it needs to adjust.

Well, but that's sort of the question. I mean, there's a few. Well, first of all, there's two different senses of scale, because, I mean, I know that sometimes when you do stuff like that, it also just is really inefficient. I don't know if that is fun, like it renders slowly and stuff, but I don't know if that's a fundamental thing or what. But honestly, I mean, in the examples that we have right here, I think that like mapping over tabular is going to be no longer than what we've written so far.

I know, but it also requires a higher level of understanding. But I would.

No, I think it requires less understanding often because it's just like you're just explicitly constructing, you know, how to make a list of pairs. And, you know, list plot takes a list of pairs. There's no magic new syntax. I mean, I'm not really trying to, you know, complain too much about the magic New syntax. But I just want to understand, like, you know, what is the value in having a super, super flexible syntax that can do everything? If you could, if it's really unclear and you can just map over your list and use style and other wrappers.

I just want to understand the thing that I'm proposing. Could you write it out the right hand side and that second thing, that second one there. So get rid of the color encoding. Get rid of the second one, second one, not that one.

Gotcha. Okay.

Okay. Get rid of the color encoding and instead wrap around, bill, length of millimeters, wrap style, comma, island, or something that isn't quite right. That's not quite gonna write, but. Right. Or maybe something like color category or something of color. I'm not sure what the right thing is.

If instead of island you had red there, would that work?

Right? I mean, so like, imagine it right now, like if you put red thing right. I mean, this is. I know it's not going to work.

Well, I just didn't hear you and actually say that it wouldn't work until that last.

Well, yes.

Okay, let's zoom out for a second. Okay, what we've got, you know, list plot as it is right now, takes things with XY coordinates and renders them. And as sort of an afterthought, it puts other kind of styling on them. Right?

Yeah, that's pretty fair.

So an alternative is something, and this is possibly a more general sort of transducer option function that takes, you know, just like we have in Graph, for example, vertex shape function, which can basically take. Which can do anything you want in terms of how the vertex is rendered. Right, right. So similarly here, what. What one might be saying is you're feeding in some number of parameters, bill length and body mass and island and all kinds of other things. You're feeding in a bunch of parameters. And the fundamental thing you want out is a representation that we don't have particularly cleanly right now of sort of the parameters for how to render the thing that comes that you want to render. You don't have particularly clean.

Point value plot does that.

What's that?

Point value plot does that effectively.

I never use this. I probably should, but I never use it. I forgot it existed. What version?

And is this one of the things that's called plot, but should really be a chart?

No. So this was older than I thought.

Have you ever used. I've never used it. Hey, Christopher, have you ever used this thing?

I think I've used it like once.

I didn't remember it existed.

Yeah, I mean, it's.

I think, Yeah. I think it. I think my, My only issue is it doesn't scale super well when you have a huge number of points.

And like the one time that sounds probably about right. Yeah, that.

Well, that should probably be fixed, maybe that the.

Yes.

Right.

But basically, if I understand, Stephen, you're proposing perhaps like, you know, a rendering function for.

Yes, that's what I'm proposing.

And we have. I mean, as you say, there's other things that do have that, you know, all the graph stuff charts, you know, were probably, you know, one of the first things that went down that path. And you know, I. The issue I see with it for something like list plot is it seems like it would be a very inefficient approach.

Okay.

Which.

That, that's a question of you figuring out how to compile it. I'm not, I'm not going to take that as a. As a. No, I'm not. I'm not. You know, you can solve that. I'll help you with the engineering if you really need it. But I don't think that's an.

One thing that we've found with this is that it's a very general solution. So maybe we should do that too. I think, I think that there should be a slope.

I mean, I completely agree. And I, and I also agree that, you know, having a general function there that isn't compiled doesn't make a lot of sense. Right. We obviously need special cases.

Yeah.

But.

But the other thing is that. So we can do every. We can do an awful lot of things with style. Not everything, but we can do a lot of things with. Okay. And. But the thing is that you are responsible for constructing everything.

So we don't want that.

No. And so if it's going to be sort of some reasonable ease of use, it kind of need to fix, you know, common workflows. Need to be easy, basically.

Yeah.

Bite size pieces.

Yeah, well, right. But I mean, so for example, the fact that, you know, okay, you want to also tag which island the thing was on. Okay. We don't have need to. We shouldn't need to have to say, oh, I want these particular colors and I want a legend for these to show what they are.

I mean, if you want. I mean. So the other thing is when we start to use graphics more and what our scientific vis. Functions do, they don't really use. Okay, how should I express this? They use geometry as their fundamental visualization primitive mostly. There's a few exceptions, but most of what they do is to show surfaces and curves and points and things like that.

Because they've got color functions and so on. They've got mesh functions.

They do, but they are sort of secondary except for things like density plot or contra plot. And. Jesus, I lost my train of thoughts here. But it's so in plot style, if we go back to sort of list plot, right. You can. Let's say that you wanted to have two directions like size and color. Then you need to generate the tuples of how it's going to split on the different data sets. Do you see what I'm saying? Because we have one all you can eat only kind of style. So you can do everything. But you will have to enumerate the cases even though they kind of go in two different directions.

Yep.

And if you can have visual, visual markup like, like say color and size that go in two different directions, then they're independent of each other and you can decode them both. And we can automatically generate a good legend that explains what the heck's going on or how to decode it. If we have a programming only type solutions, there's. We could never generate a legend. You would have to manually understand. Manually do it every time.

I get it. Right. So yes, I agree that we need, you know, but, but go back to your thing, Brett, your, your summary. Okay, so one of the issues is we're not going to have, let's say we have some visualized thing. These plots all have different kinds of characteristics. We definitely do not want to have a, you know, a new family. We've got plots, we've got charts, we've got visualizations. We don't want to do that. We somehow have to shoehorn the syntax or something into these existing functions. Do we all agree that's.

That's, that is what we're doing?

Okay, so, you know, there really, to me, are two issues. One is the convenient extraction of columnar data from a tabular.

That's without losing key information like type.

Yeah, well, fine.

And label.

No, wait a second. Wait a sec. Yes, okay, fair enough. Where's the label? Where there is. We don't have labeling yet.

Column names.

Oh, column names. Okay, fine. Yes. Okay, so first thing is the convenient extraction of columnar data. The second thing is because we now with tabulars typically have much higher dimensional data because the tabulars have many columns in them, we are interested in visualizing more degrees of freedom in the plots. Yes, fair characterization.

Exactly.

And anticipating where you maybe is going is that we can in Fact, extend the list syntax to take not only pairs for list plot, but it could take higher, higher lengths.

That was where I was going, right.

Okay, so we've already planned that out, that we will then extend them and we will be able to, you know, these other visual dimensions. These visual variables can pick off other dimensions in that.

Right? So, okay, so then the obvious thing to do is there's a default action. Just like we have default colors for different data sets and so on in a list plot, if you have four different variables that you're defining. So in other words, and it should be the case, even if I have just regular numbers, if I just put the data in, in literal form, instead of pairs in listplot, I have quintuples in listplot, right? That those extra three variables will be rendered somehow. And there should be some function, some, some option that is the disentangling function that tells you, you know, beyond plot style. Because plot style doesn't give you any freedom. You know, in a sense, even a plot style function would give you a better, better piece of control. Just like we have color function, which applies itself to the values, and then we have to have color function scaling to deal with kind of patching that up. What you're basically saying is given the values sort of, how do we actually. How do we actually render them? Which is similar to the scaling function, which I think is working very nicely. I mean, I use that all the time of, you know. Okay, so let's just zoom out for a second. Okay. We've had things like list, log, plot, I like list, log, plot, it's useful, etc. Etc. Etc. But when I want to do scaling on some random kind of plot, I use scaling function, and it works nicely. Right? And so what we're saying here is if you've got the fourth dimension of the list plot, quotes, coordinates, and then you want to say something about how you render that fourth coordinate. That's kind of like the scaling function story. In fact, you might even use scaling function to, you know, you might have a fourth in the scaling functions there. What is the X scaling function? The Y scaling function. And then actually, so that's kind of.

What we put in to those new sort of visual encoding options that you could give the scaling that you want right there.

I understand, but I think you should have the possibility of a scaling function.

Because the typical tabular case, I think there will be many more columns than the ones you want to plot. I mean, you're gonna pick out, I don't know, three to five columns or something like that. And maybe it has, you know, I don't know, 50. Yeah, something.

So what?

And so you want to point. So you don't want to give 20 scaling functions.

Absolutely not.

Right.

Okay, let me, let me paint another picture here. In the metadata of the tabular, it might be that you would like to leave lying around. Oh, by the way, if you plot this, scale them this way. You see what I'm saying?

Yeah.

It's an interesting idea.

I think it's a useful idea because, for example, I can imagine something where, you know, if you plot, I don't know, the male and female thing, you might want them. I never remember which way around it is, you know, pink and blue, whatever it is. Right. The, You know, there are things where a default rendering is just like we have a label potentially coming as metadata. Default rendering seems like an obvious other kind of metadata.

It's possible. I mean, I think the thing with these compound objects like tabular kind of.

A little bit already. I mean, it's just, it's a step further than just using the type information.

Sure.

Right. So like, you know, if it's a date column or it's a categorical column, we have scaling functions that deal with those sorts of things already.

Right. But I mean, I'm just coming back to all these functions we're dealing with. Okay. A fundamental principle could be sort of, and this is somewhat orthogonal to tabular, but it immediately fits into tabular. Okay. There are two concepts here. Concept number one, we're going sort of multi dimensional, for it isn't really dimension. We really shouldn't call it that. We're going to.

I mean, there's sort of visual. Visual. They're kind of visual dimensions.

I know, but it's kind of multi factor or something. Multi. Whatever you want to call it. Anyway, we are. And a first question is with the syntax of all these functions. I know there are a few hackerrated functions which allow, for example, listplot3d I think allows two different syntaxes. One is lists of triples and the other is an array.

Most of them, except actually all of them in the list family.

All of them do that. Stephen. There's a sort of what we call regular and irregular data. So when we get this sort of array, it's assumed to be.

Yeah, okay, fine. Okay. And that's probably sufficiently disambiguated.

Well, well, I mean, I mean, that was going to be one of the things that I was going to clarify with you is, you know, if I do list plot Of A, B, C, D, E, F. At the moment that is one set of Y values and another set of Y values. But it sounded like you were suggesting that this would be one data set potentially with XY and a color parameter or something.

Yeah, yeah. That's tricky, isn't it? Yes, I agree that's tricky. So I mean, maybe that has to be switched. I mean, because most of the time it's going to be dead obvious. Right. But in these small cases it's not going to be so obvious.

I mean, I don't know that it's going to be obvious much of the time.

Oh yes it is. We're not going to have. Look, most of the time you've got 100 data points, right? You certainly don't have 100 coordinates. You're going to be trailing off by the time you've got four or five coordinates.

But tabulars can have.

Right, A hundred, sure, they cover. That's not what you're seeing here. In the case of the tabular, you're going to see arrow and then three or four different things. I mean, in the case of the tabular, the disambiguation, you don't need the disambiguation because the tabular is disambiguated by having rows. Okay, There is.

Tabular is disambiguated by being tabular.

Yes. Right. But this case, right. There is an ambiguity here and we can't change the default. And so in the rare case where you know, you're getting, you're representing things with. It requires thought. Okay.

Yes.

My guess is that this will be some option based thing that will deal with this corner case. I mean, it's ugly to have the corner case because, you know, you'll have an array of pictures and one of them will go crazy. That's bad. But you know, let's figure that out.

Okay, let's do this list, line plot because it's easier.

What the heck was that? Yeah, okay, I get the idea. Right. Those are ambiguous. Right. But that has not bitten us very often.

Well, in the first couple of years after we launched six, we had a lot of tech support and other, you know, issues.

Okay. All right, we need to wrap up soon here. But just so I understand it, it seems to me we have those two studies, right? There's two different things. One is multi dimensional 18. So if you go back to your.

So our plan is not to do that now, but it's for 14, three.

So. So can I ask you. Right, there's.

Fine, then don't try and hack in these other things, you know, have some quite ponderous syntax for them. Don't try and hack it in and wait until we can really do that.

Yeah, I mean, there's a bunch of things we need to sort out and iterate on.

I mean, don't try to hack what in? I mean, so, so to be, you know, for clarification, you know, just be.

Be happy with, you know, do what Christopher was saying. If you really want the island tagging and this and that and the other, just write it in the code. Don't try and hack it into some clean, you know, nice syntax like this.

It's not. But what we're doing now to explain what this level means, we're not supporting anything more than the functions support classically.

Okay, that's fine.

So it's just.

We backed out of that because there's several things we need to iterate on and, and better understand. And it takes time. And if we spent all our time on that, we might have shipped nothing.

Okay, okay, okay. I mean, the only way to ship a perfect product is to not ship a product.

Right.

But we usually do converge. It's just that sometimes, you know.

Okay, all right, fine. But I think it's perfectly reasonable in. I think it's useful to have looked ahead and to realize that the, the, the path ahead is to this multidimensional data and then a decoding mechanism for the multidimensional data. I like that very much.

Right. So we want to present things around that in that.

That's post 14.2.

This post 14.2.

Yeah. Okay. What we're doing is just this accessor mechanism.

Yes, exactly, exactly. Not trying to do anything else.

Right.

Can I look at your chart for a second?

Yeah, sure.

That's where we're at.

So the ones that have warning signs.

Like, are not for 14.2.

We decided, I mean, we have a huge list of functions. Right.

Yeah.

Okay.

I just want to understand the ones that are there.

I mean, so the ones that have warning signs right now are the geoplots and the vector plots.

Hold on, hold on, hold on, hold on. Let me just understand them. Complex list plots.

We think it's unlikely that you're going to have complex numbers.

Yeah.

At least initially. We'll fix it. But I'm just. If we have to prioritize.

No, I, I understand. Geolist plot. Let me just think about that process.

So we hope to get a, A geo column type.

We don't have a geoposition lat long column type.

Geopositions A geoposition column type.

Yes.

When we have that, we want to for sure enable all the. The GEO functions. And if we have time to do it now, we'll try to do it now too, but it's going to be a little bit.

That doesn't seem so controversial.

Right.

Number line plot.

Well, we'll probably.

Number line plot.

This just had a. It has its own. This is very parser dependent and if something was written not by us, it doesn't follow the standard and we have to do special work that's all coming.

Right.

I mean, the ones that don't have any icon, we still intend for 14.2. We just haven't gotten to that function yet.

Can I look down? Yeah, sure. The vector ones. Yeah, I mean, that involves a level of structure in the data that is probably not super common for. Tabula.

Exactly.

Well, I mean, the other thing that comes up with the vector and that we ran into when we were doing the geo vector plots a few versions ago is, you know, sometimes vectors are represented cartesian. Sometimes they're represented.

Yeah.

You know, in polar.

Yeah.

And so, but.

Yeah, okay.

Right. So just to be clear, we are going to have the new rule syntax for 1421.

14 2.

14 ones out.

Yeah. 14 2. Sorry, is the one you wrote the blog about, Stephen.

I'm losing track.

We all are. That's okay.

And is. Is there a way to have, you know, plot a data set on the X axis and have Y be automatic, or do I need to, like, transpose my data set if I want that?

There is no way to have the y values be 1, 2, 3, 4, 5, etc.

Yeah, there has never been. There's never been right code. And I. I've done it, you know, a bunch of times in my life, and it's okay. All right. I should go to my other meeting, but this seems to be progressing. I mean, we got more to discuss. I really like this idea of the multidimensional. Multidimensionalizing these plotting functions.

So do I. And we hope to come back with something really interesting.

Sounds good.

Okay.

All right. It's amazing, you know, one would think we'd be finished by now. In general.

That'S not how it works.

Still getting longer, right?

It's just getting up to. And you just level up to the next level. And there's a whole other.

Yeah. All right. Look forward to all of this stuff. Okay. See you guys soon.

Sam.