Live CEOing Ep 827: Language Design in Wolfram Language [Tabular]

Sa.

Okay, Hi, everyone. You are jumping into the middle of a design meeting about Tabular, which, for various reasons, we were not live streaming until right now. We were talking about several things, but maybe we should go back to the. What we were talking about at the beginning, which had to do with select indices problem that we were discussing last time. And we had a great suggestion from Sander on our live stream to use the same mechanism that we've used for nearest and so on, and to give additional properties in select. And I really like this idea.

Sure. Then I'll let Rob share again his notebook. Are you ready, Rob, with your. What you had before?

Rob, can you start share, please? Rob, can you. Rob?

Yes, he's coming.

Okay.

Is it just me or all people only seeing a portion of a notebook?

No, I'm seeing the whole notebook.

Nope. See a little bit.

Ah, maybe I need to share. Where's the agenda? Is it in Dropbox?

Huh?

Under tabular folder. And I don't need to use. Do I need to. No, no. Oh, gosh, guys, you know what? Whatever. Actually, I do not think I have a mechanism to share here. Well, let me see. Maybe I can do something.

Okay, should I go back to sharing then?

No.

No, you should not.

Okay.

There you go. Oh, lovely.

Okay.

All right. So last time we were talking about select indices, and the suggestion from Sando was use the same mechanism that we've used for nearest. So let's remind ourselves what we've used for nearest. And I also want to discuss other functions that are similar to these. Okay. That are similar to select where they take. So in the case of nearest here, I'm going to paste in the. What we have is element, index, distance. So that's for nearest. Okay, so what other functions are like this?

Am I making largest?

So by. By doing it like nearest, does that mean rule goes to pure function? Rule of element means.

Yeah, yeah. I mean, no, it means select of list. You know.

Yes. List of stuff.

See, whatever hash greater than zero. Well, this is interesting. I mean, is it pure function goes to. Index, for example. Is that what we're talking about?

Yeah, that's what Sandra suggested.

I think that's certainly the nicer order for a rule from a, you know.

Parsing point of view.

Exactly.

Well, I think it's also semantically better.

Yeah. My question with nearest.

Yes. But my question is what functions are there like select that could have take.

Largest has this precedent as well.

I should say that in select and take largest, we put the arrow in the first argument in the data.

I see. But that's not going to work for select, is it?

I mean, I think it does because one doesn't select from rules.

I see, yeah.

But how can that be an operator form?

Or presumably allow it in both places?

No, that's a complete mess.

Yeah, because that's the operator form then.

Yeah.

Okay, what are the other class of functions like select. Okay. What are the other functions that are like that?

Cases.

No, cases has an arrow. What about some of the ordering functions and so on? Anything there. Let's.

I mean, we built ordering as the index version.

Yeah.

Of sort.

Indeed. By the way, just a note on the select page, the related links thing, somebody went crazy here. There's no reason to have creating websites and apps in the related link from select and it should be removed. Am I making sense? Do I need to chase that further? Carla, can you deal with that, please? Yes, yeah, I can do it. Okay, now select first, for example, presumably it too has this kind of mechanism.

Coming back to cases. We have discussed other times something that could be called position cases. So you want to find the cases and the positions at the same time.

I see. That's not a terrible idea.

I mean, perhaps there is an extension of position that allows getting indices at the same time. It will be positions.

In this case we get the.

Getting the elements, the value. Sorry, sir. Yes, I said wrong.

Yeah. And likewise position is the index case of cases, right? Well, well, sort of.

I don't think there's any way to decorate position, because I think that you want the position of anything in any expression. So I don't think that's a decorative decoratable thing. Ah, Sandra is commenting on our live stream, which I'm finally able to see here. Group by also has a second argument that is a rule specification and what. Let me just look at that one. I see that that's. That's to do with its grouping structure. What else? What would position cases return? An association or what?

Pairs. Pairs of the case and the position at which it is.

Except if it's called position cases, it's got to return position first and then the thing.

Yes, yeah.

So position cases of abc comma C would return. What, 3 comma. It's got to return yet another level of bracket. Right, Right.

So imagine that you have C comma v comma C. It has to return the list of positions of C and.

And then the C. I don't know. This seems like a mess. I mean, it seems like a mess to be able to actually use this. Again, I'm coming back to what else Is like select.

I mean, not many things are like select. It's a bit of.

Well, thanks for asking. The new symbol that we are proposing.

What new symbol?

Scroll down like every little bit. The names for the opposite of select. This is very much like select.

Yeah. Discard is better than reject.

It's interesting that we've built a bunch of other things like position largest, position smallest, which is basically type index.

Yes.

Well, take, take largest or take smallest is actually the overarching version of sort because it'll do everything with this rule of syntax.

Okay. And the reason we want discard is because we want an operator form. Is that correct?

Yeah, mainly this is gonna. I mean, because, because it's very far. I mean, it's like why you want delete cases.

Of not.

It is. It's exactly that. Select with some. It's select if you want an operator form is select of not composed with the function. So it's select of inside the brackets, not F, not composed with F.

I see.

So this is what this card is. And in the same reason why we have.

I think that's assuming that everything evaluates to true or false.

Yes, true. But select has a true queue implicitly done. So. Right. Yeah.

We'd have the same implicit true Q for.

Right. I'm just saying for undefined cases. Will it.

Yeah, yeah, yeah. We need to define what the undefined case is if you want to discard by default or not discard.

Well, this is not discarding by default.

No, it's not. I agree with this proposal.

I think that's the right way to do it.

Yeah.

I mean, select reject is the antonym of select. But I kind of don't like that. You know, I think it. Discard. Yeah, discard is better because. The thing that comes back has a bunch of elements discard. I don't know. I, I, I just prefer the that word discard.

Yeah, I do too.

Yeah, I, I think we all. Well, I know maybe somebody disagrees, but that's also my initial instinct that.

All right, let's build it. Now, discard. In the case of the index thing, let's say we have discard with an index. Is that the index of this guarded elements or what?

No. Yeah. Now that I think your date is not the same as select because this, what do you want? The property of what remains?

I guess I think it should be the index. I think it should satisfy that thing I showed above is that, you know, discard with index. And then you take that part and it should give you what discard does. So it should give the, the, the part. The index of the things to keep.

I see. Okay, so let's look at the proposal here.

It's. I find that slightly weird.

Okay, so wait a second. This, In the case of nearest, the name that we're using for the actual data is element. Is that right? Yes, Element and index. Yeah. So element is probably better than value.

Yes, I agree.

All right, let's look at the, the next case. So the, the. So the next case would be to return.

A.

I mean, I don't see why we shouldn't have both things. The next case would be to return a Boolean list. Maybe. Yeah, it's a little bit weird because these are singular and they say return that for every element. Right. This is a. In this case.

Well, except it's not going to be every element. Right.

Index is going to be a list of indices, right?

Yes. An element is going to be a list of elements. Right, right.

But, but my point is that index and element are going to return length of kept things and Boolean list, or whatever we call it, is going to return a list of input.

Presumably that's, that's why I included the word bit mask, because it is, it is like a mask. So it's the length of the original thing. But, but you're masking it with the one.

So maybe Boolean mask is what we want.

Yeah, yeah, well, but they're really different things we want to return. So I mean, in this case, there's a Boolean mask, There's another possible name, which will be pick mask.

Yep.

Yes.

Yep.

That thought had occurred to me as well, that this is the thing you'd want to then give to pick.

Right now. Next question for pick. Can pick, take a. And this is something we haven't really talked about a lot with functions. Can pick, take a bitmap data structure.

Yes, I did that. I mean, in, in the prototype build, the answer is yes.

Okay, but I mean, as a general matter, how much have we looked through the system to see where the data structures can be eaten easily by system functions? And do we want to do that?

The bit vector data structure? I've done a fair amount with other data structures. No, not so much.

Okay, all right, but I mean, the other thing we might want here, there's a pick mask there. And what is the name of the data structure? Is it bitmap? Bit vector?

What's the name of it?

Bitvector.

Okay, so I mean, to me, I mean, I'm kind of thinking Lin mask, bit vector. I mean, I Don't know, it's a bit over overblown, but it gets the point across to Mask. I mean, it could also be Boolean, list mask, I'm not sure. Pick mask. But I think both of those pieces of functionality are useful and they're different.

Yes.

You know, pick will yet have its day in the sun. Now, in terms of keys or the all thing that we have in nearest, I don't see any objection to saying to having something which returns an association that has, you know, it'll say index, arrow, 3, 4, 5, 6, 7, whatever it is, you know, bit, vector, mask, arrow, data structure, blah, blah, blah. I don't see anything wrong with it. I mean, it's not. I mean, that's what nearest does, right? Nearest doesn't. Nearest doesn't have a separate thing for every.

It does. It's transposed in. In nearest.

Yeah.

Well, it's got its arrow on the other side, doesn't it? Now, wait a second. What, what is that showing?

It's nearest to 5. The element that it's nearest to is 5, the index that it's nearest 2 is 1 and the distance is whatever.

Yeah.

I see. So for every element there.

Right.

Okay. But we probably get a little bit of license from the fact that we're putting this on the right hand side rather than the left hand side here.

Yep, yep.

Sandra is commenting on the name Deselect. I'm not sure I. I slightly prefer discard. It suggests more of an action. Deselect suggests that you're just, you know, tagging those things, but you haven't done anything yet.

I agree with that assessment.

Yeah.

All right, can we. Great. Can we go on here? What are we looking at next? We're not ready to look at these icons yet, are we?

Oh dear. Well, what we put in the table of types with what's supporting the compiler, but it obviously somehow that form is not supported by your version of the prototype, so.

Well, I'm not running the prototype, so maybe I should run it.

Oh, well, that would explain it. Okay, So you requested we put in a table that have the types we have along with you know what the corresponding types in the compiler and the interpreter and input output are.

Trying to understand. Think. Do this. Let's try that. That work? Yes, that worked. Okay, you see this?

Yes.

Okay, there we go. All right, There we go. I do not like those icons. We talked about this before, right?

You talked about the fact that you don't like them. I don't remember having any substantial docu discussion since then.

No, we need. We need another discussion of sort of designing them, but they're incredibly useful. It's very hard to tell apart, like a number that's in a. As a string or a number that's a number.

Okay, so let's figure out. Let's make sure we have something there, but let's. Can we please get a meeting with Graphic Design going over those icons sooner rather than later?

Yep.

Okay. What are we supposed to look at in this table?

Who's scheduling that?

Carla, can you schedule that, please? Yeah, I can get that done, please. So.

So what I tried to do in this table is when. When the type is the same in the compiler or something else, I just use same to indicate that it was, you know, it was exactly the way we're using it in. In tabular column. And when they're different, I put in how the different type is represented in the other system.

Okay, so I don't understand. So let's take a look here. For example, dates.

Yeah.

What the hell does that mean? No, no, it's not.

We don't have dates in the compiler.

Oh, I see. Okay, okay, but so where in the interpreter, what does that mean?

We have date or date time.

Those are both types that. Right? Those are right. Both, you know, elements you can request from it.

What is it there in the data?

Object of different precision.

Okay, so we're using the type specifier syntax here. Is that correct?

Yes, exactly. So, so we could. We could, for dates, probably eventually have a compiler type, which means this. The question, you know, I need to talk with Tom about is, how do we do so. So we have these very annotations, like, what is the calendar? What is this? I mean, can we have those as type literals? And I think the answer is yes in most cases.

Listen, when you've got choices here, it's just not true that there's no version of that in the interpreter, right? There's a version. You know what? There's a version of this. What's that? I believe it's called choices in the interpreter.

We didn't find it. So.

What do you mean? That's no good. That's no good. Do I have to find it? I mean, come on. Well, we can find.

We needed Carlo, but Carlo's away this week, so.

Okay, well, then you shouldn't have done this this week. Okay, okay. It's a. It's vertical bars for a set of choices.

I think choices is a horrible name. Can you imagine?

Listen, listen, listen. In interpreter, okay, we are using simply vertical bars. We're just using the alternatives mechanism. Oh, several forms. A list of. Literal. List of choices is just a list. Okay. And then there's any subset. Am I making sense? And quantity variable. So in that case, what we've got is interpreter of, you know, abc. Don't know what that will do. Oh, for God's sake. Okay, there it is. So that's the choices mechanism there.

Okay.

So the way we're doing it, that is, I believe, equivalent to.

Oh, no.

What the heck is that? What's that? I am confused by that.

Right. But doing a D, that's not.

I'm sorry.

Try doing D, you know, D in the second set of brackets. No, no, in the second set of brackets. I am. It's applying that function. Right. It's using this as some sort of.

But I'm confused because the documentation says that it tries several forms.

I wonder if that. Like. Like.

It'S trying overall forms like that. Yes. Right, right.

So if you did like date or time. Right. String date or string time.

Yes. So this here is the way that we're doing. So. What's an ordered list of choices?

It's an ordinal variable like small, medium, large or bond rating or pain scale. Can you imagine a pain scale be called a choice?

Well, what do you think it should be called?

I think it should be called.

I mean, categories doesn't make a lot of sense either for that.

Well, it could be called nominal and ordinal. Nominal is unordered and ordinal is ordered.

It's kind of like. Like everything is nominal just before the spacecraft blows up.

So linear model fit uses an option called nominal variables.

Yes, which is not great. You understand what I'm saying? Nominal is. I know, I know, I know.

But. But choices is particularly bad.

Okay, fine. Well, let's come up with an alternative. Cases. If we hadn't already used it might have been. Okay.

I mean, do you understand the difference between ordinal and nominal? So nominal is just used as a tag for something. It's like a name for something.

I understand. Whereas the other has an order relation.

Yes.

And that's useful for graphics. It's useful for all kinds of purposes.

Yes. I mean, often you bracket, like age groups. Okay. They're definitely ordered. Right. This age group is older than that other age group or pain or quality or things.

What can we call these? Let's see.

So another name used in statistics are factors. That's particularly bad for us.

It's particularly bad. It's particularly awful and bad.

Right, but you've heard it and you've used it. So is age a factor in Covid?

That's not what. That's just because statisticians are thinking that way. Yeah. It's not what we want to do.

No, no, I'm. I'm not saying we should. I'm just saying that you have heard it and used it yourself and think it sounds right.

Well, I think it sounds right for.

In that context.

Okay.

And in this context, where you're picking things in a form choice sounds right. Or in a guy.

Miss things like class.

Yeah, for the nominal one, class sounds right.

The order, it could be levels because you have pain levels. Yeah, but that's for the. Well, that's for the ordinal one. Right. Ordered class. I'm not keen on classes either. Let me see. Well, I don't have a good answer for this.

So we have. Also in visualization, we've introduced categorical scale, which is a documented symbol.

Right.

But. What are all the possible names here? What is the full name? It is discrete values, basically.

Yeah.

Distinct values. Whatever it is Right.

There is. It's also assumed to be finite.

I see.

And sometimes when we say discrete, we think of integers which have additional properties.

Right.

It's a set, basically. Yeah. Versus an ordered set.

Yes.

I think further down there is a section of a previous discussion about this. Possible names. No possible names about categorical types.

Values, values list. Let's see how it goes with these other ones here. I mean, also isn't a values list, it's one of them. Go ahead.

You know, there's a sort of a difference of view with types. So all these detailed number types is very important for a compiler, but it's not super important for data modeling or symbolic transformations.

So. But.

But having number would be really useful.

Don't we have that? Yeah.

If you go out to the right, there's kind of numbers read write, feature types.

Yes. I mean, things like real we should have here as well, not just machine real. Am I making sense?

Well, I mean, what do you mean by read, write and number? Right. Because certainly binary read has. Right. Integer 8, unsigned integer 8, real 32, etc. Etc. Right.

I think that's the higher level read write function. But I agree that one of the things we need here is integer real number. Am I making sense as types? Do you agree?

Yeah.

So.

So a number type is. Is something which is mixed integer real, possibly big nums and. And has no fast. No explicit fast code for it without processing.

Yep, yep.

Okay.

Each one of these things, when you say real 16 that is a singular word, a category of thing of which you have one.

But that's not true of biter.

That's the, that's the. See, that's the. That's the type of each element of a column.

Yeah.

So in the compiler, you would. You would refer to it as numeric type, specifier of numeric array, real 16, comma 1.

Okay, but so here, this will be. What, what does this refer to when you have the choices thing, what does that refer to as in compiler, for the whole column.

Each element comes from that set.

It would be. I mean, the way you would do in the compilers, you'd have something like a hash set or a hash table with the encoding. And then you'd also have a vector, which is the encoded values. So sort of it's a bit of a compound type.

When you actually see it in a tabular, what is its type for that column?

Right now it's categorical scale, I think. And it has a list of elements.

Categorical. It's called categorical. And then it's annotated by what the actual categories are.

Well, maybe that's what we should do here as well. Categorical and ordered. Categorical.

Yeah, that's, that's, that's what's in now.

Okay, so what, what do we do here? Is it a list then?

Yeah, it's a list. It's actually a tabular column of the categories.

Okay, but you could do it as a list here.

Yes, you can do it as a list. Precisely.

There are examples of each type of type below.

We want ordinal instead of ordered. Categorical.

No, I think if we, if we go ordinal, we. We kind of should pick nominal for the other one. But if we go categorical. Ordered categorical is okay. That's the super. Or a day at the first.

I mean, that's the name of the type. I think it's sort of okay for the name of the type. What we'd come up with here. Yeah, actually we'd come up with categorical. Ordered before. Yeah. And that goes along with categorical distribution as well.

Yes. And probably a lot future functionality we're about to build.

All right. I mean, then these things will be categorical.

Is it categorical, ordered or ordered? Categorical.

Well, it's.

It's categorical. Ordered is the current name.

Okay, what do we think? I mean, it's unsigned. Integer, not integer. Unsigned.

Yeah, I think ordered. Categorical sounds better. It sounds readable.

Then I think we should go with that. We should reorder that to be that reorder that ordered. Okay, what should we look at next? We have a few more minutes and we could return to the discussion of variable names. Or is there more stuff you want to talk about here?

No, that's, Yeah, I think the discussion of variable names would be great.

Why are there two rows there? What's the difference between those two rows of quantity type?

Quantity type? I mean, just in the morning meeting, what's the suggestion? We should have a beaker or a flask for quantity.

I mean, the one down at the bottom is crazy as well. I would have thought that the quantity would be more like a ruler. I think that a measuring device, you know, measuring thing like a ruler or a calipers or something. Thing.

Right, that's what I suggested, a ruler.

What about calipers?

Yeah, doesn't sound bad to me.

Suggesting a measurement.

Could be scales. I think that's probably easier to make small.

Yes, it could be scale too. Could be a scale. Except a scale suggests that you're weighing one against the other, which suggests more like categories.

I, I, I, well, I, you know, I still remember using, you know, actual balance scales to weigh things in, in science lab. So you know that I don't have that association. No.

But no weighing in the balance, so to speak. Okay. This for an integer is absolutely freaking crazy. It's absolutely crazy because we've got, do you understand what I'm saying? We, we have. That means something to us. And it's not integer. Am I making sense?

Yep.

So.

Could it just be one and one dot.

Maybe, or N and X.

Some kind of C for this, Z bold struck V C for the set of integers.

Zr. And then we could add C, right?

Yeah, yeah. Too mathematical.

Yeah.

Well, I wonder whether we can iconographically, if we could get 1, 2, 3 in there, then that will be just fine.

I think that we had that in an earlier revision. And then 1, 2, 3, dot. And that was very hard to.

I also don't understand why byte array is a grid given that byte arrays are one dimensional.

Yeah, it's just wrong, just wrong. I think a byte array. Will just have to be a horizontal thing of a few dots there.

Yeah.

Listen, we need design people here, here. If we're going to do this and we don't have them here today.

And, and Jeremy is out. So if we need Jeremy, then. Okay, so have to wait for a couple weeks, please.

Carla is going to deal with that and we will come back to these. Yeah, right. Is there anything else that we need to go over today?

I don't think so.

Okay, and see you guys another time. Talk to you then.