Live CEOing Ep 821: Language Design in Wolfram Language [Tabular]

Okay.

Okay. Hello everyone. Welcome to another design review related to our friend Tabula intended for the next version of Wolfram language. Okay, today it looks like we are starting off with wind. Oh, oh. Oh my gosh. That is bad. Let me see.

By the way, you're not sharing anything.

I see. I see his notebook. The tabular agenda with the windowed section open.

Shared with for some reason.

Okay.

Let us know if you don't solve it. Okay, windowed.

Okay, so a little bit of introduction. So the. The original topic was missing handling and this is a kind of upstream thing. Let me introduce why. So in the last week you've been discussing the missing callback, which is the low level, element wise way of dealing with missing elements that introduce a basic level of missing handling at the transform column level where you can directly provide things. Now we have a documentation for a function called missing transform. Sorry, transform missing that basically handles missing at column level. Now, in order to do that, we need access with a certain level of granularity to a tabular column. And this is a middle ground between the functions that we've designed so far. So, I mean, this highlighted a hole in the design between transcode transform columns and reduce tabular. Reduced tabular looks at the whole column altogether. Yeah, sometimes you can group it, but it does an operation that compresses that while transform columns looks at the single row. Now, in order to impute missings on a line on a row, you need to work row by row, but you need access to the full column to do operations.

Okay, so for example, let's say that you want to replace missing values by the mean of everything in the column.

Exactly, exactly. So you could do that with what you've designed so far. You could do that by the insert column, but this is going to be very manual. But whatever we implement, the missing transformation function that we implement is going to leverage this windowed functionality. And given that it's going to leverage that, we might as well spend some time thinking about the functionality itself because this is something like element wise. It's something that we might need for some specific implementation, but is useful in.

And are you on prototype?

I don't think I'm using prototype. I need to use. Okay, hold on.

Sorry. We mentioned that.

Yeah. But the fact that Carlo is asking that question means you can probably see the screen now.

Yeah, yeah, yeah. Well, I first tried following on the live stream and then I said, well, that's gonna have a delay. I'm gonna quit and reopen Zoom.

Okay.

You can if you. There is an Initialization cell. If you evaluate that, then the examples will work. But that's still not. I mean, transform columns can't be blue on prototype.

No, exactly.

Why is it running? Like.

Are you doing something weird like he did the other time with the remote kernel?

You know what? I bet I am. I bet I am. I bet this is set to use. Yeah, because I'm doing a bunch of GPU stuff. Okay, that looks better.

Re. Enable dynamic updating though, otherwise you don't see the.

Okay, so. Yeah, those fails shouldn't be there. So let's try to evaluate. This should work without the fail. Ugly, ugly.

Can you try and reopen the notebook now that you have the correct kernel set?

Yeah, I mean, I know this is actually we worked on. So in the version Carlo had today, it was working. It might be that this is today's version. It's not what we tested.

No, there isn't.

Yeah, I think. Okay. I mean, those should be the icons, but it's not super relevant to our discussion today.

Yeah, right. Okay.

No, now we can start looking at these functions here. We have two implementations that Carlo put in here in this code. One is a mapping function and one is this head wrapper which is tailored to be used in transpose columns to do this type of operation. This is basically expanding the scope of the function the. That you have in transform.

I don't understand what this is doing at all.

There's a definition that I think it explains it.

It's a lot like supplied.

Well, it's a hell of a bad name.

Yeah. Name is temporary, of course.

Why isn't it supplied?

Do we have a stack for supplied? I'll be happy to follow that.

And also, can supply be non local? Because the interesting thing about this is that it's.

Well, yeah, supply is by definition non local. It's picking in things from the environment. Like.

Yeah, can it do grouping and sorting in place? The reason why I called it windowed is because this is not a concept that I came up with and it's called window function.

Yeah, I understand that, but it's. I mean, you know that that's a. It's like we could imagine something like this when you're using map and you're looking at, you know, you map, you think you've got a single element, but actually you've got, you know, you've got elements in the wings, so to speak as well. But that's a very.

Yeah, I mean, with. Map is a lot more general because we have. You have this ND structure and you don't know where you are. I mean, you do, you do know.

I'm not sure. Okay, so. So I'm still not entirely clear on this, what this window thing is, but I don't think this is applied, right, because supplied get. You get specific parameters based on the head, right? Like in map, you might get, you know, like the index or the part or whatever. And this is taking some sort of arbitrary things or what exactly is windowed?

Again, this is a function that is mostly applied to.

Well, if you see that in the definition windowed of func, it simply allows to take beyond the current row also the column.

I know, but why isn't that exactly what supplied is doing?

Because you have the two arguments that you can do. Window of a function group spec and window of a function group spec, sort spec, which provides you.

Hold on, hold on.

Look at an example further down. That's something you can't do with supplied because supplied doesn't know how to do group by.

Well, how do we. Why are we saying that supplied doesn't even exist yet? What it can't do is not known to it yet.

Okay, but I read the. The documentation page that exists.

Okay, well, let's look at that then. Oh, for God's sake. Why is Docu Tools not installed? That's just really weird.

I reported that to Amin Torger, but.

Look, it's completely broken. So if somebody can send Me the supplied Dr. Tools is right there on.

The right now you clicked away from Mathematica, but if you. Yeah, there it is.

Even though its palace is not visible. My God.

What do you mean? It's palette.

Everything is. I mean. Sorry, it's not its palette.

The menu item. Yeah, but it's there.

That's even more bizarre.

Okay, it's not fully installed, which is.

Why those.

Wide formatting is completely screwed up.

When are we going to get supplied implemented?

Unknown.

We really should be talking about how to get to this. I mean, okay, so this thing, Stephen.

If you look at, for example, the example of relative rank, that's something that you cannot do with that design.

Why? Picking up anything with the elements.

What?

Well, I mean, I.

I mean, I still don't see why this makes sense for supplied. Right. Supplied is taking things right. It takes, you know, certain things that make sense for that head. This is some sort of. Right, it's going in different rows. I don't even understand how that works.

But this is something also relevant for a particular head.

Anyway, let's not worry about another. Let's look at an Example with more than one argument, such as, why is.

Hash1 never being used?

Well, these are cases where hash1 is not being used, but there are cases where hash1 is used. Like.

Well, what are hash2 and hash3?

But the examples with two and three arguments are things that don't fit in the current design for supplied. And these are all examples with one. You have to scroll down further to get to examples with two or three, like relative rank, for example. So this. This tells you like, it orders by age within groups, groups of sex and class, and tells you which or rank you are.

Okay.

Within that group.

Okay. I mean, how would you do that without this? You would extract the column. You would build a new column.

You'd have to do it with group by or tabular reduce and then try to map it back. No, you could. I mean, it's complicated. To do with group by.

Yeah, I mean, yeah, no, you can do. With TableReduce, you create the groups. You compute the. You compute the rank within the group, and then you have to use some form of matching to put it back in the original tabular. Well, this is matched by default because it acts on that line.

All right, let's take a look at what this is really doing. Okay, so. My God. Grouping the tabular byte group spec. What are you doing here? You're changing the meaning of these. 1,/2,/3.

Like in tabular reduce. Oh, sorry. Like in reduced tabular is exactly the same meaning. So the hash in reduced tabular is the full column if you don't group.

But can somebody report the bug, please? This. Whatever this is N resource file. Carla, do you have it? Carla, do you have it?

Sorry, I was trying to report this with Roger. The situation you are having. Could you repeat?

Look at this.

Oh, the picture.

Okay, look at that. That came up. Please report it.

Yes, yes, I'm taking the image right now and I will do it. Yeah, I was checking with Roger. The other thing.

Okay. Oh, we don't have that.

You need. You need Docker tools.

Yeah, it's gonna look terrible, isn't it?

It's gonna look ugly.

Yeah. I'm worried that these are really incomprehensible hackeries.

Although, Steven, I think that maybe this is hard to understand, but the point is that this is really very expressive. So maybe we don't document it right now, but. But we use it internally. If you look at, like, all of the examples.

Okay, let me give you an example of something. I was sold. Okay, Okay. I was sold. Join across on the grounds that it was really important and everybody would understand it. Turns out basically nobody understands it. And this feels like the same kind of thing.

Yeah, but my point is that if you don't want to document this, fine, we'll have an internal representation. If you look at the examples of transform missing that we took from the existing documentation of transform missing, they can all be expressed with this. And so it's going to be our internal language to do all the. This kind of.

Let's take a look at it. Let's see if I can fix the stupid mistakes that are happening here. Okay. Oh, it isn't even saving. What the heck? Why is this huge? For Some reason it's 15 megabytes. Why is this showing. Okay, let's look at reduced. What were we looking at before? Reduced tabular. And what was the other thing we were looking at?

Supplied.

We really need to implement supplied. Okay, that's a separate discussion. Okay. Have people been using tabular reduce?

I have a lot.

I used it the other day.

Okay, those are good endorsements. Okay, let's try to understand this. So what really is this?

You also have the icons now.

Yes, yes, yes. What really is this? Okay, is what I'm asking. Windowed is incomprehensible. So let's try and understand what we. What we. How would we actually describe is to what extent is it like column wise?

Well, so the point.

The second argument is a column wise operation. Yeah.

Should we end up with a function column wise?

Yes, but I don't know if we're going to. I mean so far I think it's an implementation detail. I don't know if it's exposed.

No, what it is right now is that is changing the meaning of hash.

As element wise. Not documented because it's is a new function. It's what it's only in prototype.

It's. It's not that it's not a system.

Oh, I know it isn't in docket tools yet, but the.

You look in the system documentation, I.

Think no, you searched and get function.

Page and it just didn't find it.

So for a docs.

Correct.

The point of windowed is that the first argument is element wise, the second argument is column wise. The third argument is indexed.

Yeah, I guess the third argument that the function gets is like applied. The first one is like transform columns, the second one is like reduced tabular and the third one is like applied. And you have access to all of those. I guess you could.

Okay, brilliant. Some random order unnamed. So let's just understand what the first argument. So Just like if I say, where's data? Did data come in?

You need to evaluate code, it says, then you have it.

You can evaluate data.

Okay, let me just understand what transform columns is actually doing here on data. And let's say, let's say Diff age is 100 minus hash age.

Okay.

Yeah, that, that will break. Yeah. Brent.

No, no, that works.

Yeah, now I saw them.

Okay, all right, so now that's what that does. So now. Oh, this is a hackalola. What happens if I put hash age inside here? Oh, it doesn't work because.

No, it doesn't work. It does work. It does work.

It does work.

Just do, just do hash age minus that total.

Actually, if you wrap windowed around that, it's just going to be the same function.

Yeah.

You can wrap window around it and it works with saying.

Okay, so if I say hash two here.

Yeah. That's going to be the whole transpose tabular. So you have to do something about it.

You can do like total or hash 2 age, for example. That's going to be like the distance from the or mean of age. The distance from the mean.

Somebody should report the formatting bug here.

Yeah.

And another self test, error faction box, action box, line 61. Okay. Okay. All right.

So but then maybe you want to just do that instead of with the whole column by first grouping for class.

No, no, no, no, no, no. Let's get to this first. Okay. This is already a mess. Okay.

Yep.

Okay. This is really that hash2 is referring to data, isn't it?

It's the trans.

Is literally the same meaning as reduced tabular. The same meaning that hash has reduced tabular.

So if I say data of age, that isn't going to work because that. And by the way, that has got to produce a better message than that.

Yeah.

If I did that, it would presumably work.

Yep.

Yeah.

Yeah. So with a non reduced version is you could use a whole column to do the same operation.

Right. But the reduction is saying within my current group.

Exactly. What's the mean? What's the first class mean? For example.

I feel like one should first group the things and then do this. What happens? I mean, there's no way to represent in a, in a tabular right now. A grouped tabular, so to speak.

No, there isn't.

But really what has to happen is grouping has to be done dynamically and then used.

The. The cool thing about this is that you could. I mean it. It doesn't work right now, so don't try it. But if you have transform column with the list of things, then you have two different window, those are independent. And so the cool thing about that is that the order is maintained and the grouping doesn't ever really need to materialize. So you could have something like the diff age grouped by class and the diff age grouped by sex together, and they stay aligned. If you have to do the grouping first, then they resort in some sense.

Right. Him.

Okay, this is.

This is really grouping in place in some sense.

Okay, so here you're saying a second argument here. Let's see. If I group by hashtag.

One level inside windowed, and it's just the name of the column. It's not hash because it's not really. It's not getting the value.

I think that would work too.

Oh, yeah, okay. No, just put string set. Yeah, we can do the function thing, but if you put string sec, it's gonna work. It's lowercase, I think.

Lowercase S lowercase and possibly in a list and. No, it should work without a list too.

Or it could do that.

Yeah, I mean, this is not really a real implementation. It's just some code. Yeah, it's just showing what this is doing.

It's actually a really bad hack.

Okay.

All right. Okay. So shouldn't we have some visual indication of grouping?

Well, but the grouping is not contiguous in the or. You mean that the color of the.

It's gone by then. It's gone.

Let's say I did want to show it. How do I do it? Is there some way to do that at the.

I would. I would imagine a typesetting option in. In the tableau itself.

Okay, let's think about what that. Okay, I'm.

Well, I mean, we show grouped columns.

Right?

What you could do with. What you could do is like if you put in style, you put. If you put style around the function and then you could put a color based on the hash of hash two. Like the.

Oh, yeah. Is style supported in tabular right now?

I have no idea.

Yeah, me too.

What are you saying? You're saying if I put. If I say, how do I map onto this column tabular? Transform columns.

Transform columns.

Let's see if I can figure out how to do it. Age, arrow, style, hash, age, comma, pink. I wish there was a way of auto parenthesizing this. Ta da.

Okay.

Yes.

Now, I was wondering if. Do we have something like a function that takes an expression and gives each.

Yes. In the function repository we've all been rewriting, this does what? Yes, is an expression. Expression, color. Let's say that takes the hash of the expression and generates a color based.

On that, a random color based on that, but deterministically so that you could put in the second argument of style something that depends on class and color them accordingly.

Yes, that's a useful thing.

Yeah, I think we've all rewritten that is in the function repository I think can give you the name if you want to try it.

Hash color or something should it be called. We should bring that up in. In. In. I. Okay, so let's go back to this. So what we're saying is, I mean, I just hate the obscurity of all this.

I understand. I mean this is a concept and I guess we need some design work to kind of untangle it into something that is more easily accessible.

And I think that there's also like, it's very useful to have named windows things like, you know, previous next and you know, window constant with window, stuff like that, you know, as a convenience.

Oh God. I mean, but that's incredibly complicated because that's a completely different concept. I hate the name windowed. I think it's the wrong name for this. And you're extending based on the name. Basically. I just don't believe in it. I mean, I think, look, if it's sequential here and you're essentially doing a moving average or something of these numbers, I just don't think this is the right way to do it.

Yeah, but if you. The. The cool, the. The really cool stuff you do, you do when you start also using a sorting within the groups that doesn't actually reorder things.

Huh?

Yeah.

Similar way as doing in place grouping. You can doing in place sorting and then the rank that you get the hash3. It depends on that sorting. Yeah. So you can group by class and then sort by age and then start counting how many people survived going up in age.

Let me point out something, okay? We are diving into implicitly represented this, that and the other things. We never succeed with those. So let's ask the question, if we wanted to make this explicit, what would we do? In other words, let's imagine that we want to. For example, let's imagine we want to get a new tabular that's based on this whole weird thing which we then stuff back into the old tabular or something. But rather than encrustedly, I mean, this is exactly what it makes it hard to do things with dataset and so on. It gets deeply implicit and encrusted.

Yeah, but the problem with doing it explicitly is that suppose you do something like you do group by and then you do flatten in order to achieve this. Then how do you keep track of what row came from?

I don't know. But let's ask what it would take to do it explicitly.

Shuffles it.

Let's ask what it would take to do it explicitly. Okay. Because I just don't think this is going to work well. I think this is going to get very, very, very obscure. So what would it take to do it explicitly?

So you'd group by, then sort within the groups. Then you would have somehow to always return.

No, no, no, no, no, no, no, no, no, no, no, no. I'm talking about, for example, this case first. Okay, that case first. So. In this case we just need transformed columns.

In that case, you need to do an outside call to tabular reduce. Compute the minimum.

No, no, wait a second, wait a second. Here's a way that we do this. Let's see. This is tabular reduce, right? That thing there is tabular reduce. It just says tabular reduce. Why is that not working? Is it. Oh, it's reduced tabular. Is it reduced tabular? Yes.

Yeah, I think it's tabular. I think it's. I don't know, it changed like five times. I think now it's supposed to be.

Yeah, but you can't spend the docs.

Or in the building.

Okay.

Okay, what do we do here? What do we do here? We say.

Yeah.

And then we say do we do Give the column name?

Yeah, you have to give column name. You have to put hash age there.

Hash age where?

Oh, first age, before that or whatever. X doesn't matter. Goes to.

Yeah, an arrow goes to.

Goes to the function.

Don't forget parentheses because the precedence for rule.

I know, I, I would like to address and that. I mean, we need a way. You know what, when somebody has typed arrow, blah, blah, blah, amazand. Why isn't this showing? You know, we should just have a thing that brings up, you know what we should be doing? This is stupid. We should bring up an autocomplete and then this case. See what I'm saying? If you type this, it should give you an immediate auto suggest with the parenthesized version. See what I'm saying?

Unless you, unless you want.

Don't use the autocomplete.

Okay. Yeah.

Yeah. I mean, another sort of solution to this I think we talked about a while ago, but I don't know, happy with it. We talked about having a, like, I don't know what the term for this is. Like special Brackets. That would be the same thing as function. I think we talked about, like, square bracket, vertical bar, something like that.

Yes, yes. That's another possible.

I still think would be nice.

The autocomplete. I agree, but we should discuss both those things. But the autocomplete. Why isn't this doing the thing it's supposed to do? Where the ampersand. What the hell is it doing here? Where the ampersand is supposed to indicate. What is it doing? That's a bug.

What do you. What do you think it's supposed to do?

It's supposed to highlight the range of the ampersand. That's a bug. It was. It was doing that before. It's a bug. Carla, can you report that, please?

Yeah, yeah, yeah.

Also, would you please put into the next User Assistance meeting the idea of an autocomplete. Okay. In this case. Sure. I'm going to make a comment here just to make sure that's not missed.

I think it shows it when you actually apply it to something. Not in the case where it's just sitting there.

Well, it should. You know, this idea of the autocomplete is a good idea. We should have had that before.

So. So what's the idea exactly?

That we just offer an autocomplete in this case.

What is this case you're typing? And what happens?

You type this and you get an autocomplete that says X arrow. This. Simple as that. See what I'm saying?

And this appears when you type in the ampersand.

Yep.

Maybe.

Yeah. I'm just typing what this is. Okay. All right, let's come back to this. Okay. Tabular, reduce. Reduce. Tabular. Whatever. It's. It's going to be called tabular, reduce. But anyway, let's. Let's go with this. Can we. Can somebody make sure that it's getting renamed properly?

Why are we going back and forth with, like, before, they were all called column transform.

You know what? We talked about this at length. So go look at the notes from the previous time. If we're not renaming it, which is my recollection. If. If the. If the renaming thing is wrong, then bring it back again. Okay, but that's what I remember. So if I remember correctly, I remember.

At some point that there should all be verb, noun.

Okay. We had a reason for changing this one. Go back and look at the notes, please.

I don't think that's correct, but.

Okay. All right. Okay. So you think.

Yeah, I think this one is supposed to be. When I emailed Rob up this, the Other day he said he called it tabular reduce. And I think. Jose, reduce the other day.

I think it's what it's supposed to be.

Okay.

Okay. I don't understand what this did.

Well, this returns tabular.

Okay, but, but now what this wants to do is to put age in every element.

It might help not age.

Right?

Yeah.

Right. You only have a single, Right. You know, you only have a single value there. Right. So you have one group, which is what you're seeing there.

Okay, but I don't understand why this, then, with its weird thing here. Right? That thing is basically. This thing is basically in this case, the whole tabular. Right.

Transpose.

Yes. So, yeah, hash age in tabular reduce.

Right. And that mean. Of that, Right, you've taken the transpose. So now you're taking the mean of the whole column. So you're taking 29.9 minus the current row.

That, that will return.

Yeah, yeah.

That's a simple rule.

Why isn't it showing the whole. Why isn't that.

That should be wider.

Yeah, I know, but the resize.

Yeah, yeah, I think you need to resize the, the column head. The, the, the thing is reattached to the.

Okay, that should work to resize this here.

But to resize which one I think is not attached to any.

No, no, I'm, I'm saying to grab the corner.

I don't think you can make it any taller. Can you, can you grab the bottom of it? Oh, my God.

Okay, so it's just not allowing one to grab this part, which it should allow, Right? Not just the header. It should allow. Grab the vertical line anywhere.

Yeah, yeah.

Is this an edge case because you're at the bottom, right? Or like, can you try the middle? And like, one of the tabulars above doesn't work.

Okay.

I think it's just an oversight.

Yeah, I know. I, I, I, it's not, it's not 100, it's not. That is not a hundred percent obvious.

And I, Yeah, I'm pretty sure it's an oversight. We don't have anybody relevant here, so let's just not. We don't. Okay, let's go back to this. Okay, so I don't understand. Look, It's, this is just incomprehensible. I mean, I don't understand why hash age works and hash2works. How can that possibly be? What is this function being applied?

No, no, no, no. We, this is standard. This is standard Wolfram language. #1 can be abbreviated with the. That is #1 age, #2 age, right?

Yeah. If you replace the age with 1 of age, you'll get the same thing.

Okay, but. So just tell me, what is this being applied to this function?

Well, the triple of the.

Current role.

No, no, let's see if ITAI paid attention.

The data set, the transpose data set, and then the third one is a little unclear to me, but something about nearby. No, it's the index of the row.

Exactly.

Yeah. And the first one, and I have.

Not seen this function before, you know, you're fine 48 minutes ago.

So I'm just pointing out the first one is wrong. The first one is the current row, not the whole tabular. The current row.

So what's that called?

I mean, it's the same thing that transform columns get by default. In your example, input 21, you have diff age 100 minus hash age. Is that the same hash?

Right.

And just to belabor the point.

We.

Could have written that pound age as pound one of string age and would have gotten the same thing, I hope.

Yep.

That's the index of the row.

Yes.

As opposed to the column, which also has an index.

Yeah, but you typically don't care because you talk about the column by name instead of by number.

Okay, now.

Now, how does the index by itself. Maybe we looked at some example and it just ran past me. But how does the index on its own help you? Because don't you need the whole tabular if you're going to make use of the index?

No, not necessarily. You can use it as the rank. For example, if it's sorted and grouped, the index is the rank.

I see, yeah.

So this isn't to give you access to nearby values, it's to tell you positioning relative to nearby values.

Yeah.

So in that sense, it's not windowing like I'm not. Well, let me finish. Let me finish my sentence, please, Stephen. So this is not windowing in the sense of, you know, I'm making a nice spline through my points. Right. This is some other. I mean, maybe this is called windowing, but it's not what I as a mathematician would call windowing.

No, Stephen, we just not implemented the case. Only one is a hack on a down value for transform columns, you need to do it one column at a time. It doesn't support multiple columns. It's just a snippet.

Okay. There are seven other things we could imagine using here. So there'll be up to eight or something. I don't know. There's going to be other things.

I see a point here. The Issue with namings is that we sort of. That is sort of taken by the fact that the hash one is all the columns of tabular. So we need a smarter way to name things.

Let's see. So. Well, that's what supplied is doing.

It's creating an association.

Yeah, yeah. Of ambiently named things. Yeah.

I mean, I don't know if it is big or small as a nice thing, but the nice thing so far is that the original/1 of Transform columns is the same. So you don't need to change that. Meaning if we change name globally, that would be hash data. Hash row. Exactly, hash row. And then you need to put age in there, which I know maybe it's slightly more complex to do that simple thing, but it pays into simplicity. When you look at every argument together.

See what we want there. What's really confusing here is that's a column and that's an element of a row. Right. See what I'm saying?

Right.

They're both right.

The hash is a row. The hash age is an element of a row.

Right?

Yes. And the £2 of age is. Well, it's a row.

It's.

I mean it's a column in the original and a row in the. In the transpose. So I agree that's slightly confusing.

I mean, in a sense, and I'm not necessarily advocating this, but it really is incredibly like supplied.

It means supply doesn't really have this spooky action at distance.

Yes it does. It's able to get ambient things. And that's exactly what this is doing. It's getting ambient things. Okay. It's getting the ambient whole column, for example.

Not really. Like the thing about this is that it, I mean it does get other things if you use it in things like block map, tree map and so on. Whereas this one contains its own blockiness and that respect. It's different. It's like it can change the behavior of the outside map really.

I mean, it seems like the only case that's really very different from supplied is where you have groups, like non.

Trivial groups, which we haven't really understood yet.

Right, well I think then maybe we should talk about that because supplied, right now when you're asking it what to supply, you always basically just give like a string or something.

Yeah, it's local information.

Well, it's both local information, but usually what you ask.

Well, it's not, it's not actually local. It's. It's. It's supplied is. Is. Is sort of global information at the point. Right. The index is not local information.

Right, but it didn't reconstitute what it. What this is doing is. It is.

I mean, I think the more. The more important difference is just that supplied. Right. Now, you can't ask it like none of these. None of these, you know, supplied elements are functions. They're always just strings that you're asking for.

Yeah. Which may not be right.

Maybe.

But it seems like in the grouped window case, then we're going to have complicated grouping functions and stuff.

Okay, let's understand these arguments because, I mean, basically what we're saying is you're trying to reach out to some kind of more global piece. Okay. Either, you know, what are some things you want to reach out to? The current group. Of rows. No, the current. What is it? It's the. It's the. The current collection of values in a particular column. In a particular column in a group defined by another column. Correct. So we group by sex and we then are looking at a collection of values of age within that. Within the group. Yeah, we're segregating age into two groups and we're looking at the collection of values in the current group.

Yeah.

As opposed to in the general hash, you know, hash two case. We're looking at it in the whole column.

Yep.

Right.

Which you can think of as if you're just grouping by some constant or something.

But. Yeah, yeah.

Empty list.

Yes. Okay. Now the third argument proposed is what? Oh, the row index. But what about the. No, I'm sorry, the third argument to window, the sort.

Sort spec. Yeah.

What the heck is that?

Like you. You group by sex and class and then you sort by age.

So what? Sort what? By age.

So, for example, one example that Julio wrote further down, like sort, within the subgroup, there is one with the survival cumulative survival fraction. Like this one sorts by age groups by sex and class and then tells you how many people have survived so far within that group.

Okay, but, you know, so, so, so.

Well, can we. Can we, like, copy out that window then see the three arguments on their own? Because I can't even tell what. What the three arguments are.

Second argument is sex and class, which is the definition of the group. Third argument is age.

Wait, is it supposed to be sorting by age or so.

Yeah. When does it do the sort?

What?

When does do the sort of?

Same as the grouping just before providing it to the. To the function.

I see. So this is changing the value of £3.

Also, the whole £2 is different because the column is resorted.

I see.

Okay. To my mind, #2 should be a Mini tabular of some kind that can have operations done on it before it's passed in. Because that's what you're really doing here. These things here. Maybe I'm wrong.

You.

You can't do it after the.

Okay, okay, so. So you do. You sort. Okay, maybe this is a stupid question, but you're sorting before you transpose or.

You do.

Right. I'm trying to understand. You said it changes £2, so I'm trying to understand how it changes.

No, just because £2 don't think about an explicit transposition is that you take the tabular, you create the groups. So imagine that you are just collecting groups of rows separately and then you have like, I don't know, like four or five subtabulars that are like independent tabulars. Then you do sort by. Then you do sort by age. So each of these subtabular is sorted by age and those are the subtabulars. And then you can look at the index and you can pass the column.

It's sorted by the. The order sorted by the original row index.

Yeah, exactly. So if you don't group, it gets the row index. If you do group, it gets the shifted and rescaled row index within the subgroup, but is not.

Oh, right, right. And it doesn't really. So why do you say it changes the £2? Because if. Right. You know, if you're looking at the whole transposed age. Right. It doesn't matter that you've sorted.

Well, yeah, because. Yeah, because I mean, first don't think about the single column. £2 is a full tabular. And imagine that you have, you know, only the people in first class. Then £2 is the list of columns, but you are shuffling the rows around before transposing it.

But you've.

Because you are sorting by age.

But if you've split it into groups. Right, So I certainly understand that the grouping affects what £2 of whatever is. But how does then sorting each group affect what £2 of whatever.

Look at that.

Take two survived.

Ash three.

If it wasn't sorted, that take would be meaningless because the order within of the survive doesn't matter. You're taking all the ones that are younger than the current one.

I see. Okay. Yeah. So pound two. Okay, so pound two is now a sorted column. And then you're. Right, I was still thinking of like, mean, where it doesn't matter. But yes, you're taking. Yes, yes, yes. Okay. Yes. Okay, thank you. I. I am unconfused now.

Right, so. If we had a thing like windowed Which I am not buying yet. What should we call it? What is it? What is it really? What functions would it be used in? Is this more than just transformed columns?

It can be used in Map, as a matter of fact.

What are you talking about? Map?

Yep.

Oh, and then we're talking about something completely different.

I mean, my original plantation.

Well, not a chance. What do you mean? What do you mean? Are you talking about. We're saying map of something? What are you talking about?

The first version of this that I wrote is called window map, and it does map plus extra things because it can group without grouping. Right. So if you type window map, it will work.

But that sounds like. That sounds like a head, not like. Yes.

It won't work on something one dimensional, obviously, because it tries to transpose things.

So it's a total hack. I mean, it's not anything like Map. It's something completely different.

No, I don't think. Yeah, Carlo, I don't think that it can work like Map because this relies on the 2D structure of tabular, while map works on generic ND, not even ND, it works on rank.

Yeah, no, it's. Yeah, you're right about that, of course.

Okay, so it's nothing to do with map, it's something specific. So explain to me what functions it works in so far.

It only works in transform columns and construct columns.

Really?

I'm sorry, and what, unconstruct columns?

Yeah.

Okay. Can we imagine it working in anything else? Because if it works in those two things, there's probably a different design that's specific to those functions. Right, because what we're seeing here, what we're asking for is that in the construction of a column, which normally simply gets access to a single row, right. In construct columns here, we're, you know, we're normally, we're saying, You know, well, we could.

We could provide those extra arguments automatically without any windowed thing, and that would be the window without groupings.

No, we can't. Unless we have names for them. Unless we have reserved word names for them. Right. And that's not going to work because hash2 needs the ability to index. I mean, it could work, I suppose. I mean, you. We could have a sort of, you know, a hash whole tabular type thing, which is hash2.

While it's transposed tabular.

I know, Yeah.

I mean, #2 is really columns. Hash columns. And then you get columns, survived columns. It's a list of columns.

Right. And this is row index. #3 is row index.

Yeah.

So an alternative in this one here, for example, would be.

And I will point out that columns and row index are unlikely to be column names. So this may be an acceptable bit of potential clashy bits.

If only we could start variables with underscores. Aren't we redesigning variable names?

Yes, we are.

Can we start a variable with $. I mean we can use it inside variables. You mean something that sounds a bit.

You know, something like dollar installation directory and dollar home directory.

Okay, the thing we don't have here. So we have this, but we don't yet have then the grouping stuff. So this on its own looks quite reasonable to me. See what I'm saying? That looks comprehensible. Now the issue is doesn't transform columns. You're saying that on a per column basis we might have different grouping and so on. But what does transform columns have in. In terms of other arguments.

Can I make a proposal?

Yeah.

Instead of. So if you're going with these names, then maybe the thing we could do is to have a headdress for the function that is called. I mean we could call it something like. In SQL there is something called partition by that would then just take the grouping and the sorting and you put it as a headdress for the function.

Okay. So that would look like here a headdress for this one.

No, no, no, no, no. For the whole thing. The whole thing?

Nah.

Like partition by.

Well, I mean supply. Right, Supply. There's a headdress for the whole function. Right. So this would be.

So this whole function here. Yes, yes.

Type partition by and then you know, sex, class, whatever, age and then close.

Function and partitioning by that. It's a headdress like supplied as a headdress for the function.

Then call it partitioned by.

It's not. Right? It's. It's not the same thing. I mean, look, if we say reverse supplied, e.g. f x y. Okay, we get that.

Yeah.

So what's the analogous thing here? How is it affecting the function?

Are you asking? This is. So the point is that this thing is. Is. Is non local.

So you.

It's not affecting the function really. It's always going to be allowed.

It shouldn't be a headdress like this unless it's supplied, which is the headdress that deals with ambient stuff.

Except I think it's the wrong. But what supply does tell is, you know, right. It b. Binds the external information to your local variables. Whereas this is not doing anything like that. It's saying you know how to pre process the. The. The tabular before Feeding it to this function. Right.

We call it applied to partitions or something like that. I don't know. Because that's what it does.

Really. Well, the first hash is not really affected by this at all. Yeah, so it's like hash columns and hash row index. I mean, just making a consideration. I don't know if it will have an impact on this, but it's affecting those things.

Are you still with us, Stephen?

I'm still with her. With you. I'm just dealing with something else. Sorry, I'm back here. Okay. What were you saying? I mean, do you have a different solution here? Because what this is doing, I mean, this seems understandable and extensible. It's not perfect in some ways.

I mean, you, you are making it impossible to name a column, row, index or columns.

Yeah, I'm making it difficult. What would you like me to do to make it even more difficult? I mean, it's a trade off.

Yeah. Well, what happens if you import like something coming from a CSV, the unwashed web, it has that as the name. Do we issue a message and say, man, this is going to be.

Yeah, probably. But I mean, I think we need to figure out whether, whether it's going to be a problem to rename that column. Right. If we say rename columns, which we have, right. Data, you know, row index goes to blah, blah, blah. That's probably not a problem.

Okay, but then do we, how do we do the grouping? Do you want a headdress?

I don't know. I don't know. We need to figure that out because.

I, I don't understand. If you have a problem with the fact that it's a headdress or if we can't find a good name for that.

Just a second. I don't think either of those is right. I think that what we're dealing with here is we probably need something called something like grouped columns. Or something.

But you're not grouping. Well, that's way too confusing, right? Because to me that sounds like you're grouping columns, whereas you're really grouping rows.

Also, you're. What if you had something. Hold on, hold on. I just don't think this is the right idea. I don't think having this invisible, weird grouping. Look, if you can get an explicit object that represents the grouped column, okay, then index into it. What is the explicit thing that groups the column? See, this is, this is. What you want.

I mean, do you mean like, I mean in some of these data frame things or like, you know, they'll have a They'll have like a tabular and a separate thing that'll be like a grouped tabular, which is sort of like a list of tabulars in some way. But these are usually really weird objects. So far we've avoided having ever showing one of those things.

Yeah, and also it's. If you use one of those, there's no way you can glue it back without finishing the group because it's like just partially grouped. And so there's no way to glue it back to a non group tabular because it will have different number of rows. It's weird.

Let's talk about what grouping really is. Okay, so. So here. It's this weird non local thing.

Well, yeah, I mean, I think in part it's.

It's.

Okay, so the problem is, right, tabular is all about having essentially like matrix, like data. And I think when you're grouping effectively, you have to briefly go into a world where you're depth three and then do an aggregation and then get back to depth two.

Yeah. And because most people are unable to deal with Depth 3 because their systems are only built for Depth 2, a giant hack tower has been erected that makes it seem like it's depth two, which probably isn't.

Right?

Well, sort of. I mean, right now what we're doing is we're, we're avoiding ever showing a depth three thing. What most systems do is they have, they have their depth two tabular and then they have a depth three object, a separate object that's called like a grouped tabular, which supports some of the things that tabular does and is generally kind of weird and so on. But I think there are kind of like algorithmic reasons why you can't just have it be like a list of tabulars, because, for instance, you know, you want it so that even though it's grouped, they're all stored in the same packed form and stuff like that.

Okay, but what is it that one is picking up? One doesn't want hash columns of age. Instead one is picking up the group. Right, the current group. Right. This here, length of this is one thing. Right, but that's not what we get. We get length of the current group, Right?

Yeah, exactly.

It's essentially that. Not a very nice name, but it's essentially that. Right?

Yeah.

Now the grouping, is it global in a transform columns or is it local to each individual column that's being created?

So far it's local.

And the idea is that you could actually add two new columns, which is the average survival rate per class. And the average survival rate per sect.

What will be. Okay, let's see. If we had a function like Windows, what should it actually be called? What is it doing? What is its definition?

Let me. Let me put it in a different way. What if instead of having this spooky action at a distance, instead of like we modify transform columns itself to do this kind of stuff. That way it's global. It's not as nice. You have to fold them if you want to do multiple of them. But then you don't have spooky action at a distance.

It's possible. Yeah. I mean, in other words, that it does the grouping.

Yeah. It would have something like windowed transform columns and then you have extra arguments in transform, window, transform.

This windowed idea. Windowed idea is a bad idea. It's just not a good term.

It's just.

Okay, just the way I learned.

It's not a good term. It's. It means something. We could imagine something like that for map, which means do something like moving map.

Are these additional arguments or options? Are you suggesting.

No additional arguments. Probably a different head too.

I'm not sure why we need a different head.

Maybe we don't. Sorry. For which. The transform columns. Yeah, right.

I mean if we. If we just add the. These as, you know, additional argument. Right. Transform columns is currently single argument. Well, two arguments.

So wouldn't.

I guess wouldn't break the operator form to add the arguments.

Well, okay, so. But what would windowed be called if it was not called windowed? What is its real name, so to speak?

Feel like it's super reducer. Right?

It's. Yeah.

Right. It's allow allowing you to slice and dice the. The data in more ways.

No, but that's not.

Yeah, but it never really reduces. That's the.

It's providing additional material. Yeah, it's not just.

It's not. It's not just provide. Right, but it's not just providing. It's not just doing that. Right, because. Right. It's grouping and sorting. So it's.

I know, but. But that's a different functionality. I mean it's base functionality. If we ignore that.

Right, well. Well, if. If we ignore all that. Right, then that's just adding, you know, pound row, index and pound columns as. You know, as.

I'm confused by the following thing. Why wouldn't you need the whole column not grouped some of the time? See what I'm saying? You could need both in a single operation. You could need the thing grouped by something. You could need it grouped by something else. As well, you know, it makes more sense to me. This isn't quite right, Sort of a headdress here like that. Because really you want it local. You don't want to see what I'm saying. You might want global columns, columns of age or whatever, right? Why would you not want that? And why wouldn't you want it grouped in two different ways? Right?

What do you mean by grouped two different ways?

Well, grouped with respect to one parameter, group with respect to another parameter. I want to know all the orange orangutans divided by the number of baby orangutans or something. I want to know both. Some quantity that I want here is the, you know, is the color value of the orangutan and the other is its age. And I want to group when I generate this column. One parameter I want to divide by the, you know, I don't know the. The total number of orangutans who are three years old and the other grouping I want to be by the shade of orange that the orangutang is. Do you see what I'm saying? I don't see any reason to think that you only want one grouping collector in. Within any given column construction. You see what I'm saying?

I see what you're saying. I.

In other words, the grouping really is saying what I want is this. What I want to do is I want to pick up this within the current group, because I'm doing it row by row. You know, this is kind of a headdress that ultimately means within data on this row that I have currently reached in the construction of a column within this row. I want to pick up not hash columns of age, which is the completely global column, but that column. Grouped by this parameter. But within, within the current. Within the. See what I'm saying? So in other words, this row, you know, this row has a certain value of parameter. Parameter P. Okay, I want all elements of column C that have the same value of parameter P as this row, which is.

Okay, fine. What I'm wondering here is that I think that in 90% of examples, within group is going to be used exactly the same as what the thing I proposed with windowed. It's just inside the ampersand.

I don't believe it. I think this is what you want. I think this is what you want. I want the elements. Okay, So I might want all elements of column C. Okay. Or.

Which is what it does.

No, it isn't. It isn't. It isn't. It does not provide this capability. Does not provide that capability. It does only if you have multiple windows. I don't even see how that works.

I don't understand.

Like what. I know I'm with Carlo on this one. I mean, that sounds like exactly what his window does.

How does it do it?

I mean, what I'm saying is that ignite. I'm not saying that it, it can currently.

No, you're saying it can't do this, right?

It can't do this, it does that. It just can't do more than one within.

Okay, okay, okay, yes, I understand. But that's what I'm saying is you. Why wouldn't you want more than one? I don't understand why you wouldn't want.

Because it's easy. Because it's exotic and it's easy.

Not exotic, it's. It's only exotic if you don't have a good notation for it.

It's the same notation. It's just levels down. Exact same notation.

No, how do you do it?

What do you mean? It's the same notation. It's the same argument structure.

For what?

Windowed. It's the second argument, sex. Right?

No, that's only giving you a single parameter. A single grouping parameter for everything. Yeah, this is not doing that. This is saying you can get the, the kind of, you know, common group. Okay, you're referencing a random column in the tabular, okay? But you're referencing the, the elements of that column that are co grouped with you with respect to some parameter. Do you see what I'm saying?

Which is what the thing is, you're doing it locally instead of globally. But most of the times what you want to do is put within group, just immediately one level below function, because you want everything co grouped together. Otherwise you don't understand. And if you do that, you'll see that within group is the same thing as windowed, just within the ampersand instead of without the ampersand.

Look, I think this is the beginning of a reasonable approach because I think what we want is. You're getting a blob, okay? Along. In, in this, you've usually been applying things row wise. Do you understand what I'm saying? But here you want a more global blob, and there are various global blobs that you might usefully want.

Yeah, but not at the same time.

Usually you say I don't agree with you.

And also you can do it like you can compute some parts. Okay, fine. Let's say it's super useful to have within group in different places doing different things. Don't you agree that when you do want the same thing globally, it's exactly the same syntax that you disliked half an hour ago.

Possibly, but it's more comprehensible here. I don't really like this syntax either, but I think what's important is that. And again, maybe we combine these constructs and we have a function that is basically, you know, we have something in here that is kind of like a slot, right? Like hash age is otherwise known as slot of age, right? Yep. Okay, but this isn't a slot. This is something else. This is some kind of tabular slot or something. I don't really like that term, but it's a, it's a column slot or something like that. And we have a tabular slot which, which then we get rid of. We don't have to deal with this, you know, reserved words thing. It's like tabular slots. So then this becomes. Again, I don't really like the syntax, but you could imagine, you know, tabular slot of age grouped by this.

What is that?

That means a tabular slot or, or maybe I should call it a column slot. Just like hash age refers to the local, you know, the, the row as if association. See what I'm saying? So hash age refers to the thing on that current row. Okay. Column slot would refer to. Would be. A way of selecting from the whole tabular. I, I don't know. I mean then.

And then you'd have another. Which is index slot.

Maybe. I'm not sure.

The index slot cannot take like would. The first argument doesn't make sense.

No, I understand. I mean, I don't particularly like that. But I think something, you know, I, I claim they're going to be more of these things. Things. So we might want a, you know, a tabular slot concept that has multiple things. Okay, so what this. Then the documentation for this would say. Again, I don't really like the term slot. I mean that, that's not quite right. But it would just say this puts into that place. The whole column. But it could be the whole column grouped along with this row. So it's a context dependent thing. Because this is different given that it's got grouping. It's different because it needs to have things that are co. Grouped with that row. Am I making sense? I think I'm supposed to go to a different meeting. But. But we're making progress here. This is, this is getting a bit more explicit and not as much, you know, hash two means, you know, telling joke by the number 85 is the following joke about the chicken. I'M being very unreasonable. But. But you get what I'm saying.

Yeah.

Having random hash twos and hash threes lying around just doesn't feel nice. All right, we're making progress. Let's pick it up. Can. Can we. Can you guys think a little bit about the idea of a symbolic construction that lives within transform columns and that pulls out these. These objects? See what I'm saying? I think we're going to get somewhere with that because it really, you know, it satisfies the criterion that it remains local to what's going.

On.

Yeah. Again, the thing I'm not a huge fan of here is that, you know, if you look at many of the examples we looked at for implementing things like for transform missing that are in the. Below the fold.

I understand. I gotta go.

You have to repeat sex, age, blah, blah, blah, 15 times. If you type it like this.

Maybe. Let's see. Let's start with this. And maybe there's a way to move it outwards. Okay. All right. See you guys.