Live CEOing Ep 823: Language Design in Wolfram Language [Tabular]

Hello, everyone. Welcome to another Wolfram Language Design Review. Talking about our friend tabula. So, what are we looking at here?

So, on Monday, you floated around some possible syntaxes for window functions. I think the last name we were floating was with groupings. So I took an example, a reasonably complex example, and I remade it in all the different syntaxes so that we can look at it and decide which one looks best.

Okay, so let's remind ourselves what this is. This is something where we are indexing into. Normally transform columns, has rows presented to it as if associations, so to speak. Correct. Normally what? Transform columns? The functions and transform columns normally take as their arguments individual rows presented as, you know, COL1, arrow, VAL1, et cetera, et cetera, et cetera. That's effectively what it has. So that's hash sign in the current design. Right. Am I right?

Yes.

Yes.

Okay. Now, the goal is to have additional things be available, such as, basically, transpose of the whole tabular. Yes. Is that correct?

Or. Or.

Or subparts of it, depending on whether you're grouping or not.

Okay. Okay. Okay, okay, okay. Okay. So this is. Okay. Of, you know, This is part of tabular, within the current group of rows. Is that correct? Yes, within the rows. Current group of rows with some specified grouping, which is an important point, that it's a specified grouping.

Yes.

Can I make a meta point about all of this? I think that if we are not 100% happy with any of these syntaxes, I think it's also fine if we punt on this for version one and we. I mean, we will use it internally to implement replace missings and whatnot, but for exposing it to end users, we can also live without. It's like not, I think, like the number one feature that we want to have.

I understand. Let's make one attempt for another few minutes here. If we don't succeed, let's punt it. Okay.

Yep.

Okay, so the final thing is the row index. Well, there's the global row index, which we might want to have, or there's the row index within some specified grouping. Yes. All right. So I'm going to make the following statement, which may be wrong, but I feel like these additional things are kind of like this is a function that you can get into that could be called, you know, hash row. In other words, what I'm thinking is you're either just given the row. Okay. Or you're given an addition, something more like the supplied case, where that's hash row and you're saying hash row of x1 or whatever else it is. Or you're given hash. Well, again, I don't even know how to do this. You know the thing with groupings, right? Am I making sense? Because this isn't a pure hash thing. Because it's got to specify what the groupings are. Do you understand what I'm saying? There could be different groupings for different, for different elements in the row.

I think it's fairly rare that you want.

I don't care. I don't care. I don't believe you. I just don't believe you. I, I think we need to support that case.

So you mean the group for, for, for age will be, you know, all the even numbered ages in the group for, you know. Wait, actually, does that make sense? Because normally the thing you want to do is you want to say I want to group people by, you know, age or something like that. And then I want to take the average of the heights for within that age group. Right? In which case you need to have the, the same groupings for all the different columns.

Yes, but what if you want, you.

Know, I understand the concept of, you know, we want to group by age and sex. So, you know, right, we have the males 25 to 34 and you know, the females the 35 to 49 or whatever. But, and you know, we can have an index within each of those two groups. But what does it mean to have two indices? I, I just don't understand what it is you're imagining.

Even I'm imagining that this row is, you know, female middle aged people, let's say. Sorry, this, this row. I want some of the elements. I want the ranking of. Look, I, I want the ranking of parameter x1 within females. I want the ranking of parameter x2 within middle aged people. See what I'm saying?

I don't know what you're talking about. Rankings.

I mean, yeah, the indices.

The index is a property of the row you're in, right? It's the index of the row.

I know.

So but if you're looking at one row, then I'm looking at one row.

And the whole point is that that row has one index position with respect to females and one index position with respect to middle aged people. Those are different.

But why not just have the, I mean, why not just have, you know, group by two indices at the same time? Like saying we're grouping by the tuple of age and sex or something.

What Stephen is saying is that say you rank by age and you separately group by sex and by class. And you want to say that some given person on the Titanic was the third youngest male and the fifth youngest first class, regardless of gender.

Yeah.

Which is.

And I.

If you scroll down and look at the. Like, one of the examples I made uses that syntax, which is the one with tabular's lot. And the point is that while that thing might be sometimes, but I think it's rare, it's like the one that looks the worst in a normal case. The fact that there's no way of grouping a subpart of the expression and say this is all grouped by the same thing means that you have to repeat sex, class and age everywhere in each slot. It becomes super cumbersome to do the normal stuff.

Right. And. Right. So, so, right. So really then there are three indices, right? Well, at least three indices, maybe more. Right. Your, your, your. Your race and a. Your index in age, your index in. In. In sex, and then your index in your subgroup of race and sex.

Well, and, and, and, and given that age is like a continuous variable and you could decide to discretize it and put any function of all of the things in the row, there is like an infinite number of possible functions that you can use to group and to sort. Because like those things that now are strings could be functions themselves.

Okay.

Like you want to group by decades of age.

Okay, listen, with round. Let's punt. Okay, let's punt for now. We've got other things to deal with.

Okay, sounds good.

I just think I. Let's make sure we remember this. It's something we're going to come back to.

Yep. And I think that, I mean, Leonid is on a holiday now. He told me that he had some ideas about some kind of virtually grouped tabular that is not really good, but it's something. And you mentioned something like that the other time. So, you know, maybe let's talk about it in October once this is released and with him as well. Maybe he'll have a nicer design than this.

Okay.

Yeah.

What should we be talking about today then?

Let's talk about. I mean, I think this was a good introduction to replace missing. So if you can bring up the. The documentation page of that.

Which one?

It's called Missing Transform. Sorry, the name was changed.

Okay. I just want to clarify that. We've talked about visualization stuff.

It's in cvs.

That's where I went to.

Huh? No, sorry, I'm stupid. Transform. Missing like transform columns. My bad.

Okay, I just want to clarify in the, in the overall roadmap and. Roger, you're here, right? Roger, are you here?

Yes, yes.

Okay. In the overall roadmap, because we're, you know, we need to make sure we release this on time and the overall roadmap. I believe there is also stuff to talk about with modeling. Is that correct?

Yes, we're a little bit behind on that. I also don't think it's as critical for a first.

First release. Okay. But we'll need time series stuff and things like this. Right.

Well, we'll, I mean, we'll embed time series in tabular and we'll be able to extract them. You can do a lot of operations on, on that. You know, effectively. If you have a date column or a time column, you can, you know, do that. Are you there?

Okay.

Yeah, I'm here. By the way, I really don't like these.

Yeah, I, I know. Just, I was about to say just, just a comment that I updated that illustrations like three days ago, so.

But I don't know why.

No, no, I don't know why you don't have it. Just because the comment was that this is not great, but it's just illustration, so it's fine. You can ignore it and go ahead.

Yeah, I mean, that's a, that's a prototype illustration. We'll do uniform style of illustrations all the way throughout. And that will be a feature. It will make it hopefully quite easy to understand a function without reading much text.

Good. I can't update my app.

It's fine. It was just illustration, so we can ignore it and just look at the spec.

Okay.

What is this function supposed to be called now?

No, no, this. It was called replace missing. And you propose transform missing to make it in line with transform columns.

But it's going to work in things other than tabular. Yes.

You see this possible form of tab include association metrics, data set tabular we can possibly support. Also.

Don'T put, don't put this tabular data there.

Don't we use it almost everywhere to signify like data.

I mean, it's not a capital tabular, so I'm not sure what.

Yeah, it's not, it's not capital. Well, they held with the. At least. Yes. In a data set.

Yeah. I mean, come on, get it right, Steven. Let's just look at design. This was repurposed from other pages several times. But before we do the details of the naming, I would like to focus on like.

So what would you like me to look at?

Okay, so the point is you have transform missing, let's say a tabular. And then the point is this thing by default likes to act column wise. Because each column typically have its own way, which is data dependent on imputing the missing data. So what was I.

Where are the examples?

They were in some old agenda. Sorry, I think this messed up. But it's all the same. Transform missing of tabula. And then you get class and you get for example, the next non missing or the first non missing.

Okay, so what we're talking about transform missing of a tabular. Now this is like transform columns.

Yeah, but it needs to see columns like the input arguments are more like tabular reduce. Because this needs to see the full column, not the row typically.

I understand, I understand. This is saying column name arrow. And then for example, I could say interpolate here. Is that right?

Exactly. Or mode or nearest or zero or.

Right. Okay. What about a function? An arbitrary function that gets the whole column.

Yeah. If you scroll up there. I put it as a tentative design, like futureize Julio F of your function. We need to decide.

But you can't have a string Q value because you're using strings for these things.

Well, there's a. An escape mechanism which is like this constant comma value. If you want to use interpolate.

No, no, no, let's not do that. Let's not do that. Let's either figure out a way to distinguish. You see what I'm saying? Because having a. How common. Wait, wait a second. How column will it. How common will it be to want a string for missing? It might be that you want a null string, for example.

Yeah, I. I mean, I thought it was. Was a pretty common case in the end that you might want to strain.

Okay, well, so. So fine. So let's think about the design. Because I don't particularly like these. These things which go. You know, which are these list like things that have random tentacles in them.

Verbatim as a head, which is for little things.

Just. Just a minute, just a minute, just a minute. What. What we have are a certain number of indicators. Okay. If I simply have a. Okay, and now I want to get rid of the missing in that list. How would I do that? No, no, no, I didn't mean that. I meant there's missing.

What do you mean get rid of. You mean.

I mean. I mean replace it with something. Do what this is doing. Right? Transform it.

Right.

It depends on what you're going to do.

Replace all.

No, I. I know all those things. I. I'm asking a different question. I'm. I'm saying in a future world in which all these operations are useful. See, what I'm Saying so we have right now synthesized missing values. Right. Which is an example of this, isn't it? I could say synthesize missing values of this thing here. Right?

You should be able to. This is a. And the reason why we're proposing something different is this is probably, probably over engineered. This is training a learned distribution on the data and, and using that to impute to a.

Clearly something we want to be able to do in. You know, why not be able to do that in. In tabulars?

Yes.

Yeah, right.

I mean that's like one of many there. There are a lot of much simpler things that you also often want to do.

I know. So what I'm asking is, what I'm.

Saying is so random sample is that.

Random sample from that list? This is not random sampling. This is generating the closest thing without distribution would be an empirical distribution with those numbers.

It's not in the random sample is actually like. No, no, but random sample. It should arguably should be made maybe random choice. But it would be something like we put seven there because seven.

Yeah, yeah, exactly. Yeah, it should be random choice. Yeah, Carlo noted that to me the update. I forgot to update it.

Okay, so hold on, hold on, hold on.

Julian, why we talked about this before. To have synthesized missing value as one of them, you know, which is distribution based. So it's a random sample.

Yes, I understand. Guys, guys, we're dealing with a much straight more straightforward syntactic issue. Okay. Point I'm making is any one of these things might be useful to just run on a list.

Yeah.

So synthesized missing values maybe could support all those or something.

Maybe or something. But I mean, I'm just thinking about, about let's. Let's look at synthesized missing values.

But I'm not.

Wait, wait, wait, Steven. I think that the more useful ones in replacement things are also going to look at the other things in the rope. So like I don't know if this column is missing. Look at this other column or.

Okay, well how are we going to specify that? Because what we need there the function. If there's a function F and transform missing. Okay. Right now transform missing. Doesn't that function F gets only the column?

Yeah, well, no, not.

Yes, yes, that's what you expect, right?

I mean arguably. Yeah, I mean arguably it really needs to be able to see the entire tabular. Like in general, that's what I think. Yeah, but in practice you won't do it that way. In principle needs to see the whole tabular and maybe that's what we should pass it. But in Practice. If you're actually doing something that's going to be looking at the entire tabular before synthesizing a value, you're probably just going to want to, for example, learn a distribution separately and then use that in your function.

That's why we looked at, at window functions before, because.

Let'S not get confused.

Like, one thing that is super common would be that instead of saying mode, you say mode, but first group by this so that you would take the mode.

Look, you know what, those are just functions that you run. The fact that we're trying to back, you know, turn this into a back end to this, like adding lots of little buttons you can push, is a mistake, right? There's a. Yes, you know, look, the thing we've learned over all these years and which we keep on making a mistake on, we made a mistake with it, with this, with databases, right? Don't have implicit things. If you want a group first and then do something, group first, return a tabular that has a grouping in it, then do the operation. Don't try and do it as an implicit virtual thing. That's what you do. If you don't have a symbolic system and you have no way to represent the intermediate state. You see what I'm saying?

I see what you're saying, but I also think that this stuff that we've been looking at is stuff that doesn't really do the grouping, it does it in place. It's really hard to represent as a separate step and then merge it back together.

I think it's not right. I think we should have a symbolic representation of the thing and then do something with it, not try and virtually do it implicitly. Every time we've done that, the things we build are unusable and fail. So let's not repeat that mistake.

So going back to the non windowed design, I think one concept that was ruled out by Rob that came actually looking at this function was that concept of column wise that was used basically to swap the meaning of hash in transform columns. So if you do column wise, and I don't remember how he designed it, it probably was column wise F of hash bar that would apply on the column bar, not on the element of the row bar.

Okay, so hold on, let's understand that. So let's say transform columns, tabular A goes to. So that's a column. A is a column. Right? But now hash I is a, is an index inside is a column index inside a row.

Exactly. So it's a set.

But now we could imagine something maybe.

Would be like kind of column wise, mean of I, for example, because you need to reduce it to a single element yet if you want to combine it.

Right, right, right, right, right, right. So then here. But it's not I, because I has already lost its information about that.

I'm trying to find the notebook by. Yeah, I think that what Rob had was simply column wise of mean, and that was basically assigning a special meaning to the slot. Maybe you don't like that as a concept.

Look, look, the problem is you have to be able to specify here. I like the idea of column wise. Okay. This represents. Okay. What this thing is being given is effectively an association. That is the row. Right? Yes. But now, as we talked about just now, Sometimes you want to access something. I mean, it's just the same story as supplied. You're trying to access something that is outside of the particular context that you're in, but uses its environment. Right.

Well, in this case, the environment is just tab.

Yeah.

Because it's not related to the. Well, but.

Yeah, okay, but I mean, it's. It's. It's not just tab because there's a. There's more detail to the environment. It could be the whole column I, for example. Right. Let's remind.

In general, it's tab. I mean, like, like might have to slice. May. Maybe we can reduce it to just giving a slice of tap, but in general, you need the entire.

Yeah, I agree. Remind me how this works for map. For example, let's look at supplied. Okay. So for example, we say supplied index of F.

Of s.

Oh, no, that's. I see. Okay. And then this is xyz. And then that would give us f of x1, f of x2, etc. What's that third usage there? Supplied of. What's this usage here?

Right, It's. I mean, I, I'm not super keen on it, but it was an idea and it's, you know, given that none of this is implemented, but. Right. It's an idea that like, it infers the thing. Right. You just give it, you know, pound index, ampersand, and it infer, you know, digs into it, figures out what keys you have and.

Right.

Then you'd get f of 1, f of 2, f of 3.

That's like that. Okay, all right, so. But this is very much the same kind of thing as what we need here, isn't it?

And I don't think so. And that's why I brought up window functions before.

Yeah, I think you're wrong. Look at, look at what this is doing. It's taking something from the environment and it's supplying it.

Yeah, but it's not, it's not useful. I think like if you scroll up at the agenda from Monday, we had a number of examples implemented with window functions of things that are very common.

Like I don't care. I mean, why isn't this the way to do it? Why isn't this the way to do it? It's something which is coming from the element being fed to the function.

Okay, I think that's what you need. Okay. I think one common thing you want to do is to say I want to take all the items that have the same characteristics over some column and take. Replace the missings in that group with the mean or the median or the mode from that group. How do you do that with supplied? You don't.

Okay, let's look at.

Okay, let's try to do with this. Supplied you have a hash I and then you have supplied of mean and then you have something like reverse tabula. And then you do I. Oh, sorry, you do hash reverse tabular applied to I string I. I mean you need to supply something to the function. Yeah, this thing, what you supply is a transpose tabular. So that would create an association, I assume if I read supply correctly with the key transpose tabular and then you need to take the I column of that like so it's going to be hash transform tabular applied to string. I.

Think so. But let's come back to this again. Okay, what we're trying to do here is. Okay, normally transform columns gets only row local information. Right. You understand what I mean by that?

Yes, that's what it gets.

Right. Okay, now as you point out, transform missing regularly needs more than row local information. Right. Am I making sense?

Yeah. And keep in mind that you're looking at the single column. Then there's all extra design of. For example, I know to the nearest in a single column, but to the some. Some form of like row based nearest across.

I understand. So. So you might have multiple columns that participate. But. Okay, the whole point is that the typical thing you need to use transform columns typically is using row local information. Except when you happen to want something from the whole column, for example, I mean, what do you typically want? You want. We already went through this. We want.

The previous value, the next value.

No, no, no. That's what you have if you have a loop. What? No, you don't want that.

No, no, I mean that for missing. It's a common case. People don't Necessarily repeat data.

Okay, I understand, I understand. But let's look at what that really is. Okay, so let's say you've got the column, the whole column, it's called col. Right? And let's say you've got the column, whole column, And you've also got the index of this element. You've also got. Part of the column with a particular value of another element. Right? That's the grouped case. Am I making sense?

I don't understand the last one.

Okay, so what I mean by that is that the part of this column of your current column for which.

Oh, I see, it's the part of the whole column.

Yeah.

You know, age is. Yeah, whatever.

Yeah, yes, but the, the first two can go in something like supplied. The last one needs an extra argument.

I know, I know, I know, and I hate the extra argument. So, so I'm just trying to understand what we do because here we've got a bunch of hacker rollers to do with the extra arguments. Okay, so I'm trying to understand what we do here. Now, what's the other function that we're using where we say, where we do the grouping. What's the function where we are saying that we can.

Table reduce. Is it one or the other? Double reduce. Okay.

Okay, So this has. What is this exactly doing? This is grouping by g spec. I have to say that a name for this function that involves the word by would fit in well. Well, at least if it's going to have G spec there. The fact that, that we keep on using the word aggregate in this documentation is not very encouraging. Am I making sense. Making it, suggesting that it should be tabular aggregate.

That was your original proposed name. You absolutely didn't like it.

You're. You're the one who killed that.

Okay, Only we should say reduces instead of aggregates in this documentation. Can we make that change? I don't want to make the change to my copy here. Makes sense. Does that make sense?

Yep.

Okay, so this has a grouping argument. I mean, in a sense, tabular reduce is what we are trying to do.

Well, a combination. Yeah. I mean, we want a way to switch between transform columns and tabular reduce at will.

I know.

No, not.

I mean, there is one very important distinction, which is that tabular reduce is always returning a tabular which has the number of rows corresponding to the number of distinct groups it could find. Whereas tabular transform tabular guarantees that you always return a tabular of the same length.

I know, yeah.

So this, so it's really.

This is row by Row. Yeah, Transform columns is a row by row operation.

Yes.

And, and what the. The way to understand what window functions do in general is that they bridge this gap on the input but not obscure way.

So let's think about it in a clearer way. Okay. I mean, we've got a function. We've got this function transform columns that works row wise. Okay. Occasionally it wants to pick up information at a given row position. Notice that. It wants the index at that row position sometimes, but it also wants the whole column, which is essentially a tabular reduce. On that column. Correct. I mean, basically what it wants.

That's just a part operation at this point. Unless you group it.

No, no, no, no, no, no, no. I mean, no, it isn't because in the end the thing you want to put in is not the whole column. You want to put in some function of the column which is a thing that can be fitted into that row.

Yeah, yeah, that's the end result of it. But you had like whole column as.

A. I know, but we don't want the whole column because what we're doing is look in terms of what we put in here. Right? Put in that plus the tabular reduce of the original tab. Yeah, Comma, like is like, you know, column J could be column J here.

Yeah.

Yeah. And. And the other thing.

Wait, wait, wait a second, wait a second. How do you specify tabular reduce? How do you specify the f spec.

Same as transform columns? Like if you want to. I is going to be I goes to mean of I of a hash I.

Okay, so. Okay, so that. God, I wish this thing just giving me the autocomplete there.

Yeah. You may be suggesting the other.

Actually, I have a new idea for that, which is that that this ampersand should just have some indication. Should have some little background color or something that indicates the extent of the function. I think that will be useful in general and it would save this particular problem. But the autocomplete needs to be there anyway.

Okay, well, I mean that would typically highlight everything because it's very loosely binded, but yeah.

Okay. Okay, so this thing, Okay. Or it might be that's for the whole column, or it might be the version of that for the sort of current group grouped by K and then, you know, current group version of this. You know what I'm saying? Yes, yes, okay.

With an extensive abuse of an option.

But yes, there is also a number of cases where you don't want to reduce the column, but you want to say, oh, give me the whole column. I was at index I give me the previous. No, missing one.

Or give me the cumulative sum at.

The point I was.

I understand. Yeah, I got it. We're not thinking about this correctly. Okay. We need to think about symbolic objects that represent these intermediate states. That's what we need to think about. Whether they're evaluated, whether they're lazy, I don't know. But that's what we need to think about.

I mean, I think that the correct way to represent this without any spooky action at a distance is actually a modifier. Like it's something that you wrap around tabular and say, oh, this tabular now needs to be annotated with this extra information. But that's not a tabular that you can represent as a tabular really, Like normal. That would explode. So it's like augmented tabular.

Just throw it away. Just throw it away.

Yep.

But it, you know, it's like it's really you augmenting the tabular with this stuff. If you want to make it as explicitly as possible.

Maybe, maybe, maybe. I'm not convinced.

But we are moving it for convenience to the function spec, which is ultimately incorrect in terms of composition of things.

Well, maybe. So let's think about it then. How do we get a thing that represents what we want? The problem is the thing fed to the function in transform columns is a row. Now, we could cheat and say that row. I mean, the row could perfectly well have in it, you know, value. I mean, it could have. You see what I'm saying? In other words, with wrappers like that, the rows, you know, and don't start worrying about implementation details yet, okay? Conceptually, what could be happening is that transform columns could be given a row that contains additional meta information. Do you see what I'm saying now? Now the problem is, Okay, what really has to happen is, okay, here's what really has to happen. This row has to be computed because we don't know what it's going to be grouped by or anything, right? We need a transformation from the raw row. Okay? Instead of getting a raw row, you get a transformed row. That contains the enhanced data. Do you see what I'm saying? So in addition to this, I, you're also getting, you know, whole column, arrow, whatever, or, you know, grouped by K, you know, arrow, whatever. You see what I'm saying?

Yeah.

You need certain other things. So it's as if what you want is something like transform columns. And this might not be right, but something like. A goes to transformer goes to function. You see what I'm saying?

Well, Wait, wait, wait. But if that will do multiple transformation at once.

Otherwise that is the transformer, that is the thing that returns an association. That is the thing that FUNC is applied to.

Okay, so you can get both the single row and the column by creating a transformation.

Let's say that the transformer. That the transformer function is provided with the whole tabular together with that argument A. And that's it. That's all it's provided with. See what I'm saying?

No, I, I don't know what it can do with that.

I could do anything.

The. The row index. Yeah. At least what's.

I don't understand why it needs a. I don't know what it does with A. Like A is what the result gets named to. So it seemed backwards that it also gets that because it doesn't care about that. That's like the left hand side of the doesn't care.

Okay, okay, yeah, it just gets TAB then. But now why is it that FUNC doesn't just get tab? Because then it can do whatever it wants with it. If func, in addition to getting the. Okay, forget this. FUNC gets the row a sock. You see what I'm saying? Which we know what that is what the row association is, right?

Yeah.

And if in addition to that it got tab, it can do whatever it wants. So for example, let's say it wants the whole column.

Yeah, but the. Or possibly transpose tab. The problem is that then we're circling back to having two being somewhat obscure.

I mean, also, this is not terrible to do with part. Yeah, so I mean, if you want to do the I minus the mean of column I, you could do I minus mean of tab part all I, that's going to be the same without any fancy extra thing.

You really don't want to have to.

Repeat the variable tab there.

What are you talking about? I don't understand.

No, no, I.

Okay, that's why what I'm suggesting is that rather than having this here, That you factor this out somehow. I mean, if mostly what you want to do is apply to the row association, it would be better to have a transformer function here. Okay, so what would happen if. I don't think the index is all that useful, the transformer got those things, then it.

I think, I think you're doing it backwards. Like the transformer get gets you back that it doesn't get it as input.

Oh no, no, no. It gets it as input so that you can create whatever association you want to apply FUNC to. Because you're saying the default is to apply FUNC to the row association, which has been synthesized. Do you see what I'm saying? Yeah, right. But what you're saying is occasionally you want to reach out of the row association to some other information. You see what I'm saying?

Yeah. What if I think that then the, the useful thing would be for transformer to get tab and index and not roasoc, because ROASOC is. Is tab part of index.

But, but no, no, but then you, you see, the whole point is to reconstitute. Look, I actually think there's a different thing this should get, now that I think about it, what is the transform columns argument structure.

It's two arguments right now.

Well, I'm wondering whether it should have a third argument.

Well, the last proposal that I had was with, if you scroll up, it's with three arguments. Sorry, four arguments.

That sounds right. Too many. Okay, so. So let's imagine that this is the. The row funks. Okay. And that then. See, because what I'm concerned about is you don't want to be redoing the grouping. In the tabular. Okay, let me float a possible idea. Okay, let's say there's a transformer function here that takes the main. The whole tabular, right? And it returns this thing here, returns an association that it makes up that has X and Y and whatever else in it. Okay. And the semantics then is that that association is appended to the row association, so that then the row FUNC basically gets to sample these additional pseudo variables that were found by doing something global to tab.

But.

I think about it for a second.

I think that the problem is that transformer of tab needs to return something that is also a tab, and for each row it returns one of those associations.

Right? No, look, look. Transform columns. This thing here knows it. It's mapping this over rows. Okay? So for every row. Yeah. It has to know which row it's on here. So it needs the index here.

Yeah, but what about this instead? Steven, if. Instead of like you just say that if you don't specify the third argument, that thing is part, so you just get the ith row.

Fine.

So transform. The default transformer is part. And then we can have fancier ones.

That's a good idea. That this is defaulting to part. And, and that. Would that give an association? It probably gives a tabular row, but we can.

It doesn't matter. It doesn't matter.

Okay, all right, but then, Well, so the default. Okay, so. I'm not super keen on that because the whole point is if you don't specify it, it can be this. Okay, that. That's easy to default. Right. But what I'm saying is if you do specify it, I think what you're going to give are things to be appended to the rower sock. Additional. Additional elements that you might want to use in computing a particular column value.

Yeah, but I mean it's really easy to like have. Like if you take that association that you wrote with X goes to xx, etc, that's, that's gonna like if you. Before that you put hash one, part hash two, comma, then it gets appended to that. So it's not.

No, it's not right. It's not right. It's not correct. It's not correct. What do you mean it gets appended? Where would you put it?

Type before the X in a string that. Type #1, part #2. Yeah, part #2. And then comma X goes to blah, Y goes to blah. Like associations are auto flattening. So if for whatever reason you don't want to return an association for this.

Well, I think you do. I think you do. I think you want to be made to return an association. I think it's a mistake not to because I think that I'd forgotten this auto flattening property.

I have to say that I really liked the double ruled thing that you were doing there because that's strictly more powerful. Because if you like you can do a different one for each A.

No, but this is like a transformer.

Function, that triple thing.

Well, fine, but if what that is if the transformer is given tab and index. Right. So let's say it's this and the transformer is given tab and index and the default for the transformer is part. See what I'm saying? Yeah, but the alternative is that the transformer can do whatever mischief it wants to.

Yeah, but I think that the useful thing that we have is that then there are going to be like named transformers like with, with groupings that you're going to stick in there. It's going to just do the right thing and we can document it as a separate function.

Yeah, no, no, I agree, I agree. But with, well, what would. With groupings do? I mean this is. Remember in the end. No, because you've got to specify what the whole point about this appending thing is that you can then do the operations and append them. So you say what the name of this with this grouped thing is, for example.

Yeah, but I mean we can.

How do you do that here how do you do that?

Here we do that. You mean with groupings can have sensible defaults that can be overridden?

Well, if we have with groupings, but this wouldn't need with groupings.

No, but what I'm saying is that the transformer is called with groupings. Maybe with groupings is the wrong name.

I was thinking of. Transformer is just a function you specify.

Yeah, but it's useful to have named documented functions that you do common things here.

I. I don't disagree with that. But the question is what.

And so what I'm seeing.

I see what you're saying with. With groupings of. And then you've got your standard function F. What you might say with groupings.

What I'm saying is that with groupings of, it's. It takes sub values. So instead, like with groupings of sex and class, comma, age. Synthesizes the right names for the extra keys in the association.

What would it synthesize?

Like, I don't know, row index with a dollar in front, as we said before. And if you don't like, specifies as options.

No, but I think you could specify this. So long as we have a wrapper here that is just like we have this extended key concept. Right. We could even use extended key. See what I'm saying? For that.

Yeah, yeah. I don't think you get sex age. You get column age.

Yeah, okay, whatever. Okay. But none of this solves quite yet the transform missing. Although it's getting closer. It's getting closer because it basically is saying, Wait a minute, doesn't this need A as well in the transformer? It needs to know what column you're talking about.

No, because A is the name of the new column you're constructing.

Okay, okay, okay, okay. But. But transform missing, its corresponding thing has a column, and it needs to know the transformer here. Oh, I see. There really, what it needs to know is just the column name. A. It doesn't really need to know the index, does it?

I think it needs the index as well, because, you know, how do you get the previous thing? You know, like, imagine something like you have a transformer that takes the tab and the index and just the index, and then the function itself is missing. Fallback.

I see. All right. Anyway, we got to go to a different meeting, but. But there's quite a bit to figure out here.

Yeah.

We'Re still not there. We're getting closer, but we're still absolutely not there. Anybody else? Have anybody have any final comments? Christopher? Anybody else?

Not really.

How horrifying do you think all this stuff is? I'm just worried that we're going to build something that's just completely unusable. I think it's really tricky to get this to be usable.

I think if there's a set of symbolic transformers that pre built with the very common things, this is going to get used if you have to write the function yourself.

No, I agree, I agree, but we need to have something.

Yeah, I mean, with all this stuff, I mean, it just seems like, you know, it's one of these things where 95% of the time you want one of three relatively simple behaviors.

Yes, I agree. I mean, and for the missing thing, similarly, I'm a little worried that some of the many of these missing things are parameterized, like constant and so on.

Sure. What's wrong with that?

Well, nothing, but I don't like this way of doing the parameterization. I think this is a mistake. I think by the time there are this many, one's going to want arbitrary functions like interpolation, for example, where, you know, interpolation is a. Is a complicated rich function with degrees and methods and this and that and the other. And I think it's a mistake to try and stringify it here. It's better to, you know, since what we're actually going to do is, is basically take the elements of the column, feed them to the interpolation function and get the result. Right. Or mode, for example, like commonest.

Right. At some level, though, it might depend a bit on how. Like, right. In some of these cases it might be the case that like. Right. If you just give me the column, then it's trivial to call interpolate to get what you want. But actually like for example, here it's not trivial at all. Like, because interpolate doesn't just take things with missing values, I don't think. No, so like you have to. It actually requires a bunch of.

Right. But I claim that that function interpolate with missing values is a useful function and should not be hidden.

Yeah, but, but it's. It will not use the interpolating function. It will use a set of values near the missing one and it will interpolate around.

But, but then, but Roger, my point is, given what you just said, being able to do that on a list is useful. Yeah. Don't hide it inside this, inside this tabular thing.

Right, but, but then the question is. Yeah, I mean, but then also the question is. Right, you know. Okay, so, so one option would be maybe in doing this, we realize Here are the 10 different kind of missing synthesis, you know, things that you might want and we make 10 different functions. Alternatively, we could have one function kind of like synthesizing value that has 10 different.

Yeah, I know. Usages. Those are possibilities. Yeah, right.

But I think the second probably doesn't make a lot of sense because my I. I would guess that a lot of these would take very different sorts of inputs. Like interpolation is only going to take or probably is only going to take the one column that you have missing values exactly where others like learned distribution is going to look at the whole table. And so.

Right.

So my statement is maybe one of the exercises for next time think about what top level functions which, you know, these are new top level functions, but for which we didn't really have a thing that wouldn't, you know, conveniently prepare inputs before their top level functions whose utility is revealed by tabular. That makes sense.

Yes, yes.

But we shouldn't just ignore those functions. And I really need to go and I'm sorry, but that's what I think we should do. I think we should think about what are those functions. Okay. You know, the typical elements of each column. Well, that's a. That's a very useful thing to, you know, that would be a. The tip, you know, for a list. Just give me a list.

Right? Yeah. The idea is that that is a combination of mode for categorical and mean for numerical. But we can do several interpret.

There are several things, but that's something you can give a list. You know, a list could work that way as well for ABCs or 1, 2, 3, 4s, you know.

Yeah.

Okay. All right. To be continued. See you guys soon.