Live CEOing Ep 824: Language Design in Wolfram Language [Tabular]

Hello everyone. Welcome to another design review on the subject of our friend Tabula and what are we looking at today?

Okay, so we need to wrap up the discussion about grouping. Carlo couldn't make it today, so he asked if. If we can wrap it up with. With him. And we have like touching that. There's another problem that we need to address that I'm presenting here that is related to data cleaning.

Because let's just remember by grouping, what we're talking about is the ability to make use in functions of like in construct columns. To make use of information that is not row wise information. More global information, non global information.

Exactly. It depends on the whole column, on values in the rest of the column. Or look at the rest of the.

Columns and where have we got to last time? What we had gotten to last time was you can open it.

There's the transform missing and with groupings they have all the nodes. So yeah, we talk about operation forms and things like that.

Right. So the main point here is that we need a way, as we talked about before, there is. Okay, you know, these are row wise operations, basically transform columns is fundamentally a row wise operation, correct?

Yes.

Okay, so it needs a way to refer to things that are not row wise. And in particular it needs a way how do row wise operations, although I transform missing might not be row wise at all, but. Okay, row wise operations like transform columns refer to whole columns or parts of columns that are the result of grouping. Is that right?

Yes. There's other elements that are not so far away from applied and the main one is the index. So that to me, the only connection we have with supply, given that you brought it up a couple of times, is to get the index.

Wait, wait, wait, wait, wait, wait, wait. You're saying that there's a third thing here, or to the index either within the column or the group, Correct?

Exactly. And there's actually a third thing that Kano brought up in his original prototype, which was the sorting. You know, like you.

Once you. I understand that. Right, yeah. Or the index with respect to some sorting. With respect to a buy sorting, so to speak.

Exactly. Yeah. This was a full like spec, let's say the list of things to.

To have what is in the analog when you say sort by. Okay, so if I say sort by, you know, If I say sort by this and now I want to find where is the one in that? Is there a function that gives me the index ordering by.

Do we have it?

Yes, yes we do.

Okay, so this is going to be that.

Well, let's Take a look. So what is this really telling me? This is telling me that element is the third element. Okay? So we do in fact have a mechanism already for dealing with this with respect to a sorting. Am I making sense? Yep, yep. Which is essentially an ordering function. Do we have. We don't have that, but we could have that. Okay, so now as you saw, we.

Have the method of an ordering function, A two arguments ordering function or the by function one, a unary bi function.

Yes. Yeah, fair enough. Okay, so now, all right, so let's just understand, okay, this is for construct columns and transform columns. Correct this description. Now, transform missing. Is that a different description of what we need?

Okay. My original idea was that transform missing would take a column spec, same as transform columns, just not to confuse things too much. And also because it's really column by column, typically. And then it would provide a function that tells you how do you find the missing element of that column, like a missing spec. Sorry, how do you find what to put in place of the missing element of that column?

Right, I understand, but yeah, so like.

The shortest thing would have been, you know, a goes to mean with the tacit assumption that you apply mean only on the.

I understand, I understand, but so what's happening is the difference is that in transform columns, okay, that column wise operations are kind of a. In that here, column wise operations are the exception.

Exactly.

Here column wise operations are the norm. Right?

Yeah. This is. I mean, the order of application, the direction of application is the one that we have in reduce tabular without the reduction. But you get the transpose tabular in. Now the problem with the missing transform is that the point is what's the added value compared to just using transform columns? And it has to be that the syntax is.

Let's get to that in a minute. Okay, let's get to that in a minute. I'm not totally convinced that.

Okay, then let me tell you another thing that we need to transforming thing to do. So maybe we don't do it. But in terms of things, some methods are constant, like mean mode or like commonest or a constant.

Some of them are things like random numbers.

Random numbers. Okay. Maybe we find a way to do delayed. You can use rule delayed and we can say that that is evaluated every time. That's one of the easiest. Some others are position sensitive. You know, for example, you train, I.

Give you a much worse situation. I'll give you a much worse situation go. Which is that fill in all the elements of a column so that they are. So none are repeated. You understand what I'm saying?

Okay, so you, you. It's. It's basically a random choice without replacement.

Well, right, but let's imagine that what I've got is 1, 5, 2 8, okay. With some missings in here, okay. And I have a global constraint that I cannot have the same key twice.

The same value twice. I mean.

Yeah, no, well, same. These will be keys, I mean, in the end or something. Right. You don't have to think of them as keys. They're just kind of cannot have same value twice.

Right, okay, but then you, you still need a rule about how to fill missings, so you cannot stand.

But okay, doesn't matter what the rule is. What I'm saying is it doesn't matter what the rule is. Just think about it. You can't do this locally. You can't do this row by row.

No, right, exactly.

But I don't know whether there are other methods that can't be done row by row. Do you understand what I'm asking is.

No, no, I do, I do. And this is why. This is why the moment started. The moment we started digging into transform missing, we immediately fall back to transform columns where you have this switch of looking both at the row and at the whole column.

Okay? Not what we do. This stupid shit is still here.

Doesn't transform. I mean, isn't replace missing sort of in place and transform column add stuff or at least moves it?

No, they can both. I mean, transform columns can be in place if the column already exists. And I imagine that transform columns is kind of the workhorse that we are going to use to implement most of that stuff.

You need to. Well, I guess you need the anchor column, right? You need to.

No, I don't think it resets the ordering. Like if you have ABC and you do, B goes to rule, B stays in place. The way it's currently implemented. It's not appended at the end. Well, that's what you're asking.

How things stood a few weeks ago, maybe that's changed.

But we have a mechanism to put things. Hey, we have a mechanism to put things after the column. You know, we have this whole mechanism for addressing which. Where we put a new column, the anchor.

Yeah, you could say C, B goes to rule. I mean function.

And then you've got the anchor. You've got the anchor column specified there.

But if you don't use anchor. That's the same semantic.

Yeah, yeah.

Right.

Okay, let's get to what we need to do here. Right, so look, the basic point is you Know, you know, in a sense this is, you know, it's really a column transform. Missing is what we're really talking about. In other words, it's being fed. You know, I think everything we do here, it needs to be fed the whole column. Am I making sense?

Yeah.

I mean there are versions of this. Maybe we wouldn't use transform, you know, transform missing for them, but there are versions which actually don't look at the column.

Right.

They look at other elements of the row.

Well, why isn't that just.

But it's very rare because of a type constraint.

Wait, wait, wait, wait, wait, wait, wait, wait, wait. Isn't that just transform columns if they're row wise?

If it's row wise, you could use missing fallback. I don't remember what's the final name of that? You could use missing fallback on the row.

But also exactly what Stephen says, transform columns. Look at the other parts of a row. Yeah, no, no, it's like use.

I was expanding on that. So you use transform columns and you do missing. You do B goes to missing fallback.

Yeah, missing fallback might be useful. Yeah. Right.

And that, that is basically doing that. It's taking the first non missing starting.

Yeah, but that's a different. That's a different operation. That's a separate operation.

Agreed.

The important point is that it's a row wise operation.

Yep.

Right. It does not need to go. Missing fallback is purely row eyes. Right. So in other words, the. The big picture is transform columns does things row wise. Okay. Although, wait, how do we narrow it into the parts where we actually need to do it for transform column only, the missing row, the rows that had a missing. Well, why don't we just have an operation? I mean, why don't we say just A goes to replace. Or is that not right?

Sorry if I had it again, but I thought that missing fallback is exactly. That function is selecting the first node missing. So you can say missing fallback like a goes to missing fallback of a comma the rest of a row. And if A is not missing, it will stay A. If it is missing, it will go, you know, among the rest of the things to pick them up and you can put them in a specific order if you want.

That's not dealing with a different. That's not dealing with the su minus nine.

That's pound A pound A comma, pound whatever.

Right, yeah, in there.

Yeah. Pound A comma pound B comma pound c. And that is using the rest of the row in order. That is like select first of a non missing element of that and that's it.

Well, that should be a list, right? It's a list, correct?

No, we. Okay, this sequence.

This is something that, while implementing stuff using this or like prototyping syntax is using this. I mean one thing very convenient would be to automatically sequence out list.

Automatically. What?

Like make a sequence out of a list here?

I don't think we do have sequences in here. I don't think we should. I think we're going to need options to missing fallback. I think it's a huge mistake to have a varogs thing.

Okay. And should we flatten a list by default or no?

I don't know. I don't know. Probably yes, but I don't know for sure because.

Because a great syntax will be list of hash A. For example, let's say that you want to take the first non missing element in the column. Yeah. You do a comma, some syntax to extract the column A.

No, no, no, Wait, wait, wait, wait, wait, wait. You're mixing things up.

Okay, okay, yeah, yeah, you're right, you're right.

You're mixing things up. This is a row wise operation. Okay?

Yeah.

So I mean, basically let's imagine that what you want to do. Okay. So there are two basic things. You either falling back onto a different column, which is transform columns. Missing fallback.

Exactly.

Okay.

Or.

You'Re looking column wise and I mean we're making the approximation that everything is rectilinear. You're either looking row wise or you're looking column wise. You're not looking at blocks of this and that which have both rows and columns. Am I making sense?

Yeah. In the general thing you could be doing that. And this is how we set up.

No, if you want the expression, you can do whatever you want with it. Fundamentally transform columns is doing row wise stuff with an escape valve to pull in whole columns. Am I making sense? And what would that look like? In other words, let's say we want this thing and let's imagine that we want the missing fallback here to be the. By the way, assume missing fallback is a function that takes the values that it encounters on that in that column on that row.

Well, no, it's a missing fallback is a function that is applied to arguments, right?

You can.

And they believe it's implement, right? It's black. So you can just do, right, 2 comma, miss, you know, 2, 3. No, no, 2 comma 3. And you'll get 2. No, no, no, no, no, NO. It returns the first non missing thing. So that gives 2 and then if you do missing fallback of missing bracket, bracket of three, comma three, then you'll get three.

Exactly. So for example, I'm putting something in the chart.

Wait, wait, wait, wait. What happens with evaluation? Is this something that is controlled evaluation? So it doesn't evaluate the three right now?

I don't know, but it should. But it should. I mean, I think we should fix this. We shouldn't evaluate typing.

Well, okay, so if that's the case, that's something of an argument in favor of an non of a verag's version. Am I making sense? Right. Do you understand what I'm saying?

Yes, yes, yes, I do. I do.

Do you have a comment on it?

I'm thinking about traffic use cases in tabular. Most of. I mean, I don't imagine many delayed values in tabular columns. When you pull in data like real world data you're going to have depends.

On whether it's an expensive operation. If it's an expensive operation, you don't want to do it unless you have to. Like for example, let's say that operation there is to do some anomaly detection on a column. Do you understand what I'm saying?

Yeah. You don't have simply hash column, you have function of hash column.

Well, no, no, no, no, no. It's just you don't want to do that operation until. Unless you need to. Unless.

You mean by that operation the one.

Plus one in this case, right?

Yeah, don't be silly. Julio, my point is that when you said half of f of hash column, right. You were representing some general operation on a column.

Yeah, yeah. Because to me, when you do it in, in, in.

Clear. Okay, look, look. So we've got the issue of right now this is all row wise, okay? We talked about last time the, you know, how to represent the. How to represent pulling in a whole column or the grouping section of a column. Am I making sense?

Yeah. And the thing we didn't resolve is where do you put the grouping spec?

Okay, so.

And that's the thing that, you know, Carlo asked that we postpone until he's back.

Exactly. Exactly where we put the grouping spec.

We're doing this now because we're doing this now. We're not going to wait because you wait, we'll all have forgotten how this works.

So one thing I put in chat is exactly this operation without the grouping. Is it with, you know, row and column is exactly what you're talking about, Steven. There's nothing, nothing separate. Yeah. Now, like here I Put at least. Because we're discussing.

I know, but we didn't want this. This isn't going to work. Okay, Okay.

I wasn't super sure we dropped this completely.

What's that?

I wasn't super sure that this was mixed, you know, completely last time. Okay?

Okay. But what we want. This is just absolutely horrible.

Okay?

Right.

Why is this super horrible?

Because it's a total hack. Let's think about what it is we're actually trying to do. Okay? So we have. We've been given an association. Okay. I don't understand. Okay, so the issue is we need a thing here. I have an idea. Yeah. So missing is a little bit like a categorical column. Okay. It's basically a binary vector, a boolean vector that says if something is missing or not. What Carlo talked about is group by, which is using categorical columns, but they're all effectively categorical columns. The missing is just not there explicitly. It's sort of stored underneath because it's a property we need to know. Yeah, yeah. So if we had for transform column a way to home in on the sections that we need, whether those are missing sections or categorical value sections. In other words, you're saying a grouping by the. By missings in column C or something. Yeah, for instance. Exactly. The missing, so to speak, that way, you know, so one of the things that would allow us to do is kind of just drill right down to the things we need to transform and not deal with all the others and have fallback and all that stuff. I don't think that's right. I mean, that's not what the fallback is doing. The fallback is row wise. Yeah, but you, you.

Let's say that you want the fallback is nothing, right? The fallback is just a normal function. That's like saying that plus is row wise, right?

No, but it's applied to the association. This function is applied to the association that contains the.

That's only because that's the right hand side of transform columns, right. If you did missing fallback, I mean, it would be a stupid thing to do in a, you know, in delete missing or transform missing. But if you did that in trend, you know, delete missing, it would be applied to the whole column, right?

Okay, you're. You're saying missing fallback. Okay, but the important point is, okay, because by. By we have the convention in a sense, that you put things like hash A and hash B which pull out elements of an association because the thing it's typically being fed is the row which is represented by an association.

Yep.

Okay, so let's. Let's come back then. The issue is, every so often we might need to gulp in the whole column instead of just this element. Right. That's the main thing we're trying to solve, correct?

Yep, yep. This is what I put in that syntax.

Yes, I. I know, I realize, I know, I know. And one of the things that's confusing is that transform missing generically takes whole columns. It does not typically want to refer to individual, you know, row elements.

That's true. Right. I mean, if it's some sort of local, you know, take the. There's non missing. But.

But it isn't. Itay. It isn't. It just isn't. I mean, this thing, its name is confusing and probably wrong, but fundamentally what it's doing is it's gulping columns and processing those columns. Am I making sense? Yes.

Yeah.

So an idea that Rob had suggested is a column wise, that is a wrapper to the functions. And so that function will now receive full columns. A will refers to the full column.

Yeah, well, I was thinking about that. I was wondering whether we could get. Whether we could do something where we say, is the. Is that possible to say column wise hash A? That gets us just like element wise is getting us just the single element. Right.

Why is that?

Does that have some current meaning?

Because Rob has already implemented something.

Yes, it is implemented as a headdress. So you do column wise mean of hash A. But Stephen didn't like that change of that thing.

So this is a function wrapper.

Like element wisest. Exactly. That's not completely crazy. But I don't know what it means because this is. Oh, I see that. That's.

So, for example, in construct columns, we could say the name of the new column arrow, and then these. Which is an operation on columns.

But I don't understand semantically how that works, because normally the function, rather than.

Think of accumulate, so accumulate of hash A.

So change how this works. Because isn't the function in transform columns simply fed the row aka yeah, exactly.

This is a. This is the thing we really like that element wise is not changing its argument, it's changing the semantic of the function on the argument. Column wise is actually changing.

Maybe. I understand. Maybe what we should think about is maybe the way to think about it is this. That when we say a arrow F. Okay. That the true thing that's done here is that F. Okay, I'm just. Let me try this out. Okay. That. It's more like this. That F is really given the whole tabular, the index. I mean, conceptually, it's not how it be implemented. Okay, but conceptually, it's given the whole tabular, the index. And the specific thing here.

I'm confused because this way, F gets a cell.

What?

Tab I, A. That's a single value. Yeah, it's row I, column A.

No, I mean, look, look. Oh, yeah, yeah, you're right. You're right. It just gets this. Yeah, it just gets that.

Exactly, exactly.

That's what it gets. So, so in other words, that this. Right now, that. Okay, so that's what it normally gets. But let's say that there's a function G that is the function that determines what it gets. Okay? And normally that function G is. You know, normally it's just hash one, the table, or even. Or even hash, tabular of hash, index.

Okay, okay, I see where you're going.

Right. And then. So that, for example, it would look like.

Sir.

Okay, you see what I'm saying? So that. Then what happens is. Let me think if this works. So let's imagine that conceptually, the function G gets a. An association. That's the full tabular, the index of the row, you know, and the column that we are currently dealing with. Do you see what I'm saying? And it could get other things as well, like, you know, a grouped column or something like this.

By the column, you mean the full column or like a.

No, no, the full column. The full column. The column. This would be. That would be the column. Okay? So let's write out what these things are. Okay, that's just the index. And the column would be tab of all.

All A.

Well, yeah. In list, in this form, all comma, A. Right.

Exactly.

And by the way, we could also, as a courtesy, we could also provide rho here. Whoops. See what I'm saying?

Which is tab I.

Which is simply tab I.

Yes, tab, index. Yeah.

I think this is not completely stupid. So let's take a look at what this would look like if you want to do something on a column. Okay.

Yeah. We should write it because I think this is powerful, but might inflate the syntax.

Well, let's take a look. Let's take a look. Then we can. Once we have something which is understandable. Right.

We can simplify it. Yeah.

Right. So now the.

I mean, one simple operation you might want to do with this syntax is subtract the mean of the column from each element.

Okay, fine. So let's do that. So this is a. But that's not good enough. So what we've got here. Oh, I see, I see. Look. Look at what this looks like. This is actually fairly non stupid. Okay, so.

Now, first choice, do you want to put.

Wait a second.

In a list? I mean, how do you spread out things in a list in association?

No, no. What? You've been given this function here. Maybe you want the whole thing. I mean, that function is. Okay, so let's imagine. Here it goes, hash row, sub, hash, a comma, hash column.

And this, okay.

Goes to. Hash1 minus mean of hash2.

Okay, with the caveat that we just signified that the list is provided as a. Now the function is applied on that.

What do you mean it's applied this thing. That thing returns a column.

No, no, but the whole thing returns a rule. It's a list. Yeah. And then you're using not hash part one, you're using hash one. So there's some hidden application somewhere to make it work.

Yeah. Okay.

This is my original question about how you spread out the arguments fair.

I mean, it's a little bit overbearing and you might wonder why we couldn't just say mean of hash column here, which is what you're kind of doing here, except you don't need the C there. Yeah, I mean, that's this column. You want a different column, that's a different story. I mean, you know, it's conceivable.

You.

Know, it's conceivable that if we had this syntax as the full form, that we could get away with some cheap syntax where we have column in here, and if you were using, you know, column as an explicit name, you would use the longer syntax. I'm not sure, But now I don't understand. In the case of. Of transform missing, the problem is that the default is to use the column. Am I making sense?

Yeah, yeah. I mean, it depends on where you start. Yeah. The problem transform columns is how do I get access to the full column? Implement operations in transform missing. That's easy. So if you want to do a random choice or a mean or whatever, it's super easy. The problem is how you do something like what I wrote here. You know, like I want the index and I want to just check from this point forward the first one missing. Something like that. So this is why we ended up writing those string methods. But you suggested that we think harder about how to avoid them with the good syntax design.

Right. All right, well, we've. I think we've crispened this up a bit. So, I mean, one drawback to this.

Is that, you know, the Functions you, you will now typically use will not have. Well, I suppose if, if you in in transform is in in transform column, right. Your function names, as long as you stick to the default which will be list of pound row.

Right.

Then you can use functions that use the names of the columns. If you're in transform missing now you'll need to use sort of function names that are in slots or something. But maybe that's already a problem for transform missing. I'm not sure.

Yeah, I mean, look, look, okay, I'm not sure this is quite right, but it's at least clarifying what the issues are and it is giving us the opportunity to have lots of different cases for what you actually provide to these functions. Given that there's a simple case where you're just doing it locally. Am I making sense, you know, missing the common case?

Yeah, I mean in this, in this setting, the difference between transform columns and transform missing is that G is different. Basically it has a different default G. Yes, that's right. If you want to, you know, recast it in this. In this way.

Yes, yeah. I mean in the default, the default here would be that this is merely hash row.

Exactly.

Well, the alternative here seems to me is to have a symbolic wrapper. That for example, would be. So this is alternative. Would be something like a goes to. You know, we need then a contextual. Hash a minus. And now we want something. Which is essentially a local. It's very much like supplied. Very, very much like supplied. It's. It's something that says, you know, whole tabular because it's like a self of the whole tabular. See what I'm saying?

But I don't understand why this is not column wise.

Again, it's not column wise because maybe.

With a different semantic column wise. No, no, I know, I know what it is now. I know what it is now. But do you remember, Jose, why Rob suggested this specifically? Because, you know, another idea I was pointing with for column wise is that it's something that changes the meaning of hash. By default, hash in transform column has a meaning in the part of the function wrapped by column wise. Hash is like reduced tabular. So would be mean of column wise A that suddenly you get access to the whole column of A. I know.

But it's not the right name. You understand what I'm saying? It's not the right name because it's not. Look, I mean this could be, you know, this could be, you know, I don't know. You know, this is what this does, right? That's what? This?

But why a string and not hash A? I would still like to use slot.

Because it's not the right thing, right? Because this is the way of accessing that column within the whole tabular. Do you understand what I'm saying? Right? That that's. What am I, am I met?

No, no, I understand. I think we. Okay, I thought we were thinking something. We were thinking about something similar, but we are thinking about two different things. I was thinking about something that blocks the slot in a different way. You're thinking about the right thing to do.

Something that I don't think it's the right thing to do. Because I think that the, that that slot, the slot is already applying itself to the association that is the row.

Okay? It's inside the function. So it's too late to change the meaning of slot.

That's correct. It can't escape. The only way to have it escape is to use supplied.

But then we lose the core meaning of the slot, which is the row. And then we need to manually specify what the row is or use that reserved keys that you didn't like, Right?

But no, no, no, no. Supplied could do this, right? So supplied lets you get, you know, supplied could get the column here, okay? Because it can get external information. That's the whole point. Supplied, get.

No, no, True, true information. But then hash A becomes hash row, part A. That this or hash row function.

No, no, it doesn't. No, it doesn't. This is just. Supplied is providing.

Oh, you say not for the whole function, only for that part with the mean.

Look, look, look, look, look.

I see, I see, I see, I see, I see, I see, I see, I see.

Well, I'm not sure that's right, actually. I think this has to be. No, this isn't a mess because this has to be not good. Because supplied is for the whole function. But we have to pull pieces. Well, maybe that's. Maybe that's a problem with supplied.

Yeah. Okay.

In general, maybe.

Is the supply working at any depth or only at top level? Maybe this was Supplied only works at.

The supply has to be the outermost wrapper. It tells you how to find these extra things to the functions. That's that you're going to give.

Right?

Okay, okay. I mean, no changing the meaning.

An alternative approach would be that one could say supplied of, you know, abc, whatever it is, the things. And then the function here could contain, Let's say, yes, something like that. In other words, so. So that it's not. So that it's. It's instead. Instead of being a thing where you change the whole function setup. You're merely providing something which is then accessible. Although then you might wonder why do you need the supply to begin with? Why don't you just have something that reaches out? Like for example, if you were doing map, you would say map of, you know, f of hash comma.

Know.

Why couldn't you just do this? See what I'm saying? Which might.

So like a fancier version of a dollar transpose without the. The need of being tied to an argument itself. Yeah.

Where supplied element refers to the map. I mean, I, I don't know. Maybe that's not right because. Because you really want to be able to see. Because what happens if I said, you know, two plus supplied elements, etc. I mean, that. That has to work, obviously. So what it would have to do is it would have to scan that. That. I mean, in other words, every function like map that has suppliable function arguments would have to scan its function arguments for supplied elements and replace it.

You cannot do it. It has to be done runtime. Yeah, because it could, however, notice obviously.

It'S done at run.

Okay, okay.

Right. But it can be done somewhat efficiently, I think.

But that's, I mean, I don't like this direction of supply at all.

Why?

Well, because.

Right.

The, the current design, I mean, there is a, There is a mode where, where you scan for things. But you know, the basic design is, you know, you just continue using the functions you've always used, but you can now add more stuff. And this is sort of breaking that.

What.

You can use the functions you've always used. This is just a way of inserting additional environmental information. I mean, look, this is like.

Okay, right, well, but, well, basically you're. You're forcing any use of additional arguments to be lexically inside. So if you've defined a function, you can't.

Yes, this will be a lexical thing. This will be lexical. But it's a little bit like saying, you know, putting stack in there. Although that I agree is dynamic.

No, but, but why it has to be lexical? I mean, like is something like block. Yeah, like, you know, the value of.

Supplied is blocked with the current design of supplied. Right. You can define, you know, f of X, Y equals whatever. And then you can do supplied of, you know, give me these two things of the supplied comma F. Right. But you can't do that with this design.

No, you're doing something slightly different. This is assuming that the things you're adding in the supply World are kind of add ons, right?

Whereas the current supply design is more than add ons. It's that you're going to have to write the function inside of map.

Well, only if it's lexical.

But how can it not be lexical when you, when you need a literal supplied element inside of it?

You don't. I mean, this could look supplied element at runtime. Okay. If supplied element was prepared to crawl the stack effectively, it could find what it needs. Maybe.

But you're still not telling me how if. Right. Right. Now I can define, you know, F of X Y equals X plus Y. Right. And then do you know, supply. Well, I mean, assuming this were actually implemented. But if it's not. But the right, you know, supplied of, you know, list, you know, whatever.

Can I just make a point about how this would work? Okay, let's imagine that what MAP did was that MAP's effective semantics were, you know, map of F comma X would effectively do block, you know, every time. I mean, in other words, what it would do is instead of just saying F, it would say block. You know, I'm. I'm writing this out in a. In a fake way. Okay. Right Equals four or something. You know, se of. Of whatever it is equals. I mean, this is, this is obviously fake, but you get what I'm saying. Comma F of X, right? If it was temporarily binding supplied element things, then you could pick them up dynamically inside F. No, I mean, that's just Right.

I mean, actually write what I was telling you to write.

Right?

Map supplied open list, you know, content comma index, and those are strings and initial caps. And maybe content is something else.

But. Yeah, okay, okay. Body, whatever it is. Yeah. And then F. Yep.

Right. That works. This cannot work if you need to have a supplied element inside of it.

I don't understand. I mean, in, in this. Look, this here could be.

F is F is F. Right? F is, you know, pound one plus pound two.

I understand. I understand. So it's a different thing that. Okay, you, you're. You're. You're assuming different things about when you want to insert the index.

I'm.

I'm not.

I'm asserting.

No, no, no, no, no. You're saying.

Stephen, let me finish my sentence. I'm telling you that we designed it so that you could take any function.

I know. So I'm saying. I'm, you know, I'm saying, you know, maybe we should reconsider that because I think that being able to say that would be rather convenient in other Words I don't have to. I can add that in if I initially have f of hash 1, You know, and then I say, oh, by the way, I want to add something about the index. I don't have to rewrite F. Well, I mean, I have to.

Well, hey, with my current. With the design we had. You don't. Is exactly the point. You have to rewrite F because you need to stick the things in there, literally. And. And stick.

Yeah, I know, I know, I know, I know, I know. But as I'm thinking about it, and I don't know for sure, but as I'm thinking about it, this might seem. I had not figured out that this could possibly work, right? But this seems potentially more convenient than this, because let's say I want to write map indexed. Okay. I mean, map indexed would be map of supplied content, comma, index, just like that, right?

Y.

Correct. But let's say I mostly want.

Well, it would actually become a part.

Not index, but yeah. Okay. But, yeah. I don't know. Anybody else have an opinion about this? I. I would like to look at this some more. Okay. Because we, you know, clearly. I'd like to look at this some more as a. As a. But let's go back to this, because what we're saying here is. That you need access to the whole tabular, right? Basically, that's the fundamental thing.

If I need to be picky, that's the transpose tabular for your syntax here. Unless you want to write again or.

No, fine.

I'm saying this because we always end up with this. This issue.

Yeah, yes, I understand that, but my question is, I mean, it really isn't a hash anything. It's not being applied to anything. It's just a function that is, you know, it's just a symbolic thing that has to be recognized. In other words, we basically have to scan these functions here, right? To see whether there is a whole tabular there, just as we'd have to scan it for those special hash objects because we don't want to create them. Like with handler functions, we don't want to create them unless they're needed.

Well, I mean, this case is a pointer. Yeah. In whole tabular, it. It points to tab while you execute the function.

I, I understand, but you have to set that, and that's fine. And, but I mean, this seems like it would work. This seems somewhat uncontroversially. It will work. And, and it gives you all the flexibility you want if you want to do all kinds of grouping things, et cetera. Et cetera, et cetera. You just redo them. You just put them in right here.

Okay, so it's basically. It's like having tab but anonymous.

Yes, that's right. You could equally well write this by having set tab equals tabular and then do all this stuff.

Exactly. But this way your function is independent of the actual input of transform columns.

Actually, this isn't right. Well, wait a minute. I mean, the problem is. No, actually this is okay. This is okay. I'm sorry, I'm just trying to think about. You've got access to two types of information. The local row information that you've been given an association that you can index into, and the global tabular that you can all index into and get whatever sub piece you want. Right. I think this is. I think this is the right way to do it. Comments? Roger? Christopher? Anybody? Jose?

I was thinking that this feels like a column slot instead of having slot A. It's like a different type of slot.

We did talk about that before, but I think it's more flexible. I think that's. Well, I think it's more flexible to just have, you know, to just give you the whole damn tabular because you've got all these operations that you can do on the whole tabular to do with grouping and sorting and this, that, and the other. Otherwise we have to come up with kind of crazy secondary syntax for all that stuff.

So, actually, a few questions. So the first question is, when you do know if you have. Right. If you have a pure function and you pass it an association first argument and then some other arguments, you can access the keys. Can you access the keys of the association with, you know, hash A, hash B? And then can you access the later arguments with/2, 3, 4, etc. Like, can you do both of those simultaneously?

Don't think so.

Okay, well, I guess.

Isn't hash A? Well, let's. Let's look at that. I don't know. You're saying, yeah, hash, hash A. You know, let's do this. Hash A, hash B, Hash maybe hash one, hash two.

Yeah.

Oh, beautiful.

Okay. Yeah, it works. So, okay, so two things. The first thing is, it seems like the most, like, conservative way we could do this is just by having hash to be the whole tabular.

Yeah, that's a good point.

And then the next part is, if we didn't want. If we want to be a bit bolder and have some whole tabular or something like that, then. Okay, then the first question would be, you know, are we going to get ourselves into Trouble or something. Because now we have to traverse the input function.

You're right. This is a trivial way of doing this. This is the trivial solution.

The only problematic thing with this so.

Far, we are two meetings ago.

I mean, this whole thing we have right now works also for symbol with down values. Suddenly we have.

What? What are you talking about?

The function, you know, like let's say A goes to hash A. That. That's. That could be also A goes to F and that. Then F is look up A or whatever. Yeah, but then.

We'Re in real trouble if we. I mean, we can't support a whole. You can't have it so that in the body of F you make reference to some whole tabular or some special tabular slot thing.

Right?

Because then when you pass this F into this tabular function, it has to traverse, you know, all the down values to the core of the earth.

Okay, so, Christopher, you're saying whatever solution we find, we can never support symbols here. Only two functions.

What the hell? What kind of a symbol?

So, Stephen.

No, no, I mean, why can't you use symbols?

Yeah.

The point is, then you need to define all your functions to take two arguments, not one. Every time you use transform columns.

Yes.

Wait a second, wait a second, wait a second.

Yeah.

You're saying that if we give. The problem is that if we have a function F.

Let'S do the class of plus one.

Yeah, hold on, hold on, hold on. To which we provide that. To which we provide half F of A, and then we give it two arguments.

Exactly. That will topple over because there's no pattern for two arguments. There's only pattern for one argument. Well, no, no, sorry.

Pure functional work.

But you couldn't just say F. Yeah, yeah.

The point is you'd have to write F of #1 as opposed to just F.

Right.

Well, well, but, but I mean, isn't this. I mean, don't. Don't we need to have the pound A there anyway in order to.

No, look, the problem is this, that if we say F here, that F needs to expect two arguments. If we F of uv, F of U is blah, blah, blah, and then we feed it F of this, it's going to fail. Right.

And so I think that the, the point. So maybe one question would be. So are we expecting that most of the time people are only gonna. Right. Like if I have my, my function F, most of the time I'm going to be applying.

Most of the time they're never going to care about the second argument. They're Never going to care about the second argument most of the time.

No, I'm talking more about are they going to care about the whole hash1 or only a hash a? Because if they're only ever going to care about the hash A, then it's pretty reasonable to write F of hash A. But if they're going to care about the whole hash one.

Yeah, right.

See, I just don't understand what the problem here is. Right. Because if you. Right. Even ignoring the whole conversation we've had today, can you do. Can you currently do pound a goes to F? And if so, how does it know that it's going to apply that to pound A and not to.

What are you talking about, pound A? There's no pound A. It's F a goes to F. Okay. If you do that, it will apply F to the whole association, which I admit is probably a rare case.

Right, exactly. And so I, you know, I don't see.

Right.

In the typical case, you're going to explicitly have some, you know, pound a's in there or pound whatevers.

Yeah, and so it's a good point. So it's not the whole row. And so if you are going to use the whole row, you can know that there are actually two arguments there. The row and the transpose tabular.

Yes.

So there is an option in Dataset which is called item display function, which is designed in exactly this way. So it takes an object in the first argument, the position in the second, and the whole data set in the third. And what it does is that if you just pass something that doesn't have the function head, it ignores it. A symbol will be ignored so people will not suffer from these things. That's. So you really have to.

Wait, Sorry, what do you mean? Do you mean. What do you mean it's ignored? Like as in you. You have to pass something with head function or it complains or.

Well, in that. In that case, it's a typesetting option. So if you just say item display function goes to framed symbol, it just doesn't do anything. But if it's function of framed hash, then it does something.

I mean, arguably it should maybe complain if you give it arrow framed, but sure, perhaps.

I'm just saying we do that.

Right.

Well, I think provisionally that this hash2 thing is a solution. A nice simple solution.

Gate that behind an option. I'm sorry, like handle a function. We have handler function keys. Yeah, that.

Yeah, we could do something like that if we really had to, but I doubt we'll have to. We are basically at time, although I think we have achieved a fair amount here. Now, I don't know exactly what, you know, we could have transform missing. Well, we, we haven't. We haven't really talked about transform missing because it generically operates on columns and maybe it's the exact same deal. It generically operates on columns and hash one is the column. No, because you're going to need functions that individually that just so you know, transform missing. Oh, boy. I mean, we've got these columns and then we've got, you know, grouped pieces of the columns and so on. I think we need to look at transform missing another time, and that's okay. And I doubt transform missing is the right name. I don't know what any of this is, but we didn't get to it today. What, what were you going to talk about here?

This is. So when you deal with missings structurally. Yeah, you might have, you know, specialized columns like transport missing. But in general, for tabular, we want ways to isolate, take drop elements based on, not just on indices, but on values. For example, a common way to find outliers is to find elements that have. They are more than three standard deviations from the mean. So how do you do that? Here I propose different syntaxes to make sure that we cover our angles. So right now, and I have to do it with insert columns because of this impossibility of dealing with columns, I take a tabular, which is a random thing, plus that number 20. So I have a manual outlier, and I create that and I put in a tabular, which is a column A. It's a single column with one outlier.

I'm not paying attention. I'm sorry, I'm distracted because we ran out of time and I.

And then. You're going to do this next time. It's fine.

Right. Sorry. We made progress. So let's.

Declare victory. Yay. Because we made progress.

Yep, sounds good. Okay, see you soon.

Okay, thanks.

Bye.

Bye. Bye.