Live CEOing Ep 825: Language Design in Wolfram Language [Tabular]

Okay. Hello, everyone. Welcome to another Wolfram language design review. We are continuing the long running series about tabular. Tabular is getting lots of attention as we grind it towards completion. So where do we start today?

So the idea. So while, since I was out this weekend.

We're not going back on what you were out for?

No, no, no, no. It's only going forward.

No, no.

But while I was out, I had made. Did some thinking and I thought that really this thing we are talking about, the whole tabular, the grouping and so on, looks a lot like the work that we did for relation to other tables in relational database, except it's always on yourself. And so I thought, why don't we represent it the same way by using lazy tabulars.

Wait a minute. Didn't we already agree on what we're doing about this? We're not going back on what we went through last time, are we?

We had a proposal last time of having an extra function or that whole tabular symbol, but that wasn't actually doing the grouping. That was only representing the whole tabular. If you open the transform missing, that's the last section.

We really don't have time to go.

Where is the thing that you wrote? Oh, yeah. At the beginning of the page you have this whole tabula or the function in the middle to restructure the thing. This is something that came out in the last meeting.

Okay, all right. So that's where we were last time.

Exactly. So this is a symbolic way to represent the whole tabular. Right. And it's completely.

Which can have various kinds of grouping specifications and so on. It just represents a. This thing in the end is going to be some extraction from the whole tabular. Am I making sense?

Yes. But you asked me, you had one suggestion, kind of overarching suggestion, which is, why don't we try to make things explicit? Because we can represent them symbolically. So what you have here is an attempt to work on that suggestion and make things explicit.

Okay, so let's look at this then.

Okay, so I can maybe briefly explain. This is a motivation that starts from two different tabulars. But I think it explains very clearly what's going on. So imagine that we have two tabulars. So the first one is cities. You have name, population and country. And country. We also have another tabular with countries. So we can link those in countries. We have name area and we can imagine an inverse link that goes back to the cities tabular.

Now if we go back to. It goes back to the whole tabular or to Some.

Exactly, exactly. Now, if we stay in countries, how does that thing look like? So let's say when we take the first row, we look at cities. Now, cities, effectively equivalent of that select operation there in the cities, tabular.

Let's make the same mistake we made with databases. Please, let's not make the same mistake.

No, no, this is a tabular. This is just an implicit tabular.

Let's not make the same mistake of leaving this unevaluated. Is that what you're proposing?

No, no, no, this is what it is. You can take that code and evaluate it and get out the tabular, but the tabular will always already be there. You don't have to write select. That is the result of that linking. So I'm just writing this so you can imagine what it will look like. So basically, in the cities, there's already a tabular. In row one column cities, there is a tabular. So why this is important because if you take row one column cities, and that is a tabular, you can do another part operation, and all population is a tabular column. That's valid. That was a tabular, so that's tabular column. Okay. Now, we were looking for a way to express tabular columns in transform columns. That means that if we have a way which is still undefined in this example, initial example, if we have a way to create this linking, then if you want to do the total population of a country, we can just say do total of column cities, all population, because that's a tabular and all population is a column and total is a number. And this tabular is duplicated for each row. So we don't need. Actually, with this syntax, you have this tabular accessible naturally at every row.

How? Why? I don't understand. How does it know that that's not in transform columns? How does it know that hash cities. I don't understand.

Wait, wait, wait. This is just, you know, this is motivation. Yeah. Now, how do we do this in a tabular?

Well, I don't even understand it. I mean, I don't understand what this hash cities means. Access on a row by row basis. The column called cities. Correct.

Exactly. And the column called cities, it has a full tabular with all the cities with that country.

So you mean if I looked at it and if I used normal, it will blow up my memory because every single entry of the city's column would be a tabular, Is that correct?

If you want to do normal, yes. If you don't want to do normal, no.

Yeah, you probably have to map Normal. But I mean, this is not anything we haven't done for ages in like built in entities, for example, a bunch of the values of the built in entities are either other entities or other entity classes. And this is exactly the same thing in the tabular language.

It's not the same because.

Why not?

Because in that case, those entities are opaque. They don't evaluate until you ask for entity value.

Yeah, but it's, it's these. These would be lazy tabulars, like the ones.

No, we're not going with that. We're not making that mistake.

Then let's close this and let's move on.

I mean, we're not making the mistake. We made that mistake with database.

It's not a mistake. It's the only way to work with this stuff.

Like, otherwise what we should do is to return a symbolic object. We should not make the thing itself. I mean, maybe what you mean by lazy is what I mean by return a symbolic object.

Yeah, that's exactly what I mean when I say a lazy tabular. It's like the symbolic object that we have to represent out of core tabular. Before you say something like two in memory, it stays unevaluated. It just represents a reference to something.

Well, okay, maybe one question. I don't totally understand what the setup is here, but one question would just be, I mean, maybe we should have a separate head that's like a grouped tabular or something like that that basically just has one extra level of index.

But if you look at what this group tabular does, what it does is basically adds columns. One of the columns is a normal tabular. I mean normal lazy, but normal, not grouped in any way. And the other one is just a rank index.

So to understand when you say lazy, I mean, is the point just being that the tabulars are all kind of like linked together because they're all referencing the same data?

No, but this one links together, this one links to itself. It's really like implicit class. It's telling you what to do when you want to evaluate it, but you don't do it right away.

So the problem here is you ask Stephen, how do we actually create this extra column that we had here between two tabulars? Basically, in the basic case, we want to do that on a single tabular. So this group tabular and syntax open for discussion, will just add the column that you want with this property. So let's say that we have this cities tabular which has name, country, population. And then I say, okay, group this tabular by country this will add the country grouping, which I did explicitly in that example of group tabular. But that's an explicit tabular in the last column. But just to symbolize what it is. And now you can actually query that and for example, to do the, you know, the total population of a country and append that in the city room.

Nobody will ever understand this.

Wait, okay, let me see if I can understand. So just. Right. So is the idea that now once you do group tabular, cities, comma, country, now the country tabular next to Rome, Italy is going to have, you know, Milan and Venice and things like that.

Basically, this is a long tabula. I took three rows, basically as an example. Yeah. And that's going to be every city in Italy. So basically when you talk about Rome, you can also talk about all the cities in Italy, which is in the same, you know, in the same cities. Yeah, it's not a different, a different table, it's the same tabula, it's just grouped. So you're basically. Yeah, yeah, sorry, go.

Just trying to understand though, maybe this is the lazy point. Is the idea then that if I look at the row corresponding to Milan, presumably its country tabular will be identical to the one for Rome. So is the point that those will be exactly that it's going to somehow reference each other or something?

It's the same. It's exactly the same tabular and is available at every row. But the point is that when you are using transform columns, then you naturally have access to a full tabular.

Yeah, totally crazy. It's totally crazy.

This way, you know, you look at the. You look at the tabular, what? You print it out and the column is there. It's not some weird implicit action.

It's there.

You can look at the rank, it has a number there. It's not doing anything weird, it's just adding columns.

If I were to do normal on it would blow up the memory. Why?

No, you'd have to do normal twice for it to blow up the memory.

Then you would blow up the memory.

Well, not really, because it's not huge.

Well, if I were to print it.

Out, the memory is shared, it would blow up probably the front end, but not the memory.

Right, but maybe another question would be. So because we don't have any grouped tabular object, presumably it does actually take up a bunch more memory depending on how the tabular columns, like the types of the tabular columns to have many tabulars. For example, if I had, you know, let's say I have some big, like Categorical. I have some column with a bunch of categorical stuff in it. Right. If I have one tabular, one very long tabular with this categorical thing that's going to be stored very efficiently. If I break that into 500 tabulars each with a categorical thing, then that's going to be much less efficient. Right.

But Christopher, the point is that this can be implemented. And as a matter of fact, I've already implemented something like this for relations in databases. You don't have to put any extra data in it because the only thing you need is metadata to generate this. It's basically just saying. It's just saying this, these columns.

Let's zoom out to the problem we're trying to solve. Okay, Christopher, go on.

Yes, Well, I mean, I was just gonna say.

I mean.

Right. But then it's, it's, it's. I don't know, like with databases that's easier because they're, I feel like they're more intrinsically lazy. But like, if you're actually going to provide a tabular object and the tabular object is going to, you know, like you'd need to have some. A tabular object that's sort of backed in a very different way. Right.

Well, it's not backed. I don't know. And that's why, you know, I think one thing I would really.

It's a self referential data structure, basically.

Yes.

And something we don't have anywhere else. And you know, X equals x plus one is something we've been able to live without for decades or whatever.

Yeah, but this is really a graph, Stephen. It's not nothing else. It's. It's just. It's not really the self referential like X equals x plus one.

Can we just look? Okay, we're going to zoom out a bit, but we're also going to look at this item here. Is this an alternative idea?

No, this is. I misunderstood what Rob. Maybe we look bad at the design of column wise by Rob. This is actually what Rob intended. So you didn't like column wise at the time because I was wrapping column wise around the head and you said if you wrap column, if that's a headdresser, it cannot change the meaning of what a slot is. So Rob's proposal was actually to wrap it around the function. So you are not changing what slot means after the fact that you are actually making this a different thing. That.

Why isn't this exactly what we need? I mean, in other words, I don't know whether the name column wise Is quite right. But this is basically saying this hash A now refers to the whole tabular, right?

Exactly. Yeah.

Okay, now that's what we need here, apart from the grouping stuff. That's exactly, that's.

That's like 10% of what we need. It doesn't.

No, no, we need. No, it's not 10% of what we need. It's. We have a function here that is. This is. This to me is fairly clean. I'm not sure the name is right. But now what we want to do also instead of saying A that represents the whole column, we want A to represent only the thing that is co grouped according to a particular grouping criterion with the current. With the current row. Right. So we've got several things we want to do. We want whole column, we want co grouped sub column with different co grouping criteria. And we want index within the whole column and index within the subgrouped within the co grouped column sub column. Is that correct? Those are the things we need, right? Yeah.

Okay.

Okay. So that's what we need. We're just clear that that's what we need. So now let's go and see how we can take this design and get those things. Am I making sense? Yes.

So.

This by default refers to the whole of the tabular. Now let's imagine that I don't know how to do the index yet. Okay, but let's imagine that we could have a column wise of function comma grouping and that that grouping. Basically says this is the co grouping criterion to use to do these computations. Am I making sense? Do you understand what I'm saying?

Yeah. This is the first design we had windowed instead of column wise. But it's the same syntax.

Yeah.

Okay. What's wrong with this?

Understand and go ahead.

No, no, I'm saying we propose something very similar, so we don't dislike it.

I don't like the fact that it's not clear what A means and it relies on implicitly stability of ABS in there.

Does it?

Yeah, well, A is a vector.

Yeah.

So mean A is a scalar. And then you have a vector minus a scalar, which is a vector.

But why does it need to be a vector?

Well, because it's in column wise. So A is a column is a full column. Otherwise what's the point? This is actually returning what we were doing.

No, no, no, no, no. I thought we were doing. Why are we doing it this way? That this, this is why. Isn't that the same?

Is that the whole point of column wise?

Yeah. Okay, okay, okay. It's in the. Okay, so that means that this is being given a full column as opposed to being given a row by row type thing. Is that correct?

Exactly.

I mean the other way you can think about it. Yeah.

Inside column one hash is the same as reduced tabular instead of transform columns.

I'm sorry, what are you saying? What?

That inside column wise hash is the same as reduced tabular instead of transform columns.

So why is it that you're not just using reduced tabular for doing some columns and transform columns for other columns? Why aren't you just using those two?

Because reduce tabular is to reduce the tabular. It will put the result, it destroys the rows. It puts everything in a single row or depending on grouping in n groupings row.

Okay, but then maybe there are two different functions here.

Well, one other just random comment would be like if you do this transform columns example, I think probably the easier way to write that most of the time would be just by extracting the columns from tab like that Transform columns call should be the same as abs of tab part all comma a minus mean of tab part all comma A.

Correct. Correct.

Right. That's because there's no loop in here.

This will redo mean like every time. You cannot really compile that. And also that function is not independent of the symbol that you put. It's statistical choices. Of course.

Yeah, but Julio, the problem with this example that you put in here is.

That.

What is the current row here? How do I know which one is the current row?

Oh, you don't have the control in column wide, I think Rob.

Yeah, yeah, you can't do that.

Oh no, Rob is not here. But yeah, you don't have the current row here.

There's no hash. Anything binds to the current row.

Yes, yeah, right, but then it's the.

Same as reduced tabular, right? Because then all the, you know, the like. Is this function going to be called quote unquote once for each row or just once?

I mean, you could call it once for each row, but it's the same as calling it once only once and duplicating the result.

But then are we going to duplicate.

Oh no, sorry, sorry, sorry, sorry. No, no, I'm sorry. This function produces a full column, so you have to splice it across the tabular.

Yeah, okay, listen, listen, look, this is not working, okay? To me, transform columns is really a map rows piece of functionality.

Kinda.

Yeah, but it adds things. It has the append thing, but yeah, it's map rows.

It's. It's maps across Rows. Yeah, and it just does that. It doesn't eat whole columns. It doesn't do any of that stuff.

Exactly. And that's why I'm proposing.

So that's, so that we should have a different function that is called, you know, let's say. I don't know, I mean these are not the right names but transform tabular, replace columns or something. Tabular, you know.

Yeah, but we had column map before. But this doesn't solve our problem. Our problem is how we can have access to both things in the same function one at a time.

Why do we need that? Why do we need that?

Because it's useful. Often it's useful to interpolate between those two things. Like for example, to compute some kind of rank within a grouping or you know, rank within the whole column or.

So what you're saying is. Hold on. What you're saying is it's like map indexed, which is taking two variables. Right. It's as if we say. Well okay, the. It's. It's also a little bit like splice, as you were just saying. As in. Maybe, maybe not. Maybe I'm wrong about this.

What I was trying to say is that the, the proposal of having this, this serpent, lazy tabular put in makes. We don't need to change anything about this map rows behavior. Everything is still mapping on rows. Some rows just have extra information that we've added and then.

It's horrible. I think it's horrible. I think it's like some kind of, you know, primitive SQL database hackery.

No, it's not primitive SQL hacker. It's something that people are already very familiar with.

That's your spin on primitive SQL.

What I'm saying is that we've been doing this kind of stuff, entities for a very long time. Like people are okay with the fact that some of the values in returned by entity values are more entities or more entity classes.

That's fine, but that's a very different thing.

It's not if you look at actually open up the section that is closed because there isn't a working example built with entities.

But also just understand something. The relationship between these two things is the first, the top one was just basically, you know, you essentially do the grouping and then just bake it in as a call in a column as opposed to, you know, in theory sort of producing it on the fly in the column wise example. Is that basically right?

Yeah, yeah. I mean you're still producing it on the fly because that thing is lazy. But this is something that you look at Right, So Stephen, if you go down, there are a couple of examples. There are examples which actually have the entity classes returned and then you can do stuff with them. Unless. Yeah, those examples. Group, tenancy class, country name, rank.

I don't understand any of this.

So this is just giving you the row, index name and rank. The second one, if you open it.

Never mind. I'm joking.

You open the second one, name and grouping. It takes every country and gives you that country within its continent. So that sorted entity class of entity class is going to give you all the other countries in Asia and so on and so forth. And then there is another one with elements and it gives you the percent rank within the period of the half life of the given element. You can do all sorts of computation with quantiles and so on.

We don't. Grouped entity class. Is that a thing?

No, I just wrote the code to define it. It's five lines of code to define.

It's right above, isn't it?

Yeah, but it basically just adds a grouping and a rank and the grouping is an implicit entity class and the equivalent of an implicit entity class is an implicit tabular.

I hate that idea. That's exactly what we did wrong with database stuff, which we now have to back out of. Right. Returning something which is sort of a promise to evaluate as opposed to just. Here's the thing, it's a symbolic object. It represents the answer. See what I'm saying?

I. I think that the idea of creating intermediate columns seems powerful to me. I mean, we would have to remove them at the end. So I'm, I'm. I placed something in the chat of how we could do the mean deviation operation in transform columns using this idea. Something much simpler.

Oh, can you do transform columns or you kind of chain them so that the later ones reference the first one exactly.

It's like you compute the mean first, you put it in a new column, then you refer to that in a second in the column that you are really going to create, and then you have to remove the intermediate column you created.

Is that nothing Syntax? Did you just make that up or is that a real thing right now?

I think this is something. I'm not sure it works, but this is something we wanted to have, I think, as a way of removing columns.

Good idea. Okay, let's look at this. I worry about the semantics of this triangular thing, that you'll get yourself in some tangle where, you know, you're. You're in a loop.

Yes, this is.

Yeah, yeah, go ahead.

I was just going to mention that we have discussed this sort of thing in width. Like having different levels in width?

Yes.

You mean like the undocumented with syntax, right?

Which undocumented with syntax.

The verargs with.

Oh, yes, which we still.

I really want. But whatever.

Yeah, we. We should be talking about it. And you know what? We have enough new mechanisms for user assistance and grouping indicators and so on. I think we can do it, particularly with this new idea about grouping parentheses and so on. Gray parentheses appearing to indicate grouping. We should bring that one back to lcc. Somebody do that, please.

Yeah, I'm taking the note.

Okay.

Italy pto.

But I. Yeah, this time due course.

So this transform columns is exactly the same as if you did like a fold over that second argument. Is that right, Joseph?

I think so.

Like if you transform columns, you know, if you had three calls, nested calls to transform columns.

Okay, but the objection to all of this is that you can't in one of these. Okay, let's just try one. One possibility. Okay. And transform columns. Maybe this isn't right, but. But. Okay, let's imagine that hash A is a column name. And that refers to. I mean, written out in full form. That refers to hash. Really, what we've got is two levels of function here. That was what I was talking about before, which you guys didn't. Didn't. The arrow. Right, this two level thing. A goes to G, goes to F. See what I'm saying?

Yep.

Where was I? Right, Where again, I mean, I don't understand what was wrong with that idea. In other words, that what's initially presented in transformed columns is an association, right? That you can get whatever you want from that association. And then this function here applies to that association. Okay, so this is the pre function. There's a. Okay, here will be a pre function. Do you understand what I'm saying? Carlo, do you understand what I'm saying?

Yeah, it's the thing we said that we could say that the default pre function is part of.

Well, it's not quite part. It's hash, tabular of hash, index.

What's the difference?

Because it's not one and two. There are other things.

No, Stephen, you use the pre to return an association with sensible names. But pre.

No, no, no, no, no, no, no, no, no. Pre is a thing that is applied to. Okay, what happens here is there is an association provided which contains tabular. Which is the whole tabular.

Yeah.

Okay. Index, which is the index number of that row. Okay, then it could have other. Other things here.

No, there's nothing else that's all you.

There is like what? Well, what about grouping stuff?

The groupings? That's what you use pre for.

How do you use pre? This is what pre. Look, pre.

So you pre would be something that, take the way I see it, is like you use pre to construct like.

This is what pre is being given. This is what pre has been given. Pre has given that as an argument. That's it. Fine. Right, so how do you do grouping?

It's not straightforward, but.

Okay, then we're off the. Then we're out of the game.

No, because that's not something you write. You have a function that's called like make groupings and takes some arguments, which are whatever you want, and then returns the correct association with the groupings. Done. It's not something that people are going to write because it's complicated to write. And it's also probably not going to be implemented that way anyway because you don't want to do it row by row. You do it once.

Okay, and what do you return to represent the group thing? What do you return to represent the grouped thing? The thing with the groupings having been done.

Okay, this. I don't think that this function really, really returns. It's a symbolic function.

That's where you're wrong. That's where you're wrong. That's the mistake we made with databases, is to have something which implicitly did something without returning a symbolic object that represents the result.

If you make pre return, then you're making this whole algorithm order N squared instead of order.

No, no, no, no, no, no, no. I'm not listening to the implementation detail question.

Yeah, but that's not.

I'm not sure this is the right design. But that, that's not where we're going first. Okay, let's. Let's look at what you're claiming about grouped tabulars. Okay. Is there a representation, Is there a kind of a tabular where. I. I'm not seeing it. I'm just not seeing it. Where. Because you might want different groupings for different operations in the tabular, might you not?

A.B.

Absolutely. Absolutely. And this is how you do it explicitly. Like you call multiple times group tabular with. You give it different prefixes so that you can use extended key to access these different things.

No, no, it's completely crazy. I mean, if, if that's what people have to do, nobody will ever use it. They will have died on the road before they reach.

With this arrow thing that you're proposing.

There'S no, I'm not necessarily proposing that. Wait a minute. Wait, stop, stop. I'm not necessarily proposing that, but I am trying to get us out of, you know, this, this horrible mess with extended keys and 17 copies of the tabular and so on. That's not going to work. Let's understand what the. Let's zoom out to what the fundamental thing we're trying to do is. Okay, let's list again what we're trying to do. We are trying to have these four kinds of items that you can access when you're. When you are creating. Okay, this is when you are creating a new element in a particular column on a particular row, you want to access those four things. Is that correct?

Yes.

Okay, so let's figure out what the cleanest way to do that. Right?

But can I make. Can I make a point about this whole idea of having a whole materialized tabular? I think what the way, the right way to think about this is a different type of columns. These columns don't actually have tabulars materialized in them. They're reference columns, outer reference.

I'm not buying it. I'm just not buying it. Reference columns already, even by their name. You lost the game there. It's going to be incredibly obscure. Let's see whether we can do this directly. Okay, so let's say when you're creating a new element in a particular column on a particular row, what do you need, right? You need the whole row, the whole column, a co grouped sub column. These things, right? Those are the things we need. Correct? Is that correct? Okay, so to me, the obvious way to do this is to have some syntax where the. Where what you do is effectively present. Well, let's see. It's not the whole column, it's a whole column, right? The whole row. You are on. Right? A whole column with a specified name, correct?

Yeah. Or, you know, you could call it a transposed tabular. Really? The whole transposed tabular.

Well, that's why I was suggesting what I was suggesting before. Okay, so look, all of these things here, okay? Is it true that all these things are accessible fairly easily from a whole tabular?

No.

Okay. And the reason for that is you don't know what is co grouped with you, correct?

Yes. And where you are within that group.

Okay, fine, fine. Okay, so this is what we need to do. This is. This is what we need to be able to access. We need a way to specify these things. A co grouped sub column, where you give the name of the column and the grouping columns and the sorting columns. So what do you mean and sorting?

Well, yeah, you want that sorted most of the times. You don't want it in whatever order.

Okay, all right. Okay. So that's what we have to do. Okay, so what we've got right now is pretty nice syntax for the whole row you are on, which is otherwise known as just hash, Right. That is the whole row you're on. Am I correct?

Yes.

And so its elements, its column elements are a, etc. Correct?

Yeah.

Okay, so we don't have a way to specify another whole column with a specified name. Were we to have a way, which we discussed last time. I'm sorry, what?

I don't understand what you just said. We don't have a way to. Oh, yeah, okay, fine.

We don't have a way within transformed columns to specify a whole column with a specified name.

Correct.

If we had a. If we had the whole tabular, we could just say tabular of all comma name. Correct?

Yes.

And that would return a tabular column.

What?

That. What would that return?

That would return a tabular column.

Yes.

Okay. Okay. And we don't have a way to specify what. What will be our way within. Within the tab. If we had the whole tabular, how would we specify. We wouldn't be able to specify this co grouped thing because we don't have a way to.

We are of the way. That would be select of tab comma predicate. That is the grouping.

No, no, no.

Sort by around it. I'm saying what's the way to do it without not within.

That doesn't give us a way to do it. This here. If we know the index. Okay. We. We could also want. We might also want the index in the tabular of the row you are on. Yeah, okay. Okay. One of the things that's tricky is that many of these sub pieces are themselves parameterized. It's not like we can just give a single association that contains each of these sub pieces. Am I making sense?

Yes. And as a matter of fact, most of the times you're not going to do it at the same time on different parameters.

Fortunately. Sorry, say that again. What?

I said that most of the time you're not going to want to have access to different parameterized versions. Like you're going to operate on the same thing. Like the parameterization is going to be.

Universal.

For that function. You're not going to mix grouped by sex and grouped by class at the same time.

Why?

Because it's not very common. We looked at a bunch of examples this came out of how to implement things in transform missings and Most of the times. Really. All of the examples we looked at, you were looking at one thing at a time.

I, I, I don't think we should burn that in. Doesn't seem right.

Yeah, but I mean, it's not burned.

So let's not do it. Let's not make the mistake.

It's not burned in because you can compute the separate.

Guys, guys, look, this is our objective. This is what we have to solve, right? That we want access to all of that information. Now, is that also true for transform missing, that it's the same information that we need?

Yes, in practice. But the typical syntax will start from the opposite way around, because the simple syntax for transform missing is just look at the column and do the same thing at every row. So it's going to be basically a different set of shortcuts.

All right, let's look again at what we have here and what the minimum thing that we can do to support this without any virtual implicit referenced anythings. Okay. Because what this is doing in the end is, I mean, you know, this knows that it's operating on a certain tabular now in the syntax. Okay, who has an idea about how to do this with no implicit anything? I mean, syntactically. Go ahead.

Want a different variable or different description for the column, depending on how you want to access it. In a nutshell, whether you want to go element by element or look at the whole thing and take the mean of the column?

I think so.

Right.

Yeah.

So if hash A is taken for, you know, looking element, element, and we just maybe need a different way to denote the column A that you're going to be operating as a whole.

Yes. Right, I agree. So, I mean, column wise is potentially such a way.

Right.

But we don't believe we can do this. Where.

I think Rob's idea was that column.

Wise.

Wraps the whole expression. And then if inside we want something to be addressed element by element, we would have to write element wise. But otherwise everything is considered to be a column.

Yeah, but element wise just helps with the threading. It doesn't give you access to the current row. It just makes something like, you know, subtract is already threading.

Yes.

So you don't, you don't need any special thing. Let's say that you had greater or less. You might want to use element wise, but that still doesn't tell you where you are, just makes it work.

Okay, what if I don't know what the right name is? But essentially column wise of hash A, what is, what is the actual way of Extracting that column column wise of hash A would turn into tab of all comma A, which returns a tabular column like that. Right?

Correct.

Yes. So, And why exactly isn't it the case that we just say something like. Which is what I have proposed before, whole column of a. Why isn't it that we do exactly what I was proposing here? Well, here I'd suggested whole tabular of a. But. But we could say, let's imagine that we did this down. We say whole column of A. Right? That refers to the whole column. And then whole column of a comma, you know, groupings, whatever it is, group one comma sort one comma, etc. And that this returns a tabular column. In other words, when this is encountered in. When you see whole column of A in transform columns, you see whole column of A. Right. Then it just gives you. So. So you could have, you know, hash A, which is the single element minus mean of whole column of quotes A. Right. And that would be the result of a. That could be. We could even make. If we really wanted to be ornate, we could do this right? In transform columns.

Yes.

What I'm saying. Okay, so what do we fail to get here? Very little we don't have. So we also need in here, we could have a thing called whole column index. Then I think we have everything. Does this make sense?

I think it. You know, as soon as you take any example that is not trivial like this one, it's super verbose.

Oh really? Why?

Well, scroll up, scroll up. There was an example for like a week from like a week ago. Okay, where is like syntax comparison was the header. It's further up. Syntax comparison.

Okay, I need to see a real example here.

Yeah, scroll down. It was the tabular slot thing is. Exactly.

Let's take a look at that. Let's take a look within detail. This is going to be the winning result. This is going to be the winning answer. But exactly how we deal with the syntax, I don't yet know. This is going to be very clean and by the way, you know, minimizing. Okay, all right, let's take a look at this example. Okay, I don't need all of this nonsense here. Okay, so what. Let's take a look at what this is.

I mean, that would be the first one. Instead of tabular slot will be whole column.

What's transposed? There's no transpose anything.

Yeah, there's no transpose. That's whole column.

So all that nonsense is gone. Okay, now we go survived. Okay, now what are we Doing here? What is this? I don't know what any of this nonsense is. Okay, so tell me what this nonsense is.

So, okay, so you. You have to take the subpart of the. Of the existing column of the existing survived column up to your row index and then you have to divide it by the total of the survived. Because what you want to know is basically how many survives.

It's a quantile kinda.

Yes, I think it survived.

Okay.

It's a cumulative thing within your class.

Class and sex order by age.

Okay. It's a very elaborate thing you're doing here. I'm more interested in getting the simple cases to work in a simple way. Let's make sure this case works. But I'm sure it will.

Yeah, but you're optimizing for things where.

I'm optimizing are even more complicated than this. I'm optimizing for things people will actually do, not things that people will anyway be writing out half a page of code for.

But Stephen, the point is that this thing that you copied is a typical case in which you have to reference three times the same thing, and then you have to repeat age, class, sex, class, and age in all of them. This is what I don't like about okay, and you're optimizing for more complicated cases where you have to. You group one thing by sex.

What are you talking about? Okay, let's be clear what you're talking about.

What I'm talking about is I think.

You'Re saying that you pre group the things and then you do operations on them. That's what you're saying, correct?

What I'm saying is that it's way more common to write a function that operates on the same grouping throughout and that this syntax, this type of syntax.

Okay, then you know what you need to do? You know what you need to do? You need to rewrite the tabular with those groupings already done and then do your operation. If that's what you want to do. If you want to operate with those groupings, then make something where those groupings have been done or break it into multiple tabular.

What do you mean where the groupings have been done?

That's what I was proposing. That was that.

That's.

No, but not, not with the thing with it implicitly inside the existing tabular. That's not going to work.

Then what are you going to do? Are you going to blow up the memory just because.

No, I'm going to. I'm going to make multiple tabulars which presumably don't Take any more memory than the individual tabulars.

No, they do. They do, they do. For the reason we talked about earlier.

But we're not going to have this implicit thing because nobody's ever going to understand it. We've already discovered that in multiple years. Nobody's understood things we've done before that work in this implicit way. So let's understand what it is that you are actually trying to do.

Understand implicit entity class, like things that say city, country goes to Italy. That gives you all the cities in Italy.

That part people understand, that part people understand.

That's an implicit lazy thing. It's a reference to a group of things given by a criterion. What else is there?

Carlo?

Okay, we need to move on here. Go ahead. What were you going to say?

I just, you know, just looking at this example here with the survived. So you do grouping, right, but that's kind of like the reduction.

So it's not because you take. Take up to the raw index. It's like a cumulative thing. So it's, it's not reduced because everyone is different.

Okay, we spent 53 minutes on this. We already spent an hour on it last time. I don't think we're going in the right direction. Okay, let's try and move through going in this direction, figuring out how to make this work. Let's not go backwards onto something that is not going to work.

That's. That's.

Let's try and figure out how this. We're going to make this work.

Show me that example with your new syntax.

I haven't a clue. You're going to help me figure that out. We're going to make this work. We're not doing the. We're not doing the simplicit thing. So let's figure out how we make this work. And I'm not going to spend more time on it today or at any other time. We need to make this work.

Yeah, I can type it out for you, but it's going to just look as ugly with whole content.

Well, then we have to figure out how to make a simpler elided syntax for it, don't we? But we're going to make this work.

There's no way to make an enlightened syntax if you insist on repeating the same information for every slot when.

When.

That's a silly thing to say. You know perfectly well that there are plenty of scoping constructs we can imagine where you don't have to repeat the information.

Yes, so let's make a scoping construct instead of putting everything in the slot.

That'S fine, but let's figure out how to do that with this as the thing that you do without scoping.

Okay. I mean, I agree that you should maybe move. Move on. Let's try to do one round of implementation and then come back because.

So. So here's what I suggest, right, Roger, Maybe you need to jump in as well. Here's what I suggest. I think this is reasonably clean. Okay. Now you're saying if there is a grouping construct that's being used, you might want to have the same grouping construct all over the place. In fact, you might want to have it so all over the place that it's in everything you're doing inside a single transform columns. Am I right?

At least for one. One function.

Yes.

Well, maybe everything.

Yeah.

But we need some mechanism to solve.

One thing that we've noticed. So categorical variables, like sex and class is a way of selecting subsets. And so we've noticed that, like, for visualization, you often want to pick out one of those subsets and think of it as a data set, like we did with lists before. And we experimented a little bit with a syntax to do that, you know, so you have an infix syntax to pick, you know, the grouping variables that you want. We didn't have sorting because that wasn't particularly important for visualization, but it can be important for computation. And so I've noticed that these other ones, you know, data frames and so on, they have something special for groupings. So grouping is a complicated thing, but relating to what we used to have, you know, I mean, with lists and stuff, we always think of them as multiple lists, you know, like subsets, basically. And it's just that the whole tabular, include them all, and you often want to sort of home in on one. So I think it will be. I want to. I want to sort of work on it separately and see if I can come up with something, because I do think it's pretty universal that. What is to have grouping in lots of processing functions.

Fine, so let's do something where we do that, right, but where we have this concept of being able to pick up the whole column or whatever it is. I don't know. We need the right name for this, right? A way of referring to the whole. I mean, to me this is fairly clean. Do you agree? Yes. No. Maybe.

I don't know.

Yeah, but it's a trivial example. It's clear.

I don't give a damn. I don't give a damn. I'm sorry, I don't give a damn. Let's move on.

Okay, so this is next. Yeah, scroll up.

I don't have a lot of time here. So we learned most of our time on this, on this discussion.

It's an important discussion.

I think it is an important discussion, but we're not going to make the same mistake we made with databases. We're not doing that. We spent tens of hours talking about that and ended up with a design people can't use. We're not making that mistake again.

Okay, maybe let's look at this.

That mistake happened because I got kind of browbeaten into some kind of SQL like structure. That is not a good idea. So let's not make that mistake again. Right. You always have to have a design that goes forwards, not implicitly looks backwards. It has something which generates a symbolic object that is then processed. Not something where there is an implicit overhung thing that you're looking. You know, Am I making sense? Does that make sense? Roger, do you understand what I'm saying there?

Yeah.

Okay, no overhung, lazy reference anythings. Doesn't work. Okay, what are we looking at next? Okay, maybe we should wrap up in a minute here.

Well, I have a couple of concepts I wanted to bring up. I think if you can go through this, then we can free up time for other stuff.

Okay, fine.

Okay, so this is one example practical why we need this column wise thingy. But anyway, this highlights other shortcomings that we have, maybe in the language, maybe for tabular that I wanted to look at. So I call this predicates and indices. Because the point is, in tabulars especially, you have basically two ways to extract things. By part normal, part spec and by putting conditions on things. And they are both very common. And we need to make sure that the bridge is stable, wide open, easy to traverse. Now I have some examples here that are highlighting the syntax shortcomings that we have here. So this is for something that came out immediately for handling outliers. A typical way of handling outliers is that you look at how many standard deviation is some value from the mean. And to do that is quite easy. So right now this thing we don't have column wise or something like that. So I did that with insert column and I added the Z score column which is just abs of standardized of a column. So it's easy. And then how do you get outliers then? Okay, you can get positions with row indices. And I want to fall back row indices in the discussion because that's kind of a way to get to go from predicate to part. So it's like you select from the new Alphabet and you get this. Now with part, it's easy to get the appliers. How do you drop the outliers? That's the first hole in the syntax. So delete is parent to. Sorry is relative to extract is multiple things in a ragged way. We don't have something like a rectangular remove stuff that is equivalent to part that you can use once you have that list of rows to drop it. So that's. That's one. One hole that we have. Okay.

Actually, I think you guys need. Go ahead, Christopher. Yes.

Well, maybe one.

One question. Right. Which might relate to, you know.

Yeah.

Solution would be. So in this case, row indices makes a lot of sense because you're not expecting a whole lot of outliers. But I'd imagine there's also a lot of cases where you. You're going to get, you know, half of your 100 million row tabular. Yes, Christopher, you know, and so you want.

Correct.

Okay, okay, fine.

Let's keep looking. So the other thing is that you don't go to part syntax, but I think we should cover both cases. Yeah. Because part is useful. If you need to kind of create or remove. That's still useful. Then you can use select directly on the new tablet. Also here, part has the advantage that you can use it on the. On the unmodified tabular. Yeah. So on the original one, if you want to operate on the tableau with the extra column, you can use select. What I have here is directly get outliers. So select z score greater than 3. This will get you the outliers again. You can do the opposite with not. But I think that's a quite common operation to have to compose not to your function every time. I call it drop by here. It could be called many things, but this is a quite common operation to do. I think it calls for some direct symbol drop by tabular z score greater than 3, which is the opposite of select.

Right. So it's literally the same thing as select, except that it does not on the test.

Exactly. That's kind of avoiding you to put not in a lot of places. And the usefulness. If you scroll down a bit, Steven, you see, if we move to missing handling. How do you remove rows that have at least one missing drop by tabular. Any true missing queue. Super terse, super fast. How do you remove tabular that have every element missing in the row all true missing queue? You can add not, of course, with the Composition. But that's some extra mental gymnastic. We're asking people. And we typically tend to give verbs that are, you know, they put the action in the verb, not in some modifier. Inside.

What's the relationship between drop by and pick?

Well, it's still taking a test. Not a, you know, not a parallel list, but actually that was the thing I was going to say. Which is. Which is that several of these things. It seems like it would also be good to have a parallel way of doing this with like, I don't know, with like parallel lists, like. Yeah, right. With like bit vectors or something like that.

Possibly. Literally a bit vector data structure. Yeah.

I mean we be list of, you know, list of booleans or, you know. Right.

I think drop by seems like an interesting function in general. Right. I mean, there's no drop by if I'm not.

So maybe one question would be what the. I don't know, drop by, delete cases.

Hold on. Why isn't it. It's basically delete select.

Exactly, exactly. It's. It's a. Yeah.

That seems like a useful operation. Not sure what its name should be, but it seems like a useful operation.

Yeah, I mean, and it's true that we have, you know, the metrics of direct inverse. It has some holes like, you know, you have positions and we have delete. We have part and we don't have the other one. And here cases. Delete cases, but select and not the opposing one.

Reject.

Sorry, Reject. Reject. Yes.

I was looking for the opposite word of select and it's a reject. So you want to reject by the. By the criteria. Yes.

It's not a terrible name even.

Yeah, I would say drop by. I feel like both has an unclear relationship to drop and also accept all the buy functions.

No, I agree. I mean, select is also select buy. Yeah, it's just implicit buy. But yeah, I agree that drop is. I mean drop is like the opposite of.

Take the sequential thing.

Yeah.

No, reject is the appropriate. I mean, it's a little weird, but it's not. Okay. In any case, I think we should wrap up here.

Maybe one last thing, if you can approve. A syntax I think is a very simple one, I think uncontentious. Delete duplicates and Delete duplicates by the last one in the. In the. In the thingy. If you scroll down to the bottom here, I think Delete duplicates is uncontentious. Delete duplicates by. You know that it works on Metabola, probably. Also.

I don't understand. Delete duplicates by is a function that. I mean, if this.

Delete duplicates only one. One row for the same with the.

Same value of sex, that will just work. Work, right? I mean, yeah, yeah, yeah.

We just need to hook it up. But just. I mean, just before we do the work, I just want to make sure that this is fine.

Probably. Okay, but this is a bit.

Yeah, it's a bit strange, but this is very nice to just shorthand.

I don't think you can mean anything.

Okay, okay.

Well, you know, there's actually several syntaxes like that that we have that are supported inside of data set but not supported, you know, elsewhere. It's like, for example, I think group by and also sort by inside a data set. You can give them just a string key, but otherwise you can't.

We might want to canonicalize that then as a LCC discussion, maybe. Okay. Yeah. This was the bulk of what I had here, so if you want to perhaps even.

Okay, Okay. Okay, sounds good. All right, thanks, guys. See ya.