Live CEOing Ep 798: Language Design in the Wolfram Language [Tabular]

Hello, everyone. Welcome to another Walton language design review. We are continuing to look at our tabular capabilities planned for version 14.1. All right, do we need to look at the recap or are we digging right in here?

No, I think we can. Yeah, please.

My question, column wise combine, should it be columnized join, or does that not make sense? I think columnized combined makes more sense. I don't consider it a pure join operation because it has all of this combination. You know, it's sort of conflating things by columns.

Right.

So. So can we quickly re page in exactly what column wise combined looks like these days so that we can better discuss that question?

Have you not been involved in these? You were involved in these, weren't you?

I wasn't involved, but I. I just came back from. So A, just as you don't necessarily have perfect recall, I don't necessarily have perfect recall. And B, I just came back from 10 days of PTO yesterday. So.

Yeah, yeah.

All right. So we said. Yeah.

So I have a suggestion for the. For the syntax, a change in the syntax, which is to get rid of the same test function and instead have that one Julio has under the capital or is just have arrow goes to function. So instead of matching on columns, you're basically matching on this function is the idea.

I see.

So what you would write in that would be, you know what it would be like row T1 of key 1 equals. Or like.

Right.

You'd say like hash 1 of key 1 equals hash 2 of key 2 or something.

Exactly, exactly.

Yeah. We have an explicit example that can also work.

I don't understand. What are the arguments given to this function?

Okay, let's look at the explicit thing. It's going to be easy. Here we have column wise combined.

Let's tell it I. It's a good exercise. Let's tell it I. How works?

Okay. Okay, so here we have column wise combine with alpha, beta, key or he.

No, no, let me try explaining it. Okay. Column wise combine combines tabulars or other similar objects, kind of knitting them together, merging them on repeated columns. So if one tabular has a column called A, another one has a column called A, another one has a column called A. What you're doing is you're creating a tabular in which there is a single column A that has the content from all of those tabulars. By default, it does this in an outer way so that if there's an A which has one of The A's has a 3 in it, and the other ones don't have threes in that, in the corresponding position there, the places where you would need to get data from the other tabulars are filled in as missing. That is the default activity. That is the default action. Yeah, with me so far, itai.

Yep.

Okay, so then there's the question of. So now you have essentially row select. Well, so first of all is what are you joining on? What are you combining across? And that's what this arrow key thing is doing here, is saying this is. This is what. These are the columns that will be used to do this combining knitting together of tabulars. The next issue is that sometimes the columns that you are not knitting together, you want to may have. Correspond. May have the same names in different tabulars. And you want to use extended keys to represent those different. Those same name things. And that's achieved by putting this, the collection of tabulars not in a list, but an association. Am I correct so far?

Yes. Yes. Or a list of rules. In the more general thing, if you want to repeat things, but concretely in association. Yes.

Okay. And that's pretty much the design now, I think we've discussed, which I.

Well, there's a richer thing here that is, you might want to like here. This is a super compact, simple design where key is the column you want to match on across multiple tables. But then in the more general case, you might want to say that I want to merge what is in key one in table one has to be the same as what is in key two in table two. Yeah. So this is the richer one for which we have same test.

Okay. So just to make sure that I'm on the same page. So if I look at that output, then the reason there are two rows with fives is that every input table had two rows with fives or correct?

No, it's because the multiplicity of rows that have the column B as 5 is 2. Because you see, everything is matching except the last column, E, which is five or six. Oh, no, actually also key A is one or two. Yeah.

My claim is that that output below there is not correct.

We're going to see that in a second. But to answer Itai's question specifically, yes. I mean, the point is you are creating all the possible combination of rows with a very large outer product. And then you're saying, okay, my condition is that I would consider two rows the same and merge them. If in this case the first table and the second table have the same value for B, in the first case it's five. And then what Stephen Was saying, we have to decide what do I do for all the other values. In this case we keep everything. So we create two copies, one with the value of five.

And so basically we would conclude. So I would conclude that A had one, row five, B, or sorry, Alpha had one, row five, beta had one, row five, Kai had two row fives.

And. No, it doesn't matter in that case. Those are the same rows. Yeah, in this case it doesn't really matter what, what, what key has for it could also not have the. Right, because I'm not.

It's basically you get the cross product of all the things that have a 5 in column B. Right, Right.

But table chi. I mean, when I compare those rows, right, the alpha entries are the same, the beta entries are the same, the chi entries are different. So is that because the chi table had two different fives?

No, that's because the same test is different. So this is forgetting the basics.

Oh, there's the same test.

Not a good example to look at. So the better example to look at is association of tabular section for understanding basic syntax.

Yeah.

This is a quick clarifying question, actually, that we probably went over before, but I forget. So when you have two different. Even in the simpler case. No, same test. When you have key one and key two. So you're joining with keys of different names, what is the name of the resulting column? Like the column where.

No, sorry, in that example, key one, key two is not. Is not. You're not matching two different keys. You are saying that you want to join by a tuple.

And, and, and the answer is. So those keys go inside the association as rule values. Okay. And, and then that those keys, the different columns will, will they? Won't you. You'll still get, you'll get a separate column for each case. So if I write, those will have the prefix names. Because. Because they may be, you know, because. Because if they're different names, you can't decide, you can't make a choice. Right. So you, you need to have both of those names in the result.

I see, so, so just to. Right, so as you were saying, Julia, Right, this is a different syntax, but there is a syntax for the, you know, I've got, I've got one tabular and I've got a column called country, and I have another where it's, you know, country of origin or something. And I'm drawing those. So the resulting joined thing will have both. It'll have redundant columns. Like there'll be one country column and one country of origin column Basically, yes, yes. Okay. But if I, but if all of them were just called country and I did the join, then there won't be redundant country columns.

If you give it, if you give.

It, if you give it in the single rule notation.

Okay.

If you do association goes. So everything given in that single rule will be merged. If you look in this, the associated tabular section that shows all of that, that notation. So it's. Yeah, this one.

Okay.

Okay. But, but now if you gave it as a list of, list of country or something like that, then it would give you. It would not.

So those second two examples show exactly what I'm talking about. So if you just give B, it merges B. If not, it keeps a B with the prefix for each one. Because those might be different names in general.

Yeah, great.

Sorry. I'm sorry, say that again. It merges in this case here, it merges the Bs and leaves a pure B column.

Right. And in this case below, the Bs are associated with the specific tabular. So it keeps the. It. It maintains the prefix associated with that tabular.

It's a local B. Let's say in the second case. Yeah. It's not. Not a global B. It could be any. It could be difficult to be B, C, D. Like you, you are giving.

How does it know it's joining? I mean this, in this case, the right hand side of this rule is the thing that is the, is the thing it's joining across. Is that right?

Right.

That'll be a country or country of origin or.

Yeah, exactly, exactly, exactly. And if you look below, there's the example just, just below your screen shows where I, I'm. I'm choosing to merge A into a single column but keep the B's separate.

Right.

Because you always have to pick one. Well, let me see. Does there always have to be. I guess you could do a list. Yeah, I could do list of A.

B and then that would be. That would merge both A and B, right?

Yes.

Why is there a B on the right hand side here? What is that doing here?

Is that because say, say, say if you wanted to do that in Table 1, didn't have a B, but it had a C and you wanted to do the C instead of the B. You, you could put a C there.

As the merging, as the merging.

It's merging over two columns here. A and B both have to align in this case.

Exactly.

So, so in the very first input in on the screen. So is the reason it knows to basically look at A and B is that it looked at the entries and said, oh, these are the columns I have that are common to all three things.

Yeah.

It determines the common keys and uses that precisely.

That is like arrow automatic, basically. And it's resolved by the intersection of the keys.

Okay, let's go down to the same test here, which I still don't understand.

Exactly. So now here you see that the first example here with the same test, the practical example, actually it has a problem here because we are merging on B and is a global B. But the same test only involves 2 out of the 3 tabulars. So doing the same thing that we do above and having a single B column because it's merged. Yeah. Having a single B column here, we are dropping the B column of chi T. How does it chi. Okay, we are dropping the B column of chi, which is in principle, it could be anything. Yeah. It doesn't have to be equal to the B column of the other two because we're not testing against that. So this is a mistake. This output is wrong. And Rob claims, and I agree that this syntax allows these wrong outputs.

Okay, so let's not do it. So what's this exactly?

The proposal is to have the function.

There, instead of saying merge on B, scroll to the bottom. So you see the explicit example that this column wise combined the last one without output.

Yeah.

Okay, so you say this is the same test incorporated in the fun, instead of the column, like an implicit column definition.

I understand this. So this is saying. You've got these two tabulars, you've got a bunch of different rows in the two tabulars, and you're basically looking at the outer product of all of those rows. Yep. Pairwise out of product of all those rows and the cross. Product.

Cross. Yeah.

Yes. And this is, this is your margin criteria. That makes sense.

Yeah, yeah. So this can be used in place of the column.

Yeah.

And this doesn't support the same problem because in this case you're going to get as an output the same thing that you have above.

By the way, that could be hash alpha. Right. Not hash one. It could be hash alpha.

No, no, no, because this gets three tabulars as three tabular rules.

I know, but that hash one could be a hash alpha.

Yes, it could be. It could be hash alpha. That's, that's, that's right. We'd have to give it the association. But yes, that would, that would, that. That's a possibility as well.

We need to decide. I mean, this is a design thing that can resolve right now, I think if you give an association in. In the. In in the left hand side, it can be hash alpha. If you give a list of rules or simply a list of tabulars, then is positional. Yeah, right.

I'm not sure alpha would be the right thing here. Sorry.

No, but the issue is if you're given this thing, then what you want it to be the case is that if you say, you know, hash alpha of that ampersand of that thing. Yeah, no, we know what it gives. Okay, you're absolutely correct, but however, this will not work.

That's right.

So I claim this. We should either do this.

You need to apply the function need at.

Oh, yeah, that doesn't help me out, does it?

If you put apply around the function, it would just. It would just work.

I know. If it's an applied, not an applied, because this isn't an apply.

And actually then I, I made a mistake that should have been. No, no, no, no. Sorry, sorry, sorry, sorry.

No.

Yes, I think it is. This is not a sequence of tabulars, it's a list of tabulars.

Yeah, if it's a list.

Yep.

I claim you're going to need something like this or more than that.

Yes.

You're going to need something like, you know, key of B. F B is more complicated. Or even extended key.

Or even extended keys. Exactly right.

But. But I don't think that hash was originally considered to be a list. I thought we had this form where you have a nary function, not a. I know.

Look, the problem, Leonard, is if we have an association here, which will not be uncommon, it's very nice to be able to refer to the elements of the association just as hash alpha and so on. And the only way to do that.

We could be a little inconsistent. I mean, like, the fundamental problem is that we're a little inconsistent about what hash alpha versus hash one means, because hash one lets you index into like the sequence of arguments, but hash alpha lets you index into the first argument. Right, yeah. So I mean, we could obviously go ahead.

I think possible issue to think about it is how. How are you forming the outer product? So basically you have to reform. Does, does an outer over association recreate an association? I'm not sure, I don't. I'm not sure what happens there.

No, it does, it creates an association, I think. But by the way, you know something, if, if I were going to pick a thing that I would like to have syntax for, #sub1 would be a nice thing to have syntax for. Just saying. Not that I can See anything on the keyboard that's available for that?

I mean, you could use double brackets. That would be a radical new syntax.

And you know, you can make those. Those double brackets subscripted without changing the meaning. So.

You even have ligatures for the double brackets. They get smaller when you.

That doesn't ligature. Why doesn't that ligature?

Probably because there's no head.

We should report that. That's a. That's a. That's a bug. We should ligature that. Can somebody report that, please, to User Assist? Okay, but. Okay, yeah, the NRE versus hash alpha, I think it is more consistent to have it be NRA so that you have to use part subscripting in all cases.

I mean, we could also just have it so that when it's. When you give it a list, it. It applies and when you give it an association, it just puts an association of all in the first argument. I mean, it would, you know, it's as inconsistent as the behavior of slot.

Yeah, that's fair.

Also, there is another thing that if you have the names that are already prefixed, it probably looks nicer if you just apply #2, for example, of that prefix name, rather than you would have hash of one comma, that prefix name.

I think that's what Christopher is saying. Yes, yes. Okay, so.

Oh, okay.

I think, right. If it's a list, we do an apply. If it's a. If it's an association, we do function application, so to speak. Does that make sense?

But remember, there's one case which is. Which we allowed is we allowed none goes to something in the association to not prefix one of the tables.

Oh, boy.

Yeah.

Well, maybe we should just index.

Wait, sorry, why is that? Make it harder. I mean, you.

Because then you have a hash, right? None.

Can you not have a hash none?

Yeah, you can have a hash none, but. Oh, except it's.

It's.

I guess it's hash you have to use use slot.

Right, but what happened.

Wait, is that. Is that useful with association to have none like. Or is only useful in the least syntax like, maybe we cannot support it in association.

In association. I don't see how it can possibly work. Because if you want to have two things that aren't labeled, the thing's going to blow itself off.

It only works with one thing labeled. And we discussed that a little bit last week.

Only one thing can be unlabeled.

Why only one thing can be unlabeled?

Well, if you use the association syntax, if you use the rule syntax, like, everything can be unlabeled.

Then if you have colliding keys, you have a problem.

So.

So maybe we'd need a different way of labeling than association. That's. That's entirely possible. Yep. But.

Well, okay, so it sounds to me like. I don't think a thing with a none in the association is going to be a winner anyway. It is I think this proposal that.

Right.

That we use hash, you know that it's apply for the list and function application for the association seems reasonable. Let's move on.

Okay. So down here we have few examples if it helps to qualify things. If you scroll down. I don't know if you want to see them but yeah, I mean this will happen.

Live stream comments that that does this match Color combined, video combine etc and I actually don't think it's that bad a match. This column wise combine. It's a knitting operation of some kind. It's a slightly more complicated. It's not a join. I mean you can call it a join because that's what's called an SQL. But. But it really is a more complicated operation. It is combining these tabulars in another way.

Yep.

So. So, so the decision on whether you're using application. Well, you know, if you're using compose or apply is based on the head of the first argument.

That is correct.

Yeah.

I mean that's.

You can always put apply around your function and it will, it will force the apply if you use an association.

Right. Well, I mean my concern is going the other way. Right. Because that means you can't have a use the same function for both.

Yep.

For. For both list and association, which is something we've done elsewhere.

Function.

What's that?

You have to have an outer function that does the translation. But anyway, you can't use the same thing. You can't use the same function. I see what you said.

Well, no, right. You just couldn't use.

That doesn't refer to the actual elements there.

Right.

I mean. Go ahead.

Right. I mean so for example, in template boxes. Right. You know, even if it's an association template box, you can still have display functions that use numbered slots and it's, you know, it's, it's.

But what function though that doesn't refer to the details of the innards is going to work here? In other words, you can't just say, I don't know, same queue or something. That's not going to work anyway.

Right.

I'm just saying you could have an explicit function in numbered slots that you know, that could apply equally to Lists and associations. And if we make this change, that won't be true.

Well, how would you access the association by name? In the other case, let's say that we did. The. The thing is by default applied. The function is by default applied. How do we then access the elements of the associ. I think by the time.

Yeah, I guess you're right. That way. That way we're forced to always use numbered slots. Yeah, I see what you're saying.

Yeah.

Okay.

But I'd like to know what the default is. What is automatic. I know it's same cube, but what is it explicitly? Do you understand my question? Can we write out the code that is that thing?

Well, at this point it would be two different functions depending that you do.

The combined keys and so on. You know, what is the intersection of the keys, blah, blah, blah.

Oh, it's going to be something like. If you want to spell it out, it's going to be like first you compute intersection of. Let's see if it is a list case. So first you can put intersection of a list of all the arguments.

No, of the keys.

Of keys. Yeah, you map keys on all the arguments.

So it's an intersection of keys at least.

Well, they're not keys, they're. They're column keys.

Yeah, column keys. You're right. No, no, this is. This is the. This is the. An association.

No, it's not the full.

It's a row. Yeah, it's a single row.

He's right.

It's.

It's the. It's a. How do you extract those headers? How do you extract the column names?

It's called column keys. So far for the full tabular. But. But in this case is a row.

Yeah, no it isn't. It's column.

When you get to the function, it's a row. But for the full tabular, it's. It's the column. It's the column keys.

Yeah, no, but. Yeah, but if you want the default as a function, then the function sees rows.

No, but first you have to find out which columns you're going to extract in the rows.

Oh yeah. So you wanted to embed everything in a single function. It depends. If you want to pre compute it, then this is fine. Yeah, Then this is called like, this is called.

It's basically what you have written there. The intersection of.

Yeah, so if you call these keys, then the function is same queue of hash one keys, part keys.

Except those keys might have prefixes, in which case you may probably want to remove those prefixes.

No, no, Those will.

Otherwise be considered different.

If you join a table that. That has an extended key, you know, a comma, country, and another that has extended key, you know, B, comma, country, will automatic join those two quote unquote, country columns.

Oh, yeah, this is for Rob. I don't know how the code works in this case or is for us to decide in terms of design.

That's a good question. Well, I've just. I mean, I've just implemented the thing that, you know, can choose a key based on the prefix. So we can probably do that with. We can probably do. Do something like that.

So, by the way, I think that for the default case, it should be completely identical because if I have an extended key which is, you know, mother country, and then the one which is father country, do I want to merge on that? I'm not sure. Maybe I want to merge the two further countries of the two tables. This intersection, this automatic thing, is very automatic. So I'm not sure we should get so fancy as ignoring the prefixes.

Yeah, I kind of wonder about that too.

If there are prefixes, I wouldn't ignore them. I mean, but there usually won't be prefixes.

No, typically.

Typically this creates the prefixes. Yes.

Also, what if we only join.

I would argue you. I would argue you probably usually do want to ignore the prefixes in automatic cases. Because imagine like, you know, I've. I've constructed some tabular through a million steps. And so it has a bunch of prefix keys and. And, you know, it has a. It has a column called something, something, something, country. And I have another one that similarly is just something else, something else country. If there's, you know, if. If ultimately the only thing is that, you know, oh, they both have prefixed keys, country, then that's obviously what you'd want to merge on.

Well, then, well, in that case, you can merge with a key name extended key, underscore country, or double underscore country, if you might have, like, very extended key.

Right, But I'm saying, like, if in the automatic case where it's supposed to be taking intersections and figuring out which columns, you know, you probably want to join on, I'd imagine if you have two columns called country, but with different prefixes, that's kind of obviously the thing you'd want to join on, which is what I also.

Why, obviously. But why obviously?

Okay, well, I imagine complex cases, right.

It might not be that.

Okay.

I think it's obvious in the case where that's the only thing in the intersection.

Right.

Like, they're Completely different tables, except they both have prefixed country keys. I think then the only reasonable thing to do in that special case, maybe, maybe it doesn't work in general would be to. To join the countries.

But there may be.

I agree.

I admit though, like, right, if, if you had, if they, if they have this prefixed country keys, but then they also have these two other unprefixed keys that align, then it's not clear whether.

You want to join on the prefixed country.

Yeah. I just don't know which is more common.

Yeah. And also each table may have more than one key with the same short name. They will just have different prefixes.

What's the most conservative thing to do? I mean, I. Well, actually, I would argue you might as well do the joining with automagic stuff going on because people will try it. If it didn't do what they wanted, they'll back it out and they'll do something different. But if the thing acts dumb as a stone, doesn't do any joining, they're just going to think it's dumb as a stone and they'll give up.

I, I think the behavior should be pretty simple though, because otherwise it just becomes too, you know, you have to try it to see if it does what you expect, which is.

Well, the default does what you expect. That's not to say that the. Just like you make a graphic and the default may or may not do what you expect or what you want.

No, but I think there's, it's like, you know, it's like, I don't know, there are functions where I can never remember whether the level spec is, you know, starts at 0 or 1 or something like that or like what the. You know, and that's always annoying because it's like, well, I can never write it. I have to try it and see if it does what I want it to do. And I also think we don't want.

To do that here. Yeah. Think about how this generalizes. If table one has already two prefix country. I think this is what Leonie was trying to say, you know, if Table 1 has Country A and country B, what do I match with tables?

Julia. But why don't you just pick the first one? Because it's anyway. There is anyway some arbitrariness in this default. Like Stephen said, you can just say, take all key columns or column keys. Sorry. For every table and pick the first one with the short name. Discard the prefixes.

My answer to this is, in this case of the automagicality, I use the Word count of the explanation I have to give. If it is intersection of keys, it's quite short. Yes. If it is, whenever you have extended keys with the prefix and you have more than one, always pick the first one is already. I already lost people at this point.

Yeah, well, if it does the right thing most of the time, people will be happy. Let's not get further into this.

Yeah, right.

Worst case.

But that's. Yeah, that's a slight objection I have to the function thing too, the change in function. Because right now we can explain it as the function is applied to the outer product of the normal of the tabulars. Okay, but I just tried to outer with association and it just doesn't work.

So maybe it should work. Maybe that should work. Let's see what happens. It doesn't work, but it should. Why wouldn't this work? Why shouldn't that work? And produce. What should it produce? It should produce an association of associations. Right?

Yeah.

In that case.

But what is the function Is the function then given the. The outer keys of the associate. I mean, suppose. Suppose the argument itself is an association. Do we consider that as taking the values as the variar.

I say.

I think it's all. It's all just a.

It's a big.

It's meer. So how would you describe the outer product of the rows that we're doing with the association syntax?

Oh, just merge the associated. For. For every road of. You know.

Right. You have to. You basically have to put an association inside of your function as you're doing the evaluations. Right. So that's. It's not just a simple outer product anymore.

Okay, listen guys, we've got other things to cover.

Yeah, yeah, we can cycle back to.

This we can do by default is. Go ahead. Well, the thing that's clean is the hash 1, hash 2. We know what that means.

Yeah. Okay.

I had one other.

Yeah, go ahead. Yes.

Well, I mean, I don't see. Okay, I had another question about what. What happens if you put the function inside the association? If you know what I mean? Like could you. Could one extension. This might be. You could have a function.

I think we shouldn't allow that. I think we should not do that. I think there should just be one function. If you're doing the function thing.

Well, okay, here's the argument for allowing maybe some functions inside the association. But I haven't thought through would be. If you want to say something like. Because I imagine a very common case is. I want to join these. When the string length of the country name is the same. So then what you do is you'd say B arrow tab one arrow string length.

Right.

And then now later it's treated as string length of.

Yeah, there's two different cases. One is that you're applying. It's. It's a buy type thing. It's like.

It is.

Yeah, yeah. No, but, but wait, wait, wait. If you use a function, then that means that that arrow takes as an argument the full row again.

Oh, so that's right.

You're again something else.

B or string length of. Yeah.

And then you have to do B again. Is that a modified B? I think this becomes much clearer if you just have transform columns before.

Let's not worry about this. Right.

Okay, let's keep scrolling. We have super long.

I would just like to briefly go back to the original question, which is column wise combine the right name and it seems reasonable. The one thing I would point out is that the, the row wise operation is join. Do. Do we want a row wise combine or is that just a sufficiently different type of character that we're okay that type of thing?

I mean, it's a. Because this is column wise and I mean this is a complicated operation. This is combining based on sort of confluence. It's a, you know, confirmation of columns. It's really its own thing. It's not. It cannot be explained fully in a word or two.

Right.

And there's also, you know, I, I mean, we may agree to take this out, but I've, I've made an overload for join on tabular and basically that's the equivalent of what join would work on the normal of tabular, which is list of associations.

Well, sort of. Right. In the cases where Joan. Right. I mean, right. If, if, if you know, you have different dimension matrices. Right. That, that second tabular would. Right.

That's different. But it, but the association case. So in the case where there's keys does roughly the same thing as joining the list of associations.

Well, I would say it is exactly the same thing because the only difference is the extra missings. But those you could argue that are not due to the joy are due to the.

Well, it's not the. Well, Right. Well, yes, but your, your argument that this is doing the same thing as normalizing and joining and then calling tabular again is just plain wrong because tabular will die a horrible death trying to consume the output of joining.

You know exactly how when I tested.

It last week that, you know, two, two, you know, join two matrices of different shapes and tabular on that Then.

It will die for matrices. Yes. Because they're indexed. You don't have name columns, so you cannot do the missing thing. You don't know where to put them.

Okay, guys.

Okay, yeah, let's go, let's go.

Yeah, final point here. Column wise combine presumably works on things which are like the normal of a tabula.

Yes, yes, all right, that's fine.

Or an association of associations. Yeah.

Association of associations is more complicated.

Yes.

Okay, all right, please.

Okay, here we put some examples to make exactly clear, you know, practical use cases and how it relates to the same thing you would do with entities, with the entity functions. And so how you do the same thing with this. This is what we just mixed. Yeah. Is the one to not do the prefix. We decided not to have it like five minutes ago. Okay, fine, but just ignore it for a second. You know, this is combining. Find the customers by the office in their own city. So you have offices, customers, and you combine by country city. Because in general you want the full city specification.

Yes. Okay.

Then here, for example, you have this. I think Carlos showed this already one meeting. Here you import two different country properties coming from different places, like one imported, now built in. So in this case you can just do your automatic thing to combine by country code.

That's nice.

Yeah.

Or here you can use the internal arrow syntax to say that you have name in one case uncapitalized and you have capitalized name in the other case.

Which are now going to being. What does this mean here?

I think this was this an old syntax.

This is the old syntax. This is the old syntax without the function rules. So basically what it's. What it's doing is it's finding the cases where the country code is the same, but the description name of the country are different.

Okay, this is. This is the tuple matching. Yeah, country code is the same for all of them, but then the first important one has lowercase name and the built in one is capitalized name. Yeah. So this is one reason why you might want to split and not use a tuple syntax. With the list in the. In the right hand side, you have one global match and. And two that are like specific by the. The table. Yeah.

Okay. I mean, this is all going away, right? This is not.

Yeah, we can rewrite this. I didn't rewrite all the examples. If this proposal is. You approve these proposals now we can redo all this. Yeah.

What is this?

So basically this is one particular way to find all countries which have more than one office. Basically you self join, you know, Offices with offices. But because this is the same table, you need now the prefix in one place to disambiguate because all the fields will be the same for name.

Again, I will update with the new syntax all these examples.

It's just if you understand inners and so on, you can use that. Now there's also the thing that we invented, which is the selector here.

Right.

Which contains all or 1, 2, 3, or whatever it is. Yeah, okay. I really want to move to these other topics. I mean, I believe these. All right, let's do tabular cleaning. Is that what we want to look at? We already look to construct columns.

Well, no, the title is reorder columns. So here I'm proposing. Let's. You know, some of them we can drop. Like reorder column is subsumed exactly in construct columns because you need to specify the full order. So even if you want to reorder some of them, use transform column. If you want to reorder all of them, you use construct columns. I don't think there's any reason why you should still use reorder columns.

Yeah, okay.

Delete columns, on the other hand, is still useful because you don't want to list the complement of what you want to delete. Yeah, so this is the fast way to do that. And you can delete Here we have two syntaxes. Your one string or a list of strings if you want to delete at once. More, I think these sound contentious.

Yeah, I agree. What's the other thing to construct columns? The one that transforms. It's transform column is exactly.

Transform columns. Yeah.

Okay, Rename column. I think we wanted that.

Yes, rename columns. It also could be done with construct columns, but, you know, very verbose. So again, this one we want the short syntax. Now this thing here, there's a point here which will become clear with separate column and combine columns about the order of a rule. I don't remember. Jose, if this has also we had this. What is new? What is old? In construct columns and transform columns, we have new goes to function of old.

Yeah. So we have the same problem here. Which order to prefer? The temporal order of old goes to new or the consistent with constraint columns of new goes to what you had before. It's a difficult decision.

Wait a second. I'm sorry, I'm confused here. So.

In other functions we use rules to say key goes to a function of what you had before. So in a sense here you would say, oh, the new columns should be on the left, because that's a new thing.

And the old things combine for.

Combine for everything. Let's start from combine, which is the easiest. Yeah. You have combined columns tabular. The old style would be the column latitude and the column longitude are combined into the column position.

Yes.

But if you want a syntax which is the direct equivalent of construct columns and transform columns, you would use a second one.

No, sorry, the data set is super confusing. I think. Combine columns. Now the only question is column wise, combine and combine columns are a little bit weird.

Yeah, I was just thinking that.

Now. Now we have this new name. Yes.

So.

So we are saying that the rename colors, for example, should stay as. As it is here. The old goes to the new.

Yes. Which is what I think these other ones should be as well. Which one is not old? Goes to new.

Construct columns.

What is it? Syntax with an arrow?

It's new. Goes to funk some function or.

I see also in data set we have the precedent for this kind of syntax.

Yeah.

So we're kind of using rule to mean an assignment in a sense in that case, as opposed to a replacement.

Yes.

And so we played. We played a bit with the inverted rule, but we immediately figured out we could. We cannot do that because it's less minus right now and we don't want conflicting with that. Yeah.

Okay. I think. I don't think that's devastating. I think the thing arrow function is kind of a sufficiently different concept than this where the function is separated out anyway. So I'm concerned about these names separating combined columns because I also worry about separate columns being read instead of separate columns. So let's take a look.

Split columns.

Yeah, I like.

I don't think that works because split does something very specific.

No. Yes.

Let's see, A lot of these words we've used, you know, partition, divide columns. That sounds wrong as well. Splice columns which way round.

Yeah, that's the exact reverse of.

That would be the combined columns.

Yeah, well, no, that would be a separate one. Yeah. Because you apply a function, you get a list and then you splice the list in the tabular so it becomes two columns.

Oh, I see. That's more like spliced columns. Gosh. Okay, so what is it doing? It's taking one column. It is turning it into a list and then making that list supply sort of all these columns. Right. So in other words.

Yeah.

This thing, the operation is, you know, take the element hash, apply the function, and then what you're going to do is essentially a construct columns, if I'm not mistaken. Is that right of tab. Whatever the Names are. Oh, yes.

I mean, it's. No, this is closer to transform columns. Right. Because you're gonna have everything else that stand around it.

Yeah. Okay, fair enough. All right.

Yep.

Right. I presume the primary difference is that, you know, this will do it quote unquote, in place, as opposed to putting it at the end as transform columns would do.

No, Wait, wait, why in place? There's no order here. Yeah, yes, there is.

Or you've seen taking the time column and it's going to. It's going to triplicate it.

Right.

It replaces the time column.

Yeah, Well, this is like names part I is F of hash time part I. In a very inefficient way, that is.

Okay, well, so another possible name here will be subdivide columns. And then what would go with subdivide columns would be more like merge columns.

Mm.

I think subdivide is better than separate anyway, because it is taking a thing and subdividing it.

Well, for date list. Yes. But it could also be the column square, the column cube, the column to the power four.

I see.

Is that a subdivision?

No, and actually it's a good point that it could be. Yeah, it's an operation on the. On the column. It's a list. It makes a list. Yeah, that's the better. Better situation would be a column called X. And this would be, you know, x2, x3, x4, or maybe x. I assume you can do this. And then we could say here hash, hash squared, hash cubed. Correct?

Yep. Yes. And consider this now looks like a design matrix. This could be a design tabular. That's right.

Nobody would understand that.

Of course.

I have to mention one thing about this syntax, which I'm not sure if it's mentioned already, that if we want this to be supported by other backends, including SQL, for example, then the syntax where you have a single function that produces a list may make it difficult to directly support this.

Tough luck. Then we. We'll compile that function into something that can be looked at that way.

Yeah, only through some compilation procedure either.

Oh, Seth is suggesting expand columns. That's an interesting idea. Well, let's.

And then we have reduced column.

It's all. It's always a single argument on the left, a single column on the left hand side. Or can it be multiple columns?

I think it is a single one.

So should it be expand column?

Well, originally this was called column separate column combine before we reverted it to matrix and you could give it.

Yeah, I think it would be nice if we kept it to a single one.

I mean, in principle you could give a list. But then it's probably rare that the same function applies to multiple columns. It's clearer if you do it in separate steps. And if you have an operator form without the tabular, then it's not much more complicated to write and read.

The column expand is. I mean, we're really turning. Let's see, we're turning a single column into a list of columns.

Well, a sequence really. A sequence of columns. Right.

Yes. Okay. Turning a single. Using a function to turn a column into a sequence of columns.

Sequencify column.

And the other one is the combination is turning named columns is indeed combining named columns. Applying a function to combine columns into a single column.

Right. And obviously there's no requirement, I presume that those columns be contiguous. So besides the number issue, I think there's, there's that big. They're not exactly inverse operations.

They're not. They're not.

No.

When I think, I think Roger's ideas, this applies. You know, of course the general case like you have first is.

Is.

Is possible. But, but the more common thing is going to be something like date list, which might have, you know, date list can produce different lengths. So depending on the. The number of columns you give, it'll give a date list of different lengths.

How the heck would it do that?

Well, does date list always just give year, month, day?

Well, it gives six elements actually.

So it's, it's, it's, it's a. It. The idea is it's a context sensitive thing.

So if you just did to not work.

Well, Roger's here if he wants to speak up for it.

But.

Roger.

No, I had not.

No such implicit thing in mind. You know, I was just, you know, I think you have to give the.

Function that you want to be executed and then. Was it using a list?

Sorry, what?

In the ref page there were like extra examples or maybe you had like string that were doing some special like more automatic behavior. Maybe.

But right now the list returned by that function.

Yeah. So what do we do? If we just match the length.

Yeah, that. Right. Unless it returns an association.

I would say in this case you just take the first three.

Yeah.

And you call them year, month, day. Even if your time zone settings or. No, I guess date list will always do that. It's not like date string where you can have.

Yeah, yeah, that's different.

Yeah.

I think it's a little dangerous to do that.

There. There is a draft ref page for. It's a Separate columns or column separate. Which I. I think.

I think it's the column separate. Sorry. Yeah.

Which certainly isn't complete, but it had, you know.

Yeah.

Some. Some more details about how this might work, but maybe it's. Right. So I guess the idea was to provide some stringed arguments for, you know, common cases.

Yeah, exactly. I don't think separate is the right concept because of this. Because I think this is a seriously useful case where you want to puff it out by just adding more. Although maybe you do that with transform column. Why would you do that this way? What's the point of this? As opposed to transform column. The main point of this is that you want to apply a function to a combination of columns, to a single column that gives a combination. Okay. To do this in transform column, let's. Let's just write out what you do. Say transform. By the way, there's something very slow with the autocomplete which needs to be looked at. Okay. So what we would do here is we would say X arrow. Well, just X. So there's not really much to say there. X2 arrow. Hash squared. No, hash X squared. Right.

Yes, with appropriate parentheses.

Okay, I am not buying.

Yeah, I mean, I would. I mean, Right. I would say the one thing that I would expect the separate columns to do, that transformation column to do. Well, I suppose, I guess I have been assuming, maybe I'm wrong, that new columns go at the end. So x2 and x3 will be far away from X, whereas something like separate columns, I would expect those to go right next to X, which.

Yeah. Well, let's talk about whether they go at the end. It's not obvious. They should go at the end.

Well, there's no other choice, though. Where do you put them in full generality? I mean, if you don't want them at the end, there could be hash.

X, there could be hash Z, that.

Could be hash W. I mean, if you don't want. If you don't want them at the end, you can name the other columns in the middle.

But there are. So the case where you basically have a list of functions like this is simple.

With this.

The case that's more complicated is. I think dateless is a good example. How would you. How would you do a separation with transform columns with a function that fundamentally returns several. A list of.

Well, that depends if we support that. That syntax that we never actually decided if you want to support.

That's.

No, I. I think that. Okay, hold on, hold on. That's assuming it's expensive to compute the list. If you can just compute it a zillion times, who cares? Okay, I've got a different question. Reordering columns, how do you do.

Right.

Well, well, well, Wait, wait, wait, wait, wait. I mean, computationally it's fine. Right. But in terms of input, Right. You'd need to do like, right. A day, you know, day goes to date list part three and. Yeah, you know, et cetera. That'd be kind of.

How do you reorder columns?

First, first. First subsection here in this.

In this group, construct columns.

Yeah. Without the arrow.

Right, right.

And the one downside is you. I mean, of course, the question, you know. Right. You'd need some sort of insert, like syntax. Right. Other. In order to avoid having to name all. All the columns. Which is kind of unfortunate, but, you know, I'm not sure.

Well, for the order columns, this would be exactly the same.

Right?

Well.

Right. You could. Right.

You could imagine like a hackerrated syntax where you say, put this after this column.

Right. Or swap these two columns. Right. You could use. Imagine using two way rule, too.

Yeah. I don't think swapping columns is going to cut it. I don't. You have to bubble in a column, so to speak, to a new place. Okay, so let's come back to.

So you brought up one thing, though, that is interesting, that we didn't decide on how to do this with the transform columns. Like, do we want to support. And Leonard might have an opinion about this in transform columns year, month, day, arrow function that returns a triplet.

I would be against that, because in general, it will be not very easy to support in other backends. Like, in particular, that's not a good idea.

I mean, have we considered in, you know, transform column or construct columns using a sub list to say these columns should go next to each other?

Well, if you list them together, they go next to each other.

No, they don't. Not if they're not.

If they're already present.

If they're already present.

Yeah, yeah. Look, I could argue that if one is already present, that the ones you're adding should go after. The one that's already present should go in the order that they are. Okay, so here's a claim, maybe wrong. That if you have anchor things already present in transform columns. Okay. The things that you add that are new should go directly before or after. In other words. Okay, so let's talk about the. Let's call a one as an anchor. Okay? And a two was. And here there's a. I don't know how to define this one, A two is another anchor. And here we've got Y arrow, whatever it is. Okay, so I'm claiming that you could be sticky near the anchors.

What if in the original tabular you have first a two, then Y, then a one?

I think, I think, I think I could make that work, Stephen. As if we had the idea that you give an anchor and then anything following that anchor will come immediately after that.

That's what I'm suggesting.

If it's new.

Right. And then you don't.

And for the anchors, you don't even have to give it as a rule. You could just give the name of the column to make it an anchor.

Oh, I see.

And.

And that would work. I think I could implement that quite, quite easily.

I think that would be super useful. So I mean, this one here, it would otherwise be a one arrow A one. But you could just say a one. That's an illiterate one.

Yeah. Do we want to support that? I don't remember, do we have a string as a special case?

Well, no, I mean it. You just showed your, your case for construct column above, you know, for reorder, it's just.

Yes, but without the rule. But we have string as a right hand side of a rule, as a special case instead of function.

Oh, no, we don't have that. We don't have that. By the way, a one goes to A one doesn't work. You'd have to give it as a function.

A1 goes to identity.

No, it's good A1 because identity gets the entire tabular.

Right. Just a side note that for extended any class, if I remember correctly, we do have such a shortcut. So if you point to another string name, it's considered to be a column.

Yeah, well, I've got a whole. I mean, I made this suggestion some while ago and we haven't. I wanted to revisit it, but I have a whole parser based on the column keys, which would take. So if you did like a one goes to, you know, quote A plus quote B with a rule of delay, then that would personal function. But I think it's slightly dangerous. But it's horrifying.

Let's not.

It'd be convenient.

So I think the only advantage if you, if you want to swap column with this is very easy. Like B goes to C, C goes to B.

Okay. Yeah, but that's renamed B goes to hash B. Right. You could say construct columns then.

Yeah. Okay. I mean, there are other ways to do this, so it's not a big deal. Yeah, let's look at the main meter. Exactly. Do we want to support that? I mean, in that case, that would be very nice. That would enforce F to be a function that produces a triplet for every row. You know, Any other case we should.

If it doesn't, it's an error.

Exactly. Yeah. This is fed.

Okay. So this then gets fed in here hash time or something.

Exactly. Yeah. Yeah. This is the way to do this with transform columns, basically. Yeah.

This seems reasonable to me.

It's super powerful.

Yeah. But also I think it's the separate case has only got three sort of cases that are kind of cool and they're, you know, that's about it.

Right. And I mean, we could get rid of combined columns if we're willing to suffer a transfer, a transform column followed by delete columns.

You know, we don't have. I mean, you could do delete columns or you could say column goes to nothing or none. Do we support that, Rob?

Yes, we support that.

Right. Yeah. Okay. Right, well. Right, you need to say it twice. Right, I see.

So let's see. Let's redo with transform column, the combined columns example. Yeah. Like latitude and longitude, longitude, I mean, they get combined in position. So if you do that, if you want to do that with transform columns.

So then what we do is let's.

Do that as an exercise.

Tab of position.

Yeah.

I mean, this is pretty easy, isn't it? Yeah. Arrow geoposit. It's actually very clean. I mean, it's just geoposition. Yep.

Well, that's not all of it, right? No, that's only 50%.

This part is clean.

Yeah.

And. And then you have to do latitude goes to nothing, longitude goes to nothing, or call after these delete columns, latitude.

Okay, so the issue here is this is a combination case of transform.

Right. And actually you'd need to add, you know, longitude as an anchor or something. Right? You'd need to.

Yes, well, well, sorry. But also in the one before, also in separate columns. Yeah. The time column there is still there.

Right, right. So there's an issue here of construct starts from nothing and gives you the columns you want transform. So, I mean, one thing you could do here would be to say this, but by the way, it's not obvious what you want. I mean, you might want position in addition to these other things.

And I think it's actually fairly common in practical data analysis that you don't destroy data, you add new columns with the transformation.

And I don't think that is particularly ugly. I think that that's better Than I think it's quite nice actually because I agree with you. I think it's more common to want to puff it out than it is to want to replace it.

Right.

But I mean if we really find it's useful to replace, we could have another function. Transform column, construct column and replace column.

Can you scroll down a bit?

Oh, what do you know?

Replace columns. So these two replace and insert. We were a bit on the fence. Those can be done with transform columns. If we support. This syntax that you pass in the right hand side the full tabular column which written like this seems cumbersome. But to get tabular column you can simply do tabular part all something. So that's quite.

Why do you do this? What are you doing here?

I don't.

Yeah. Why is this not just transform column. The symbol argument case of transform column. I get. What is this? Replace.

Yeah. What. What is the. Why is it. Why do you have to give the tabular column?

Well, this is the. Is changing a column with something completely new which is not derived by anything in. In principle.

I see.

This is also what was making Leonid uneasy that this has like. No. No length control. Ye.

Not just me, Carlo as well.

I'm sorry guys, if you want to do this, why aren't you just doing a column wise combine? Because you've got to specify the rows here anyway.

Well, column wise combine on what? On row ID or something.

Right.

I mean realistically, the thing you'd actually want to do is column wise combine. Right. I mean this would be like if you, you know, you're taking advantage of the fact that there's ordering if there's like a row id.

Exactly. Hoping they line up. Right. And without the row id, you're. You're toast here.

Well, I would say, I mean there are times when you do rely on row IDs for like vectorization and stuff. Like here, you know, I take out the age column or I take out the population column and I take out the GDP column and I divide them. I take advantage of the. The fact that there's ordering, it'll be nice and vectorized and so on. So this is. It's not unprecedented to care about stuff like that.

Right.

But it's.

But in this, in this particular case, what really put us off was that transform columns typically binds to things row wise and here you're basically adding an entire column. So it's a different.

I think it's the wrong operation. I don't think we want this. I think. Let's not have this right now. But, well, you know, let's get some experience with these other things.

Well, wait, but I mean, we're already going to have an insert, right? And a delete, so we're maybe not going to have an insert. So how else if I, if I just literally have, you know, I mean, how would I do this?

Yeah.

If not, if this operation doesn't exist?

Well, one way is to do a structural join. You have two tabulars. So your new column is a tabular and you just join the old tabular with a new tabular with a table.

What's in a tabular column? It's a collection. It's an association of row IDs, arrow value.

No, no, no. A tabular column is just. It's just, it's just like a vector. It's just positional. There's no row IDs or anything.

Wow. Okay, so this is like zap it in in the order that the tabular happens to be in.

Yes, correct.

Yeah, exactly.

So there's no metadata in tabular column.

Well, there is some metadata. Right. It's like the type only type.

Right. And also there is no guarantee that the original tableau even, you know, preserves the order. Like if you, for example, join using tabular combine or how we call it now, you can join on row id. At least this is clear what's being done. Right.

To me, this is kind of like a raw insert. And I think it needs a name that's different from these other names, like transform columns and so on, because they're all things that operate kind of, you know, element wise, this thing.

Yeah, this is very much a structural operation. Right. This is, you know, extract this column from this table and magically replace a column in this other table.

Yeah. And besides, this will cause all kind of constraints like what happens if the lengths are different and those constraints are already dealt with in that other operation in the combine thing. So it's essentially a combine operation. It's not the element one.

Yeah, I know, it's a. It's like column wise combine, but it's more like column wise insert.

Yeah. Or you can think of it as column wise combine on when you are joining on row IDs.

Yeah. I mean, so.

Do we have a way of joining on row IDs?

Yeah, I don't know how you refer to a row. How do you refer to a row id? I mean, do we have a magic #row ID?

Yeah, I mean, the only way to do that is some. With the current syntax is with like a reserved name.

Like hash Row id. Not great.

Well, I mean, like, consider that every. Every name that we reserve means that you cannot use that name as a.

I know, I know. Somebody's going to have a row ID in their actual thing.

Yeah, you can. You can use a symbol. You can use hash of symbol, row ID or something like that, which will never appear any.

Well, it can't be hash of symbol. Right, that would be. Because that would be £1. It would need to be slot of a slot.

Slot. I meant slot. Yes.

That'S pretty weird.

Yeah, but I have to say that row ID is weird in all. Like, also in databases, it's always a special token.

Right.

I mean, I suppose we could just invent a new symbol which is interpreted in the right way by whatever functions.

Yeah, but for example, if you have the binary function which is a comparison where you have, you know, rows of first tableau and rows of first second tableau, you'll have hash one of row 80 and hash two of row 80. Sorry, slot one and slot two. That's what I meant. Like, if we were to write explicitly what this operation would be, insert columns in terms of the tabular combine or however we call it, we will use this function like slot 1 of row ID double equals slot 2 of row ID.

I'm fairly confused. I mean, I understand the point you're making that they both have row IDs.

Yeah, but that will be resolved by the hash. It was. Sorry, by the slot contextually slot one and slot two. That will be enough to disambiguate.

Yeah, I think you meant this learning. We don't have any arguments.

I think what we want is a function row ID of slot one somehow or that.

Yes.

Yeah. What would it mean?

Yeah, if we have. Okay, hold on.

Because that is. That is a tabular or. Or wait, no, because if we do. If we do ease. Roki.

What about the possibility of just having a function called tabular row id, which is symbolic and is resolved only at the time when is somehow resolved later in the process.

Yeah, exactly. I mean, that's what I kind of meant as well. That it's not something that like, it make. Only makes sense inside some construct that is being resolved at runtime.

That seems useful to have tabular oid, because then you can. You can do things like say. I mean, you can echo a tabular row ID and stuff like this, but that still leaves us with insert column or column wise insert of a tabular column. Because columnize insert is an el cheapo version of column wise combine. I claim where the thing you're inserting. So let's go through what this will actually look like. You know, if you have this, you should be able to say new arrow and then just a list row one, row two, etc. Etc. Etc.

Sure.

Right.

That's what the same as what tabular column is representing.

Right. So what, what if the number of rows doesn't match the length of the tabular? Do you pad it?

Oh that, that's. Hold on, hold on.

I think you.

So you also want to do this where you have row 1A, row 1B, et cetera, et cetera, et cetera.

Wait, what is that doing?

That's putting in two adjacent columns. Okay, so I'm going to claim that the padding.

I think that's the worst way to do it though. You'd much.

Yeah.

If nothing else, because probably want to.

Thread this would be nicer.

Yeah.

Okay. Okay, so look, in terms of padding, I claim this is very similar to. I claim there's an outer and an inner version of this. You might want column wise insert to be outer ified, in which case you've got more things in your column that you're inserting just have missings for everybody else.

Like look, if you're, if you're trying to insert a column that's a different length, you've made a mistake. Right? Like there's no. I can't imagine a world where my column's a little shorter and the right thing to do is to trim yours. Like it just means you made a mistake.

The one case that might not be a mistake and I agree with you completely, Christopher, the one case it might not be a mistake is if you give a certain scalar. Do you want that to fill the by constant array?

Okay. Yeah.

All right. But also, apart from this being probably a mistake, if you want the padding, then we have the combine operations that actually if you, if you join on the row id, that will do the padding for you. With all this, you know, intricacies already taken care of.

Except that we haven't had a way to join on row ID yet. Right.

But we. With, with having row the stabular row ready function we will.

By the way, for the constant case, you could just join, maybe you could just join with the scalar.

Yeah, well we, we, we do that in, in the, the two tabular of column. The two tabular from the column type. We do, we do do these padding operations and I think that's probably appropriate there. And we do scalar, we do scalar filling Basically using constant array, I think.

Why is not the place to put column wise insert.

Now it could be. That's, that's another. So you could do instead of column wise. So the way you would do column wise insert is you could, you would do from tabular of tab comma columns.

Yeah, okay.

And then you do, you, you basically.

You just join of a new column.

You join, you put new, goes to new column and join and then back from, you know, two tabular again.

You might imagine the two tabular though. You, you, you could, I mean, I don't know how flexible we want to tabular to be, but you could imagine a syntax where like if you give two tabular a list of tabulars, it just, it, it turn. It essentially does what you said like it does from tabular comma columns. It joins those, all those associations. Or it joins those two associations.

Yeah, look. Okay, hold on. What is column wise insert? Answer. It is basically joining a very tiny tabular that doesn't have row IDs. I mean, it has only its, its raw row IDs. Sorry, it doesn't have, it doesn't have. It's not joined by a column name. Not joined by. Sorry. It's not joined by the content of a column. It's joined by the content of the fake column that is the row id.

Exactly.

Yeah, yeah.

And this operation though, is the same as an operation I think we talked.

About before, but I forget if we.

Have a different name for it, which is just joining on the sequence of columns.

Basically.

Like join is in the function join.

Yes, yes.

Like presumably you want to be able to have a full tabular in the second argument.

Talk about tabular row ID for a second. Okay. And the question of whether we should simply just have effectively an undisplayed column in every tabular that is the tabular row id. Conceptually.

Well, for what it's worth, that's exactly what's happening in databases.

Yeah, okay, but I mean, so in, in normal of tabular, we can strip that. I mean, what would happen if there's always a hidden column that is called tabular row ID or that is the symbol somehow how would we access it? I mean, let's imagine it was just called quotes tabular row id.

Well, the only place where you typically call these things will be in functions which are bound to row individual rows.

Yes, but then, then, so what we would say we would have, I mean, that would be a reserved word. And so it would go, you know, tabular row id, and you wouldn't be Able to reuse that for anything else.

So with tabular row id. So it's implicit. So if you. If you do a permutation of the rows, does that. Does that get permuted the same?

No, it just. No, it always labels them.

Okay. It always labels them by index. So it's like a tabular index, basically.

Yeah, it's like my appendix. If you rotate the list, you will get different positions.

Yeah.

That'S actually what it is. It is map indexed in a way.

It is. Right. I mean it's kind of like.

Right, yeah. It's like map map index with a. Named.

With a named index which is like supplied.

Yes.

If you had supplied this. These. Yes.

I'm assuming that would supply. Make this irrelevant.

Possibly.

I mean, I think what you do then is in the functions on the right hand side you would. You'd have a supplied and they would contextually pull in the index. So maybe we should not have this immediately. So now the only question is column wise insert and. I could make an argument. Maybe this is wrong. It's not quite that. It's. You've got a tabular. You got a tabular and you want to take something which is a fairly unstructured thing. Okay, we need to wrap up here, but. So we've still got some things to figure out with this insert operation. I don't know about this.

This is very nice.

This is element wise layer doing the same thing.

Not this element wise operation. Element wise layer is a single argument. Thing like element wise layer is going to be like.

I agree that this looks quite nice. This is a kind of a listability. This is an auto threading.

Yeah, it's a.

It's a.

It's a. It kind of goes with thread and listability and broadcasting and it's. It's a bit. It's a bit separate from tabular, but I think it's super useful outside tabular as well.

I agree it's nice. I mean that, that thing there is, if I'm not mistaken. Is that a threaded. No, it can't be thought of as a threaded.

No, because less doesn't.

Well, you, you would have to. You say F of at least comma that threaded and then replace F with less. Yeah, it's very cumbersome to do. Right, well, or you.

Or you use. Or you use that third attribute argument of function.

This is thing is a nice thing. We should have had this before and it's not the case that this could be. No, there's no way in which this is an operator form of thread.

No, it's different the way Rob implemented this. Sorry, you can go, Rob.

Go ahead, Julio.

Okay. The way this was implemented, the design we used is what NUMPY is using for broadcasting, which is expanding basically every axis of dimension one. So you could have, for example, if you have an arbitrary tensor that has the first dimension is length three, the second dimension is length one and the third dimension is length four, you can combine it with another one and that has still three axes and the second one is expanded. So this is doing expansion of scalar automatically. It's very, very powerful to combine all sorts of things.

A second argument that is some kind of level spec just like.

No, because this is. No, because this is very well defined. You can only match two tensors if they had the same rank. I mean, the first case is one is a scholar and one is a tensor. That always works. The second case is that the two tensors, they had the same rank and you then expand every dimension of size 1 into the corresponding meshing dimension on the other side. And this is in both directions. So this is very easily explained. You just explode things all automatically.

Level spec, as it does for outer, for example, where there's a question of you drill all the way down to the leaves of the outer thing and then sort of combine them all.

Or.

Anyway, this looks like.

Yeah, I think it's strictly more powerful without the LeBespec algorithm, but we can discuss it separately.

Yeah, there are different semantics. I mean, maybe this should be an. I don't know if it's LC ILD.

Or whatever that lc, but I think this as it is right now looks just fine. We should try to do this.

How exactly is it different from listability? Like how is. How is element wise.

That's the key question, Christopher, is. So my original implementation was just like the stability and the plus of that is you can use things like threaded inside of it and so on and so forth. So I changed to this after Julio. I had, you know, many discussions and this is to sort of give, you know, users used to Python away in our language to do the. The broadcasting thing.

Okay.

That still doesn't answer the question of how is this different from list.

I'm sorry, it was a lousy argument.

Rob, you can see below how it's different.

Just a lousy argument. Just what, what is the point?

The point is that that case. The case were the. The plus, the plus where you have the message.

So what should.

That.

That should expand the two point.

It should be a singleton. It's a Singleton dimension. That's why it's different. You see, it's a dimensional single element.

So, so what happens if in, in that penultimate example, instead of 2.3, we had 2.3, comma, 3.4, what's the result?

It fails.

It's just not worth having. I'm sorry, this case is not interesting. Let me. Wait, wait a minute. You want this to work or you want that to fail?

To fail.

I want that to work.

So let me say something here.

So.

So we have had several discussions in the past of how to generalize map threat. And so this is effectively that idea using an operator form from outside. And these are some of the generalizations that have been suggested. We, we wanted some, we didn't want some others. And this idea of confusing and scalar and a list of a scalar was one of the most problematic ones.

It's horrible. It's a really bad idea. Let's not do it.

There is a reasoning why, why this can be useful. You may buy it or not, but I think the idea is that if you think of something as tensors, then certain dimensions can be like singleton, like, you know, one element in the dimension. And then even though it's in a list, you can still consider it as a scalar.

No, it's not a good idea.

So I must say this is something that Nick M, for example, has requested as well.

Okay, well, we'll have some other way to do this, but we're not going to do it this way.

Okay, let me make a full case for this in a separate meeting, because I can give you a full specific.

What we should do, if you want to do this is you want to do something where you have a wrapper around this that says expandable scalar or something.

But this can be in an arbitrary place. Let me tell you an example. With only dimensions, the first tensor could be 4, 44 4. And then I have a second tensor which is 4 by 4. How do I decide that? I add those two four at the beginning, at the end I reshape.

You need another argument for element one.

No, it's not one, it's not one, it'S arbitrary. I can reshape the second tensor to be 4 1, 1, 4. And that means where I want to put it.

So there are two separate cases.

Is it. No, no, wait, the same case.

There are two separate problems. One is threading above and threading below at the bottom. And a separate problem is assuming that list of one is the same as the scalar. The second thing is a Lot more problematic.

Exactly. And it's called singleton dimensions. It's present in literally every vectorized language, like in Matlab, in Numpy, everywhere.

You know what, just because somebody was extending Fortran with its way of doing arrays 35 years ago, 40 years ago, isn't the reason for us to do this. This argument will not get anywhere.

Well, fine, yeah, we can use listable, we can just have it use listable semantics.

But yeah, I think the other thing, if it's actually useful, there's another symbolic way to specify it and it's not FORTRAN arrays, which is what that's based on.

Maybe two other things. I mean, one thing is. Okay, it sounds like regardless of what this does, there should be some way of doing the listable thing, because I think that would be useful anyway. Like I've always wanted a listable less. Yes, Another thing would be we could always give this, you know, one piece of behavior. Maybe there could be a second argument to the headdress to the element Y. So you'd say min comma, you know, Python style or something and then it would, you know, it would do all these other things.

No, no, no, comma, confuse scalars and vectors.

There you go. Okay, sure, sure. Right. Or, you know, it could be an option, it could be, you know, singleton, Whatever, whatever.

We can, we can. Right. I think there's probably, if it is useful to have those, you know, one element vectors be the same thing, which I don't think it is. I think that's because of an implementation hack from Fortran. You know, I think that's the reason that was done.

Okay, we, we, we agree, let's agree to disagree. Right, Right now.

Well, that, that leads to. The question is if you want. We, we don't have any way of representing a typed, a typed number in our language. So we have, you know, tabular column, which is now a typed list.

That's, that's going down some other slope. We're not going to get to that today. This element wise thing with listable semantics seems like a really nice thing and we should have it.

One last quick question is for use in tabular and like tabular columns and stuff, does it ever make a difference which of these we do?

Yes. Because the last example I have with plus is the way, which is not least of a semantic is broadcasting from the bottom, not from the top is what every person doing data science wants. If I have a column, I want to add the mean of a column. I will never do the opposite thing.

But this is the easy Case, Julio.

Right.

Because the bottom dimension is two and the list is two. This is what we do with threaded and yeah, this is how quantity array works too.

So this is the easy case, but with less is the same. If I have a list of pairs, I want to do less with one per. The other case is almost never useful.

So is the example that you like. You know, you have your so on the left, you have your tabular. On the right, you have an association that just says, you know, age arrow and then whatever the average age is. And you know, height arrow, average height. And you want to subtract, for example, so you know, they all mean zero.

Okay, I didn't. I wasn't expanding this association syntax with name columns.

This requires another design.

I know. Yes, it does. It does.

Okay, I want to hear about Roger came late here. So we didn't hear about visualization. What are the other sections that we need to cover here? There's a visualization section, as I understand it.

Roger. Did we lose Roger somehow?

Six we did.

Okay. We lost Roger, unfortunately.

Yeah. Well, who's project managing here?

Me.

Okay, so what has been requested in terms of what are the other things that we need to review? There's visualization.

Cara has not been supplied with that information.

I would say it's visualization, connectivity, type, setting of.

Tabular.

So let's get meetings on all of these things, please.

Okay.

As soon as we can, because we got a lot of stuff to pull together. Hopefully these will all be straightforward.

Okay, I'll check that with the. With the team and I long organize those meetings. Okay, One thing, and I don't want to add extra delay. So one thing that we already discussed at the beginning with Roger and the others, and we thought it would have been okay to delay to the second version of tabular is modeling, you know, like how to extend old classify, predict and the others. Because then we need to create a new modeling language where we have formulas that are based on the column names of the line.

We can do normal to get it into. You know that for now.

Exactly, exactly. I mean, because I think we can do something strictly more powerful if we really give some thought to that. Like in linear model fit, you know, you could use name columns in the formulas, but it's a completely new language.

Yeah, I agree, I agree. All right, well, lots to do here. Okay, I think we wrap it up. Thanks a lot.

Thanks.

Bye.

Bye. Bye, bye.