Live CEOing Ep 832: Language Design in Wolfram Language [Tabular]

Hello, everyone. Welcome to another Wolfram Language Design Review. Again, about our friend tabula. Let's. Let's see, where are we? What are we looking at today?

So we have two main topics. The first one is how to distinguish element wise operations from column wise operations.

I thought we had this idea of element wise wrapper.

Well, that's for element wise operations. So we suggested column wise to make functions like transform columns that operate on elements to work on columns directly. And you had suggested an alternative called whole tabular. If you open the section two ideas, you will see the other two ideas. So the previous section is just motivation for why do we really need this double mode of operation. So there you have transform missing anomalies, things that naturally work on columns as a whole.

Okay, so there are things that work on columns, but how does that. What is the element wise mechanism in those cases?

So transform columns naturally works element wise. So it goes row by row and performs operations. And so if you see the first example that you have on the top, you see a case in which we want to, together with the hash call, we want to do a mean of hash call. So that requires getting the whole column the previous line, the first one, the first line of code.

Yeah.

Okay. Insert columns.

So that's how we would do it right now. But we want something nicer than that.

Yeah, we want the above transform columns to work.

Okay, so we're insert columns. Is. I see it's operating on the whole tabular there. Because if I said hash call, I would simply get a single element of that column. Is that correct? Correct. Okay. And it is the case that most of the time you do want a single element, is that true?

I would say yes. That's why transform columns works row by row, addressing individual elements.

Okay, but so this wouldn't work. Right?

Right, exactly. It does work now.

Right, right.

So the question is, how do we support something like that?

Right. So I mean, one possibility, there would be something. But I don't know whether this semantically can work, you know, whole column or something of col. I don't know if that can work.

Well, that's what we called whole tab. You had it as whole tabular before. And that's what we have below in the whole tabular thing.

Right.

And. And I've implemented that to sort of work.

But wait a minute, in this case, then what would. What would it look like? Whole tabular of quotes call or what?

Quotes call?

Yes, yes. It would be whole. Whole tabular of pound call or quote call. Yeah.

So, okay, so the concept here Would be this becomes whole tabular of quotes. Call here.

Yep, yep.

I think it would make more sense to call it whole column since it is always a column, isn't it?

Yeah, fine, fine. I mean, you know, the name. The name. The name is not, you know, we'll figure out the name. But if, if it's the right concept.

So is there a reason why it should be a function like this and not just a symbol that's effectively set with block?

What would the symbol look like? Oh, the symbol would just be. So is it the case that if I say tab of quotes col. No, I have to say tab of all comma quotes col.

Right, right, exactly. So it's like that insert columns is awkward for the very reason you have to use the all.

Is there any object which can be indexed by a pure single fun. Single sub value? The only object we can construct from a tabular which is indexable as getting out columns with a single sub value.

I don't understand what you mean.

So given that tab is a tabular.

Right.

Okay. What is a decoration decorator of tab such that this then works. Oops.

Transpose. But I mean.

Yeah, would transpose work for that?

But I feel like just writing all would be easier than whatever this is.

No, no, I know, I know, I know, I know. But I'm trying to understand that if we have if in your symbol suggestion, Christopher, that symbol should be bound to an object that has the property that its sub value gives you the columns.

I feel like that's just. I feel like that would be probably more confusing than it's worth. Like you know how to deal with a tabular. You don't necessarily know how to deal with some weird transposed funky tabular.

Right.

Well, that, that's, that's a little bit what we're proposing in the column wise thing. But it's, it's more global than just a single argument.

So maybe just a more general question though, about, about the need for this. So like in the example written here, you don't really need a construct like this because you would just refer to tab again inside the function.

Right.

But are there situations where that's for some reason difficult?

You're right. I think.

Well, the, the problem.

Yeah.

So you could refer to tab inside of the function. But suppose, Suppose you're evaluating that function. Okay. And that this comes to a fundamental issue. So you're doing it row by row. So inside a function tab is unevaluated. Right. Because function is hold all.

Okay.

So. So we would have to evaluate mean of Tab of blah for every single row. Because inside, without processing the function first, we wouldn't. We'd have to evaluate that mean every single row.

But then aren't you gonna have to do that for. Oh, that.

That is a problem with the whole tabular.

That.

Yeah, that's a problem with the whole tabular design. That's, that's an advantage of the column wise design. That, that.

Okay, well, let's look at that. But in this particular case, you could certainly cache this. Right? In other words, if you ask the whole column of call, it's going to return. Return some kind of tabular column object. Right. And. And you could then catch.

I mean, it seems like it's less a question of whether you can cache it and more of.

Well, we'd have to catch. Yeah, so it's.

Right. Like.

Yeah, so.

So the way I've implemented it is, is as described below. So basically first I've. It. It's a lexical thing. It has to be lexically in the function. So I do a replace, replace all. And so it. Actually I'm. I'm inserting the column into the function expression in place of the whole tabular of quote, call.

Why does that help? You still have to reevaluate mean.

Well, right, but. But in many ca. In many. For functions like mean, we can figure out, oh, that's mean of a tabular column. We can evaluate it and, and be done.

Okay, so you're Lexical. You're proposing to do some lexical gymnastics in that case.

Yeah, that's, that's how, that's how Jose and I understood. You know, I wasn't in some of the meetings where you discussed this. So that's how Jose and I.

Let's take a look at the colum.

Okay.

Okay. I think we understand this idea and this idea has that, you know, fairly major but not devastating downside.

Right? It's a, it's a downside. Okay, so. So column wise is a special case. Okay. So we say key goes to column wise around the F. And then what we do is we interpret the hash arguments differently. So it's basically doing F of transpose of the list of associations, you know, with. So we're, we're transposing it. So we're getting the association of columns instead of a list of associations for the row elements. And the, that's exactly what we already do in reduced tabular. So it's, it's. It's saying we want and transform columns. Similar semantics to reduce tabular.

Well, essentially what you're saying then, then inside that column wise you would have an element wise. Is that correct? So if I were to write your thing, if I were to say column wise of hash col. No, of element wise, what would you actually write there?

Well, in that case, you don't need to write it because it's listable. So it just works.

Rob has examples below with.

Great, there are examples below. So look at some of the examples and I think you'll see how, how they work. And I start with that mean case. Simple.

And you should be able to evaluate this into this prototype.

Okay. Did I need to evaluate the initialization cells in this notebook or not? No. These icons are still not fixed, right?

Yeah, no, we haven't had a discussion about them.

So.

So here the column wise works super nice because, because it's listable. But there is next example.

Wait a minute, wait a minute, wait a minute. Because I see, so this is replacing the whole column. Yeah.

So when you see high, you're getting a whole column anywhere you see high, you know, pound high.

Yeah, I see.

And that's, that's again what we do in reduced tabular already. So it's not. Okay, so now here. Right, exactly. That's where. That's how we'd have to use the element wise.

Oh my gosh, that's pretty horrifying.

But again, there's no issue in this case of having to reevaluate the mean because it's done once.

Basically what we're saying here is, okay, in reduced tabular, okay, in in transform, missing the meaning of quotes, blah. Is a whole column in transform columns. This is a row wise transforming of columns by default. Am I making sense?

Yes.

I think it's going to be pretty confusing. It's going to be reminiscent of the crazy up evaluation, down evaluation of data set to have these functions. I mean, this is really a row wise transform columns. As opposed to a whole column transforming of columns. Am I, am I right? I mean, in other words.

Yes, yes, you're right. The column wise is saying do it, do it on whole columns versus doing it.

Right. But so, so one of the questions which I think we, we also addressed last time is let's say there are two cases, row wise and column wise. Okay. Are people going to want to mix and match those in a single operation on a, on a, a tabular or are they going to be good, happy just doing the, You know, just doing one operation that is the row wise one and another operation that's the column wise one.

Yeah, yeah, that's A reasonable question. Since we have operator forms for these. Maybe, maybe we could have. Yeah, I hadn't thought of a separate command which, which has this different semantics.

Right. I mean, I'm not sure if that's the right thing. I think you're thinking about. I'm transforming columns.

Another way to think about the mixed case is that one could obviously use a block or a width outside. Compute the thing that you want to compute column wise and then use the row wise or vice versa.

Yeah, sorry, sorry, say that again, Roger.

I'm just saying that in terms of what's a workaround today? A workaround today would be that you use a block or a width outside and you compute the mean for a column, give it a name and then you do the inner operations, let's say transform columns because then that mean is already computed, you know, and it's just a scalar. It gets from its environment, the block environment or the width environment.

But people, people don't know how to use with. To put them in.

Yes, many people do, but I mean, I think that's not. Look, this is not a winner in my opinion. That's pretty obscure. It requires sort of a higher order function modification, etc. Etc. Etc. I'm going to change.

Having a separate function won't change that.

Right, I know, I know. It doesn't really help.

No, it doesn't help.

I mean reading wise, this seems to me to read the best.

Yes, I agree. In that case it does, it does read better.

What are the problems with this? Here's a question, here's a question, here's a question. What about the possibility of saying, here, just saying something like this. Could we allow things like this, where you do a whole column transformation here? Like, like for example, we say here, let's imagine we do this. Is that crazy?

No, that's not, that's not crazy. I'm not sure we even need the, you know, maybe, maybe within the whole column thing, maybe the pound. I mean, that's an alternate to my column wise. Right. It's saying the result comes from, from thinking of it as, as whole columns.

Right. But this is, this allows you. I mean, I don't know, I mean you could say table of 0 comma, whatever it is. I mean, it doesn't. How do you get. Without some kind of supplied mechanism, how do you know, like the number of elements in the columns or you can say length of whole column. In principle, yes.

Right.

I'm kind of liking the whole column idea. Does. How does it work in other functions? I Mean, if I saw this in a piece of code, I would understand what it meant. Okay. How messy is the lexical stuff?

Well, the lexical stuff, the lexical replacement is easy. I mean, it's just a, you know, it's a simple, super simple C function that, you know, if it, if, if it's actually, if it gives one column, it works and if it doesn't, it, it's going to give a message saying that's not really a column, we can't use it. So, so the lexical stuff is easy. The, the stuff that's hard is, is, is how do we know when to evaluate that, that thing one time or multiple times?

Why would you ever evaluate it multiple times? By the way, what happens when there's grouping here? What happens if I want to say, I mean, no, you could imagine a similar syntax where instead of whole column, you say something like maybe this is crazy, where you say something like whole group and then you have the grouping thing.

Yeah. Yeah. And that's. I, I had one idea where we put, we put the grouping as an argument to transform columns and then it did things grouped in some sense like, like reduced tabular. Does.

We have a group?

We can, we can do a grouping for each, each transformation. But, but does each argument in that thing have a separate grouping? That's going to be, that's going to become very, very complicated. But there is lots of applications for having grouping grouping in things like, like me. Because, you know, you might, like in the penguin thing, you might want to group by species or something.

Right? Well, here, if whole column had a second argument, for example, then that second argument could be the grouping criteria. So you could say group by species and by fluffiness. Here. Would that make any sense?

So in other words, yes, that could be done.

Okay, how does it relate to the other grouping stuff that we're doing? So in transform in tabular reduce. Okay, how does this. Wait a minute. Tabular reduce takes whole columns and crunches them to single numbers. Is that a true statement?

That's right. Typically. Or yes.

To give an object, which is what I mean, is it a tabular at the end of it?

Yes, it's a tabular with the. So, so each, each row in the result reduced tabular comes from a group. So. So you know, if your, your grouping finds 10 different groups, then you're going to have 10 rows in the tabular, one for each group. I mean, the, the transform columns becomes complicated because. Okay, so I say, you know, I say mean column of high.

Yeah.

And we can do that, that particular thing and get the scalar. But that's only going to work for things that come out to be typically going to work for things that come out to be scalars.

Okay, but hold on. So having looked through all of this, do you, is it your opinion, if we did the whole column idea, other than implementation issues, do we think that that looks nice from a. I think this is a. Lose all this stuff.

The element wise doesn't. Isn't, isn't. Is. I agree, isn't lovely. But, but, but I think the implementation for the, for the, the whole whole column is, is huge. So, so suppose, suppose you do suppose instead of mean, you're using a function defined by down values and that we don't know if those down values might have side effects or not or evaluate different things in different times. And, and in this transform missing, I've actually added something where there is a side effect and so it, it, you know, it, it has to evaluate it every time it goes through the loop.

Okay, hold on, hold on, hold on, hold on, hold on. You're saying. Okay, look, I, I think it's perfectly decent to say if you say whole column, it will be evaluated once. Period. End of story. Sorry, we're done.

What is it then? What is it that gets evaluated? How do you decide what that is?

Okay.

See that, that involves a function processing thing which is non trivial.

Well, the problem is that what you've got is an up value story. Right? This is as if it's an up value where. Because this whole tabular is affecting the evaluation of mean in particular, it's making it get evaluated.

Yep, yep.

So essentially what has to happen here is that the whole column function has to infect everything outside. Okay, what is the criterion? Everything. For example, any numeric function is safe. True statement.

Yes.

What are the attributes of me?

It's not numeric. Q because.

Right.

It doesn't, it doesn't produce a number when you give it a single number. You have to give.

This is a question for the compiler team, probably. I mean, in other words, you know, this is going to be a type analysis thing because it's got to be whether the, whether the functions given the types that the whole column is. Are those functions. What, what is the criterion? Christopher, can you see what the criterion is here?

Not immediately.

Okay, criterion for what?

For whether you spread the evaluation, how far you evaluate. Clearly if you hit hash high, you can't evaluate that. See what I'm saying?

So in other words, they're basically just asking what can you memorize yes.

What can you memorize what is safe?

I think it's pretty non trivial. Like. Well, yeah.

Arbitrary.

I mean it's basically. It's like asking if a piece of code is deterministic. And in general that's really hard to answer because, you know, oh well, if it's calling module, it's using the module count and then it might, you know, whatever.

When I think something you've said before, Christopher, and I've tried to keep this, so I have this optimized evaluator, but I've tried to keep everything so that we have a normal evaluation mode which is just, you know, just the simple thing that just works with our language. And, and that's sort of how I came up with this column wise thing. I know it's not ideal, but the semantics of that are quite, are quite simple. There's no extra processing that has to be done besides doing that initial transpose. And, and I'm just leery of trying to say. So if I define a function which is, you know, little, little, little M mean as a down value and define it in some way, how. How could we possibly ever know that that can be evaluated?

Well, worst case, we don't evaluate it. I mean, it's just an optimization.

Well, yeah, but you're talking about N squared. If somebody has a big tabular that's, that's dead in the water, right?

I mean, I think, I think the point is that it's often. I mean, look, if we can do magic optimizations, that's nice, but it's also very useful if you can write things in a way where the optimizations are, you know, you ask for them and they're visible and understandable and not just sort of magical.

So look, look, look, here's a way we could certainly do it, okay. If we said whole column here and then outside this. I mean, okay, we could just do this for example or something like that.

Yeah, but is that really that much better than having the element wise?

Yes. Okay. Because this is just somebody says, you know, it runs slowly, the code assistant tells them stick in and evaluate. I mean, people have been familiar with the idea that you sometimes have to do an evaluate from, you know, for millions of years from plot and so on.

But that's a totally. I mean, evaluate only works on the first level in a hold all thing.

Right.

Like this is very different use of value weight.

Yeah, it's.

Well, okay, I mean, I admit, I.

Mean, I think if you wanted a construct like this, you could call it something, but you couldn't Call it evaluate. Well, I mean and honestly the way that we do this right now, I mean this is something that people. You could already want this.

And.

And as I think was mentioned earlier, you just use width.

Yeah.

If you're going to write it explicitly like this.

But how could you use with. You can't use width on the mean of whole column because you don't even know what it is until. I mean, unless you're. Unless you're just doing the. With taking the original tabular object.

Yeah.

Let's look at this again for a minute. Okay, so the issue is with an outer wrapper that tells you. When you know it tells you this is a thing that can be evaluated now. Okay. Where the subtree inside the function is a evaluate it now subtree. So the issue is. Okay, what I'm looking for is this. Is this particular piece of craziness, isn't it?

Actually, I mean implementation wise is not how you want to do it. But isn't this basically once.

It is, Is this semantically insane?

Yes.

Because.

Because. Okay, so. So how do you describe the semantics of how it works? I mean is.

Is there a process Column wise is first and in the column wise it's taking the. The meaning of the association it's giving. The column wise is one which has whole columns. Then having evaluated the column wise pieces, it goes through and does the others, in which case the association is done row by row.

I see, so you're using column wise sort of like your. Your tabular evaluate the. The thing that tells it to evaluate that. Yep, yep, I see what you're saying now. I. Now I understand it. I. So it's a two stage evaluation.

That's correct.

I honestly, without trying something, I can't tell you whether it's insane or not. It probably is not completely insane. Yeah. Column evaluate. And. And then so it. It processes all the column evaluate things first and substitutes the columns in there and. And evaluates that and then maybe nice.

To use column wise, but I think it has a semantic. So semantically different from element wise, isn't it?

Yeah, it's diff. It's semantically different from el wise.

After all this discussion, I'm wondering whether to go back to your original suggestion, Stephen of a separate function, some sort of transform whole columns.

Oh, the problem with column evaluate is what if. What if column evaluate is something like transform missing that gives an entire column. How are you. How do you compare that with the greater than Then high is an element, you have a list and greater just returns unevaluated for example. So, so that's why that doesn't.

I see. That doesn't work because it might.

It might return a list or, or a whole column. So this, this would work for scalars, but only for scalars.

Well, wait a minute. How does this work for. How does the whole column thing work for vectors? It's not going to work either. This, this, this is never going to thread. It's not the nature of greater than to thread.

Right.

I don't agree with you. I don't agree that this is that what, what you can do if you really want the threaded version of this. Yeah. Then if you want the threaded version, you're going to have to put it inside a column evaluate and do the thread. There's no alternative. I mean, then this becomes really no alternative but to say, you know, if hash greater than.

So, so in essence a single column evaluate around, inside. In. In the main part of the body is exactly what I'm saying column wise is. And so you're just adding possibilities of doing an additional levels.

I'm confused by that statement.

Okay, well, suppose, suppose I, I'm. I just do column evaluate around that whole element wise. If thing. That's, that's exactly I'm saying, you know, I'm evaluating that on the columns.

Yes. I mean, I don't understand quite. I mean, if we say here, transform columns, temps, blah, blah, blah, hot arrow. And now we say column evaluate. This is a function. Okay. The right hand side here. Okay, go on. What we. What were you saying? I'm. I'm. I've got myself confused here.

Well, actually, I think I might have been incorrect too. I've gotten myself confused as well, Steven. So you're not alone. The problem is if we do suppose. Suppose something in a column evaluate returns an entire column which is a. Is. Is a slightly different problem. Okay, what are we, how do we, how do we handle the mapping over that thing that returns an entire column?

Well, there's less stability and that there isn't. You have to do something because it's going to return. I mean, this whole column idea, which has the possibility of a grouping as well. Then row by row, it returns, you know, either the whole column or the group that's relevant. Oh boy.

Yeah, it's. I mean, it's.

I mean, unfortunately from the outside with, with. You can't do things that involve groups. At least not easily, I think.

No, no.

Well, I have to say I'm. You know, this whole column thing seems to me, I mean, given. Okay, we can't Given that there's got to be some awkwardness somewhere.

Well, I think I, I will mention that element wise looks, looks horrible. But, but the more, much more common case is you'll be using listable commands like that mean thing or.

I know, but it's going to be so ugly. Okay, what would happen?

I don't think that's the common case.

The column evaluate, which then also allows grouping. Yep. And column evaluate. Returns whatever it returns, which could be a list, it could be a, you know, or it could be a single element, depending on what you have in there.

Yeah.

That to me is the least outrageous of the things we've seen here. And I like the neutrality of column evaluate where its second argument is grouping information.

Yep. That would solve, that would solve one problem, I think.

I also like the fact that has the word evaluate in it because it suggests that you should be thinking about an evaluation semantics here. Okay.

Implement something along those lines, Steven, and we can bring this back up again. We can compare and contrast. How does this.

What are we doing next? What are we looking at next?

The next thing was the, the, the, the binning and the number categorization. I mean, these are just more examples of the same thing. But this is where you want to transform, you know, you want to get a whole column out of a transformation.

Okay, so that's okay. So we should try column evaluate and see how it does for all these things. Makes sense.

I will, I will give it a try. Yeah, I think, I think it's the sort of thing you have to, to try to see if it. See where the corner cases are and what works and what doesn't. And I'm happy to try things.

So.

Sorry, can I mention again the possibility of a separate function? Because reduce tabular and insert columns only work with columns. It's the idea that we want transform columns to work simultaneously with row wise and column wise that is generating the problem. So perhaps having a separate function is a simpler solution.

Insert columns though, takes, you know, its hash foo is a row wise hash foo, is it not? You're creating a column not from whole columns, but from rows and columns. Isn't that correct?

The right hand side is a column, not a function, right?

Yeah. And insert columns, the right hand side is something that you, you just, you take a list and you convert it into, into a column.

So we have the functional notation in transform columns and reduce tabular. And construct columns.

Yeah.

Sorry, I was confused. And construct columns is the one that is row wise, like transform columns. Right, Exactly.

Both of those are row Wise.

Yeah.

I mean, I think it's too bizarre to have a row transform columns.

I don't even understand the name.

Well, I mean, it's a row wise operation.

What's that?

It's a row wise operation. Hash foo means the row element. The element, you know, separate elements on each row.

Oh, so this is renaming what we're currently calling transform column.

Yeah, I mean the name would be transform columns by row. But that's. That's horrible.

But following the. The name whole columns, we could have construct columns and construct whole columns.

I see. Transform.

That's an interesting idea. Yep.

Well, it's semantically simple.

Yep, it's super simple. Yep. And actually, you know, the transform whole columns if, if we wanted to. I don't think we should try this first version, but it could take a. It could take a grouping argument and you'd be transferring the whole columns by group, effectively, which is actually a really interesting idea. But.

Whole columns, then the question is, right, this is mapped over groups. Then whole columns wouldn't actually be whole.

Right.

That'S true. It wouldn't be whole then. So that's not a good name for that. But it's a possibility with a different.

You're still transforming the whole column. You're just, you know, slicing it up a bit first.

Yeah, you are transforming the whole column in steps, but not all at once. But yeah, I. Yeah, I. Maybe, you know, the, the transform whole columns. The, the different and. But construct whole column sounds a bit weird.

So what's wrong with construct whole columns?

Well, I mean, construct column constructs entire columns as well. But yeah, it's. No, it's not wrong. It's just a little bit weird.

In a sense, insert columns is an insert whole columns.

Yeah.

I mean the alternative is row transform columns and transform columns.

Yeah, I think the transform whole columns is much better than row transform columns columns.

Yeah, I don't think that row transform columns makes any sense.

I mean, the point is emphasizing it's a row wise operation.

I mean, once that was explained to me, I sort of got it, but I still don't.

Okay, the problem is that insert columns then kind of has to be insert whole columns, does it not? Otherwise it's very confusing.

Yes, I think that. I think you're right. Insert column.

What are the other functions that are related to this?

Those. Those are the ones.

What about delete columns?

Yeah, delete whole columns. But I think that's.

Well, the deletion operation can be thought of as element wise too. So it doesn't matter whether we Delete the whole column or element by element.

That's true. Good point.

Yeah.

It's like when we say call goes to nothing that removes the column, even though it seems to be an element wise operation.

Okay, what does Roger think? What does Christopher think?

Well, I like this semantically simpler approach to things. Still a little bit dicey about naming, but I think we can come up with maybe better naming.

But the issue is you're going to bank deciding I'm doing a whole column operation or I'm doing a, an element. A row wise operation.

Yep, yep.

Yeah, but.

And this might, I think it will also help us having these two different commands, you know, emphasize the difference in semantics between reduced tabular and, and our transform columns. We have sort of a bridge between the two, which is the whole.

So.

So actually when we started this whole thing, we had these two. I forgot what they were called.

No, no, we had, we had row map, which was transform columns basically. But we also had, we had a column map which was different. I think, I think this design is much better than the, the column map. And, and it's, you know, it's. Carlo can implement it for our, for our virtual back end too.

Okay. All right, well then I think this is provisionally what we should go with.

Okay, I, I will, I will do this. I'll also take a little bit of look at the column evaluate thing, but I won't spend a lot of time on it just as a, Just to be thorough.

Okay, let's go on to the categorization stuff. So we, we. Last time we thought binning was a better term.

Yeah.

And I thought, I tried to think. So what I'm doing is I'm showing you what's different between this and bin lists. And I think bin is a really good name, but I couldn't come up with good name, including Benny. So I just left the names the same and demonstrated what's what. What we can do now.

Okay, so I understand this. What is this?

Don't worry about the. So right now the functions, they're limited at a low level. They only work for tabular column. But I think this should be a general thing that works for a list of numbers.

Yeah.

Okay, so, so sorry about the. The extra and the endpoints in right. Is to make it consistent with bin lists because I, I got the default differently.

Okay.

For whatever reason.

So let's remind ourselves. Bin lists, if I give it random integer 100 comma 100 comma 10. Right. Okay.

Yeah, that's bins of width 10.

Okay. Now what would number categorize like to use?

I.

It. It can use the same source, but I think a nice one is to have, you know, bins of equal length or n. Bins of equal length.

Yeah.

Wait, what is bin. Bin list is a little inconsistent with histogram and maybe even bin counts, isn't it? Because histogram, when you say just 10, it means 10 bins approximately.

That's how we set this up, this up to work. And so that's why I use that strange range of 3 over 4 argument is to make it consistent with the case that we could, we could compare it to bin lists for, you know, getting exactly the same results.

But okay, so what we've got to do here is, I mean it's, it's kind of like, I don't know what the right term is, but it's kind of like number binning.

Yep.

Okay, now, but, but what you've defined here is other kinds of binning as well. And we've also got, we've talked about this categorization function, right, which is analogous.

Can I set up a prototype for that? Unfortunately, it won't work in your prototype. But, but I did, I did set up an idea of a prototype for that. But the thing is, the thing that's important for tabular versus bit so bin lists, you know, is, is effectively doing a group by. It's grouping by whatever bin it goes in. Right, right. And, and for tabular, we want something that gets the categories but doesn't necessarily do the group by yet because we want to do the group by for the whole tabular.

How are you going to actually represent the results of a. So where is this going to go? This is going to go in a, in a transform columns.

It's going to go. Yeah, typically in a transform whole columns. Yeah, precisely.

But then how is it going to return its results?

Well, look at the, look at that tab table I put above. You know, I used an insert a little farther, farther up in the notebook. Farther up. Look at that tabular. So those are the numbers. And that's the result is the, the. So the Q is the result. And that's a symbolic representation of the interval is a category. Those are strings and we can decide what the symbolic representation. But each number has a category that it goes in.

Okay, all right. Okay, fine.

And, and that's, that's, that's absolutely key for uses in tabular is that you have ufo.

So I mean the function that we could get which isn't bin lists is a thing that just Goes through the list here.

Yeah.

Okay. And essentially is, is. I mean, you know, how am I going to do this? I'm going to say hash comma, I don't know, you know, I'll just make it a constant thing. Right, like that. Yes, yes.

Well, and that's what I set up with that number categorized function. I called it that, which is a, you know, not a great name. And it gives. So given a, you know, you set up, you set up the data based on the numbers in the list and then you can categorize any, any number based on those categories.

Okay, but that's different thing. I mean, in other words, the function itself is, I'm a little bit confused here. Is it making a categorization function or is it returning this object here?

It's. It. I, I set it up so you can make a categorization function.

But the.

Categorization function either returns a category or it returns tabular column with categories.

Okay.

So I mean the, the examples are classifier nearest and so on. Right, Right. So this is saying. So, So I mean in this case it would be like a categorization and then there's the data comma, the spec, and then it could just be an X, in which case it would just do it or it could return or this would be equivalent to that. Right? Correct.

Yes, yes.

Okay, so what will be involved in. So categorize versus classify? So classify is a machine learning type.

Thing and it's example driven.

Right.

You see, it's not a spec per se. It's.

It's sort of.

Yeah.

So classify returns a classifier function. What?

Oh, I think you need to give it rules.

Whatever. Anyway, whatever I can do predict. So it'll make a predictor function. Okay. So this would be a categorizer function or categorization function. And I mean, but I think, I.

Think it's a little bit different than learning from examples, you know, in, in the sense that this sort of the symbolic specification language for dates and numbers and strings and categories is a little different.

Sure. But let's see what that specification language is.

Okay.

Yeah, but do you think it's different at this level or you think it's different in terms of how the spec.

Works, how the spec works? Because when you talk about strings, you might want to be able to have pattern. String pattern.

Yeah, yeah, I'm, I'm not worried about. Okay, let's look at the spec. And the question is whether categorize is capable of being a single function or whether it has to be different for different, you know, whether you have to have different. Right, yeah.

And we couldn't make it work. I mean, it was just. We tried. And it's like the way I started.

Down that line because, because I found, you know, I find the bin list thing vastly confusing. You have to look and you have the right number of lists and then you get this and that. And I, I put in, you know, like an association type of thing. Like, you know, width goes to width or, you know, number of bins goes to the number of bins. I mean, actually using, using association type things to indicate what you're getting versus having this sort of arcane language with various nested lists and, and that could generalize to the, the other cases as well.

Yeah, but of course it can. But are you getting one function that's then really complicated as opposed to a few simpler ones?

Here's another issue. What if the data is a mixed thing that's pairs of a number? You know, for example, it might be, you know, penguin species and fluffiness and you might want to categorize those as being, you know, zoo able or not.

But, but then I would, you know, I would use transform columns. You know, that's a custom thing. I don't think we should try to parameterize all possible combinations of things.

Okay, well, let's see what the argument, what, what the specifications are here. I don't understand these, but where's the, where's the stuff with. Where is the proposal for the different kinds of categorizers?

They're in the documentation.

They're in the documentation. There's a number categorized and a date categorize.

And here's a reason why maybe categorize is not a good name. Because we want the category categorize. Yeah, because you might want to reduce the number of categories, you know.

Sure. So let me, let me see what I would imagine. I mean, Right, So I mean, we've got categorized by. Categorized by ranges of numbers. By. By. What am I saying? By sequential. By adjacent. What, What I mean by. By. By ranges that have holes in them. What do you call that.

Partition Subdivision or partition.

You mean intervals?

Yeah. Okay, but I mean by coherent intervals, so to speak, or adjacent intervals. Okay, I'm glad that there isn't an obvious name because I wasn't thinking of it. By ranges basically categorized by ranges, sort of.

I mean, there's a difference between domain and range. So when you do it by range, you might. You want to do it by quantile. Just what I'm saying. By, by distribution of values.

Right. If you only know by ranges, I. E. It could be literal values.

Or.

Equal bins, for example, or it could be quantiles. Okay.

Yeah.

There's also just, you know, categorize by categories, basically by. By sets. So you might want to say, you know, eg, all mammals in one bin, birds in another type thing. Does that make sense? So for example, for us, if they were entities or eg, cities by country.

Right.

Let's look at what that looks like.

So that's a good example because that's kind of hierarchy. Yeah.

Actually another thing, categorization. I mean, eg, locations by country.

Yes.

So in other words, what's in the. The data is a. Is a geop. Position, but you just want to say what country it's in. Although, I mean, how would you do that? Right now? How do you do that? You would say geo. Geo identify, if I'm not mistaken. Is that right?

Yeah, I think so.

What is this? What is all this?

This is something that entities do now. Downloading metadata.

Why is that happening? Why is it taking so long? Okay, so that's an example of a categorization. Yep. I mean, another one would be, you know, are you in the set ABC or the set xyz? Now, what about for dates? That's often categorized by granularity. Is that true?

Yes.

But.

But you can also have kind of multiple ones. Let's say you have day and month, maybe year, maybe quarter.

Yeah, what about that? Right?

I mean, I think it's much. You're much. I think you're much more likely to categorize dates by. By span. Like what year are they.

Essentially. But by. By a granular level. By.

Oh, I see.

Yep.

Okay. Which is what I mean. In, in. Let me see, what other things work like that? I mean, dates, unlike numbers, which don't have named precision levels, dates do.

Exactly. I mean, the closest thing would be integers versus reals, something like that.

Okay, that's a good, That's a good example there. That's a good example. What other things?

So the other thing, categorize numbers is to use sort of a log transformed, which is kind of like addressing what, granularity in a way.

Well, yes, it's like decibels or something. Which is a log scale.

Yeah.

Or the Richter scale.

I see, so that's essentially log, know, log bins.

Yes.

Okay. What else. You got? Quantiles there. Okay, So I mean, one, One kind of thing here. I mean, It seems awfully weird to use such a short name, You know, bin by month. I mean, in other words, if I said categorize and then I had a list of dates and I just say by month and what comes out are those dates granularized for a month? Is that what you imagine? Yes.

That'S. That's not what we were imagining.

Well, what were you imagining?

A new column that has the categories.

But these are the categories. The categories are dates with month granularity. See what I'm saying? That is the way of describing those categories. Now you know, if you say categorize numbers, Then you need a different spec here. Right. And then those numbers might come out as a list of intervals or something like that. In other words, because what this is is a representative of either a representative or a descriptor.

I don't think we have a good way.

Isn't this going to give us a pure function or the. Right. The categorized function because you haven't given a date?

I don't know. I don't know. That's one possibility. Or it could actually just do it.

But. Well, right, but wouldn't it? Well then the question is what does. Right, right. But you won't get the categories out from this, which I think is what categorize is wanting. Right. It's. It's going to give you the elements of the categories rather than the categories themselves.

What do you mean? I mean the categories. You're saying, given a thing, what category is it in? This is not solving that problem. Correct. This is just telling you. This is binning. This is taking, you know, it's like.

I mean what you're suggest. What you're suggesting is. Yeah. Is giving me a hundred numbers and you know, 10 sub lists as opposed.

Doing this and then saying round of that. Right. That's effectively what it's doing, If I'm not mistaken. I mean this is, this is like labeling like, like in a image processing function. We talked about this last time, where it returns the labels for every pixel.

Right?

Yeah.

But you still want the original picture, like.

Right. Well, so the alternative is that in this case with dates, there's no real, really exciting thing that is the function because the function is sort of a trivial regranualization function. Whereas if you want to do quantiles with numbers, you obviously have to process it based on the particular numbers you have. Yeah, that makes sense.

Yes.

I mean, honestly, the number of cases where you need, where it's data dependent seems small, but they may be important. Like quantiles for instance.

I mean even the innocent looking specs where you say you want 10 bins, you need to kind of know something about everything.

Yes, right.

But, well, so another question is what the category, how the categories are represented? So, like. Right, with numbers it's obvious there'll be other numbers, but for, like, dates will be a less granular date.

Yes, but for numbers, what's that for.

Like images, what will it be? Suppose we have a way of binning images together. Like what. What are you going to put as the category?

How do you represent images, what the binning is? I mean, let's take colors. Okay. How would we cut.

I mean, it's. I'm not sure I agree with that statement. I mean, Right. If what we're wanting to get out at the label, right. Then it should be like July, not, you know, 20, 24, comma, seven.

Okay, okay.

I mean. Right. I mean, if you know that what you want to do is group by month or whatever, that's a gather by. Right. And then. And then you project it back to get the, you know, back to the original. The. The shape of the original list.

Okay, but I think it's an important thing to understand whether. Whether what we want is essentially things like July.

See, I think in a future release we could have like, categorical object that knows about what the full list of categories are if they're ordered, whether they're not or not. Um, just makes different things work, even for the list language and stuff, you know, in. In the rest of the system. But I think that's. That's hard to pull off now.

Okay, so you're saying that this will be a category, object, date, July, date, or maybe it's month, date, month or something, you know.

So category, object would probably say July, comma, list of all the months, list of all the, you know, all the months and that they ordered in that order. January, February, you know, whatever.

Well, it can just name that it's a date, month, category, object, and the particular one is July.

Yeah, if it knows. Okay, so here's where it comes into the curation. Let's say that date, month is a curated scale, which it very well could be. Then the only thing you need to know is the name of it. Right. And then the specific one that you have there. July.

Yeah. All right, I think I have to go to another meeting here, but. But we will continue this. We may miss this Thursday, but we'll push this to a different day. Okay. Anything else here right now? Okay, see you all soon.

Bye.

Bye.

Bye.