Live CEOing Ep 835: Language Design in Wolfram Language [Tabular]

Okay. Hello everyone. We are back to doing a design review for our new tabular functionality for Wolfram language. So today we get to talk about. It seems missing an anomalous data.

So yes, I'm going to present this part. We haven't cycled back to this in a while. We thought this time give you some updates ideas.

So by the way, in terms of formatting, obviously issing is not a great. There's a bit of an issue with issing. Presumably we're going to format missing in some graphical way.

It should be like a dash that what you see in the first column. This is what we decided for dataset and unless we get a better proposal.

I would like to. We don't have anybody from design here. I don't expect. I would like to suggest if it is possible to do it, that we go for something which is more sort of a little bit more graphical. Like a slightly a dash like that, but thicken it up and make it gray. See what I'm saying?

It should be if you do data set of that. If you tabular and you do data set of a tabular, it should be gray as I remember.

Do I want to initialize?

Yeah, yes, yes. For later.

Yeah. Okay, so that helps. That's much better. Yeah, that's just fine.

I think we can use this.

That's fine. Yeah, we can use that. Just get it right. I mean.

Yeah, yeah.

Okay.

Okay. So I put the documentation in CDS or transform missing, which is, you know, derived from transfer columns and such. You can click on that link if you have it.

I have no idea. When do I have to update?

Is it from today or late yesterday?

I haven't a clue. I don't know.

Okay, fine. But yeah, we can also just look at the examples here. The notebook, they convey the basic of the syntax.

Okay, Well where is it?

Yeah, yeah. I don't know why it wasn't pick up today. It should be in the prototype build. I will check tomorrow.

It's probably picked up the documentation just updated. So I don't know why that was, but it seems to be updated now.

Okay, so our proposal for this is something much simpler than what we had before. So now that we have designed.

Is this the proposal or is this not the proposal?

No, this is the proposal. This is the proposal with the syntax that we have here.

I just want to say that in these things here, I really don't like seeing literal data. I think it's a mistake.

Well, the image is one of many. I think Roger is working with Tim on a Unified dimension.

I think we should. Roger, I think we should never have explicit numbers here. This is not an example. We should have something where it's sort of grayed out, it can look vaguely like numbers, but it's.

I completely agree with you. It's an illustration. It's not.

Yeah, right.

Well, but what, How. I mean, especially when you're talking about missing, right? It's sort of important to say like, this is actual data versus this.

Oh, no, no, I know, but how.

Do you show it?

But I think it will be possible to show it. We need a design language where it's kind of like the squiggles there that are obviously meant to be numbers, but they're not actually showing numbers, if you get what I'm saying. I think we can come up with something. This is not, you know, we've done this, but you know, in the past, you know, we've literally in more diagram and more drawing type things, we've just done literally a squiggle, you know, a wiggle, wiggle, wiggle type thing.

Right, but the point, the point is, what I would worry about is that we use some notation like a squiggle or like some hash signs or something like that. This is a number. Right, but then it's going to be. You're going to have to know a fair amount to know that the dashes are the missings and not the, you know, squiggles.

Let's see what we can do. And that's not good. But I take your point. But let's see what we can come up with.

Okay, so basically the concept behind this that we try to simplify a lot is that instead of defining a new transformation language, now we already have transformed columns plus addition, like whole tabular, column wise and such. Reduce tabula, which is the other way around. So this was kind of a weird mix of both. So what I hear is like, let's simplify this to the most common cases and just have a more complicated cases that will also be interested into less. Like a fewer amount of user. Sorry, small amount of user. Fewer users to be implemented using transform columns and missing fallback. Because it's going to be very difficult to cover all those specific cases and we can have a very tight list of things that are the things that, you know, 90% of the time you're.

Going to want to do by the time you say forward and then use the next. This obviously should say next element.

Yeah, this is implemented as next and previews. I have those as synonyms. You can both use forward and Backward and next and previews. So if you prefer next and previews. Okay, let's. And then I go with next and previews and.

All right. But anyway, explain how this works.

Okay, so this. Basically the syntax is quite simple. So you have the examples here in the notebook, whatever. Like you can scroll down and you see all of them. So mean will basically, you know, if you don't specify any column, it will apply this to all the columns. And we have only a few of them. We have mean, median, and mode. And those cover the vast majority of cases about what you do with statistical functions. Again, if you want some weird statistical thing or special case, you can do transform column with missing fallback and then you can put whatever function you want. And I think that's preferable to have.

For example, the syntax would be column name arrow mean would mean transform the missings in that column. Is that correct?

Exactly. If you only want to do column A, you do A. Goes to mean.

And what. What does mean mean?

Mean? It applies the function mean, so they.

So it replaces all the missings with the mean of the whole.

Yep, exactly. It's, it's. It's a. It's a basically very simple constant replacement. And then you can use constant to escape whatever, like constant zero in this case or constant minus one, or whatever special value you want.

It might be nice to just allow the possibility of just saying zero.

I know I had that. And then I added strings and then I added other things. And then people came to me saying the design was met. So this is kind of a very conservative design. We can. I think we can start from here and re. Add some sugar. Being very conservative. But let's say. I just want to present you the tighter possible design.

Yeah, but this is bizarre.

This is actually. No, you can just say random. And I have an example. These are just. And that will randomly sample from the column. And I have an example here.

Or you can randomly from the existing distributions. But that's not. Look, I think that should be random element. If you're picking up another element at random. Is that what you're doing?

Yes. Yes.

Okay. So I mean, that's an element. Yeah, and that's.

Okay, that's.

And then we have something more like, you know, random distribution or something. And then what we want to do there is random distribution.

It's not random distribution, though.

It's random sample from random variate.

Okay, sorry. Yeah. Okay. But then I don't understand why. I mean, what I would like to see there is just. Literally, I mean, don't Invent a new language. Just have random variat of normal distribution. You know, whatever it is, you know, it's not.

It's.

It's.04.

It's not really a new language because the, you know, you put a distribution there with. With the. You know, I just. The most exotic sample in. In here. But if you go to the scope in transform missing. Of course you can put normal distribution 04 if you have. If. Yeah, sorry, go.

The problem with just saying random variat of distribution, it means that's going to kind of evaluate and gives you.

Of course, of course, of course.

But we don't know how many you need a priori. Right. In a way.

Yeah, of course, of course, of course. So the only way we can do this is if we can somehow find an excuse to put it on the right hand side of a delayed rule.

Yeah, exactly. Exactly. Which was a previous design with arbitrary function that was kind of getting out of control in terms of semantic. Of you do it for one column for the whole tabular. How do you pass the whole tabular in stuff like that. So again, the design principle here is simplicity. So if you do put normal distribution 04, it's just going to sample from that. If you do put normal distribution with the symbols. This is documented syntax in estimated distribution. So it's just going to call estimated distribution on the column on the fly. And I think this is a kind of easy sugar.

No, I think it's completely crazy to have names there which are not scoped.

But in estimated distribution is. I'm following estimated distribution.

I know, but it has scoping. It has scoping.

Because it says no. Well, it shouldn't open it. I was also surprised. It's not like ND solid.

Roger, can you comment on that or somebody else please?

Yeah, random variate sort of distribution.

Because you can. You can pass parameters that are numbers.

You can also. You can.

There is a version of it where there is scoping added. If you. If you give initial values, let's say for all of the parameters, pq, et cetera, then it's more along the lines of scoping, but you don't need it.

That's weird.

I also like. And can you do things like estimated distribution of normal distribution MU comma 2 MU.

Yes.

Yes, you can.

Yes.

And just like that. And then you don't have to say mu somewhere else.

Yes.

Weird. I was thinking.

But it's not an analogous function. What's an analogous function? This. Because what this is doing is it never has to tell you the values of alpha and beta, right?

No, it returns the symbol with the values instantiated. There is a. Which is kind find distribution parameters.

But the similar thing to this, it's a distance, for example, distribution fit test, where you're gonna fit to, you know, to the distribution. But, you know, unless you ask for a full, you know, test conclusion, you will not know what the, what the estimated distribution is. And you can have rejected, not rejected. Can you fit a normal distribution to your data? And so this, this, this kind of.

Way, you know what this is analogous to is estimated function, which is a function we don't have. Right. Because what we have is find fit. See what I'm saying? Which has. But I mean, we are.

There's a concept that we're missing which is sort of, you know, model structures. And if. And a distribution is kind of like a model structure. It's a family of distribution. If you think of all the possible parameter values.

Yeah, okay, but, but look, look. One of the things that's driven me crazy is that find fit returns, you know, you have to substitute in values and things. It isn't doing the thing that this does, which is to produce in what instead it should do is if I say find this fit and I give it a general form of the fit. Like I say sine, you know, a sine, B, X or something.

Well, I think they're use cases for both. And I didn't really find, you know, distribution parameters, but there was a lot of argument within the team, so we decided to provide it.

Okay, but look, look, look, look, look, look. The thing that we're useful here, and I've needed a million times is just fill those freaking parameters in to this function here.

We did something like that with fit that allowed you to get different things out using I think a rule syntax. I can't remember exactly what it looks like, but.

But this is crazy that, that we have distribution. Estimated distribution. And what we want here is just this. That's what you want most of the time. I mean, you also sometimes want the parameter values, but much of the time you just want this.

Well, that's why we built estimated distribution for workflow reasons.

No, I understand, but I'm saying we need an estimated function as well, Right?

What if you, what if you. Okay, crazy idea. What if you just drop the.

The parameters?

Oh, no, this is more general. Yeah, I was thinking if you just drop the parameters, but then we need a clear definition of what is a parameter. What is a symbol with no value, basically?

No, no, but if you, if you did. If what this was was something like this.

Distributions have only pars. This is both pars and vars. That's the problem.

I understand, but let's say you give the var here.

Exactly. So the problem here is that the distribution are a fixed list. While here you could have sin, or you could have an arbitrary other thing which is not sin. Yeah, it's not a fixed list, it's an arbitrary list.

I don't understand how much you're saying.

Like in estimated distribution you have like normal distribution. Normal distribution is from a fixed list of distributions.

That doesn't matter. I mean, always say here is we can make the assumption that every unbound, you know, every undetermined symbol is a thing you have to try to fill in with.

Yeah, this is what I said. We have to agree on a clear definition that the symbol with no values attached is a free symbol.

Right. Well, if it isn't around.

Well, it has to, because otherwise you won't get a number and you won't be able to fit the data.

Yeah, exactly. Exactly. Well, I mean, I think, I think this is neat. It's short. It doesn't require you to give any, any property.

Right. I mean, I don't like the fact that there are these. Just these, you know, this, this should be added to our discussion about variables because this is another case of variables. See what I'm saying? Where these things are really formal variables whose names are utterly irrelevant.

Exactly, exactly. They could be whatever.

Yeah, they're a bit like pattern variables. Yes. And you know, there's a certain argument that really the way this should work is with pattern variables. Because if we have pattern variables, I mean, if. If what we had here was blank sine blank X, that would be semantically reasonable.

Except you have to be able.

To return what the blank like above it. Oh, I guess if you.

No, this is a syntax, no parameters. Yeah.

If this is basically estimated function and it's just blanks, that is semantically exactly right. I claim.

Well, one weird thing would be if you use. Right. If suppose inside of the body of your estimated function you call, you know, match queue or something and there's a blank on the right hand side.

Right.

So, Stephen, we already had a discussion about this in LCC and we decided to add a string argument to find fit, which is like, you know, best fit and function to give it, return it as a pure function. And you know, it's basically waiting on Rob to have enough free time from tabular to actually implement these things.

Oh, so we solved this.

Yes.

Okay. Although the Way estimated distribution solves it is different, and I think we need to bring those together. But that solution for fine fit is certainly a good stopgap. Rob, can you speak to that? Rob, you're speaking.

You're muted.

I forgot I muted. Yeah, it, it's. It's. I think it's a reasonable solution. It'll, It'll just take a. A bit of a block of time to actually do it.

It'd be nice to get it done. Is there anybody it can be delegated to? Well, delegate. Okay, whatever. Let's get it done. I mean, but I'm, I'm telling you, I don't like the design inconsistency between estimated distribution and the lack of an estimated function function. And the way fine fit works. I mean, I would say we leave.

This is how we make progress.

Whatever that. I mean, I think that the correct way forward is find fit with this additional argument that gives the functionality and then let's do another design pass on, you know, estimated function, estimated distribution. Does that make sense? And I favor using pattern variables or at least allowing patterns.

We should fold the discussion about nonlinear model fit with tabula in that because we need to talk about model language. The.

Yes, that's fine. But I'm, I'm just, I'm just putting it out there that I think estimated function with blanks here will be very useful.

Okay.

And is semantically reasonable. I mean, it's the thing one needs endless time just, just to get a workflow clear. You've got some data and you want to plot it together with its, you know, fitted distribution. Right? You want to plot it together with its fit. And, you know, if we could say plot data. Okay. I'm, I'm prepared to allow myself and evaluate here, although I don't like it. Oh, no, we can't do that. We have to say list, plot the data. Right? We'd be saying list, plot, show list plot of the data together with plot of estimated function data, comma, blank sign, blank times X, X. I, I don't.

Understand what blank sign is doing there.

Says where the parameters are.

That's the parameters.

But doesn't. Isn't blank sign something that has a head.

Sign?

Oh, but the space, admittedly is pretty darn obscure.

Yeah.

Okay. Okay, okay, okay, okay. So, I mean, there. You can name it if you want to. It doesn't make any difference. You can call it, you know, a blank if you want to. A blank or B blank if you want to.

Overall, I feel like this, this design is, you know, kind of Nice. Except completely different from the way every other function currently works.

Yeah, exactly. We generally scope. We generally scope things that are variables.

Yes.

And write them as symbols where not as blanks.

I understand. I mean, and I think we scope.

Them in a weird way though. Yeah. Because like stuff like solve doesn't really scope them if you're already defined.

Well, remember, solve doesn't have this parameter issue. Reduce has something of the parameter issue, but only just. This is the case. These are parameters that don't matter.

Why are you drawing a distinction between solve and reduce?

Well, it's not a very real distinction. I mean, because there are extra parameters in reduce.

There can also be extra parameters and solve.

Fine, fine, fine. But the whole point here is that you're getting. You are taking those blanks and instantiating them. That's maybe.

This is maybe closer to minimize, actually.

Right.

Because minimize. Similarly minimize returns two things. It returns the like evaluated, you know.

Objective, but it doesn't return a functional form. This is. This is something that returns a functional form. That's the difference. Yeah, we don't have very many things that do that. I mean.

Well, what. I guess. What Minimize. I mean, it semantically does return that. Right. The first argument is the objective function with the parameters plugged in.

Correct.

It's just that it might be, you know, unevaluated symbolic expression that you need to call replace all to actually evaluate.

But does it. Okay, can minimize. Yeah, minimize can return a symbolic form for the first.

Yes.

I mean, that can be parameters in minimize. The symbolic minimize function can deal with parameters.

But.

But the point being in the first argument of it, it plugs the parameters in, which is the distinction you were making earlier about plugs the variables you.

Minimize with respect to variables. There might be additional quotes, variables which we usually call parameters, in which case what comes out is a function of.

Those parameters, explicit function of those parameters, whereas here it is not. It's only a function of the variables.

Right, exactly.

Yeah.

Because we minimize the weight of parameters.

Yeah. So, I mean, I really do think this is a somewhat different case. Go ahead.

Well, no, I mean, the distinction you're making earlier was that this is different from some other functions like, you know, solve or something in that we're going to automatically plug in. You know, you never need to see what alpha and beta are. They're just going to get plugged into the function.

And I claim this is unique in that respect. Are you?

No, my point is that minimize does exactly that.

Okay, let me understand that. But give me an example. Is it parametric Minimize of some kind?

No, just, just normal minimize. Like look at that first example. But then maybe give it one other parameter like say that, you know, that the first input, you know, make the objective function times Y or something.

Actually, why don't.

I mean, why isn't what you want a fine fit value which just like.

Right.

We have solve and solve values, solve returns, rules, solve values, returns, you know, the, the values plugged in.

Do we have a minimize values?

Actually, no, we have arg min and yeah, we do min value. Min value. An argument they, they give, you know, the first or the second. And those are very useful for workflow reasons. But it, we added them many years after we had this first one.

Right, yes, right. But so, so what's the analogy here?

Because just, just like in the case, you know, with the underscores right. On the min value is just you never actually see X. X is just injected in.

I see.

But, but right and right. Just like we have D and D solve value. Right.

And actually to fill in a different thing, you might want to constrain X somehow and you might also want to constrain these parameters here.

Right, right.

Yeah.

Arguing against myself there a bit.

Yeah.

Yes.

I mean also, there's things, I mean, I guess you can do this with patterns, but there's also, also things where you want to be able to make the, you know, X appear more than once and have to be the same X.

Well, I just, I just think that the processor for patterns is. So you go through and you take, you take a blank and you make it a unique variable and, and you know, so two blanks. If that were a symbolic thing and solve, that would be dealt with as a symbol blank and it would, it would consider it to be the same parameter.

Yeah, which is wrong and solved.

Which is. Yeah, but so, so we need a special processor for patterns, which is going to have weird corner cases.

Well, I don't know. How would it be? Go ahead.

Here's maybe a question because I think this, this relates to broader. Well, yeah, so what it seems like one of the benefits of using blank would be that at least I think blank is hold first or something. Right. So it's not. If X. If you have x defined as 12 and you use X blank, that's not a problem.

Right, Correct.

Which is, which is one of the really annoying things when using solver and minimize whatever that you've got to be careful that, you know, oh, X isn't defined or something. So, but, but maybe we could have a wrapper that, you know, that would work and minimize and solve and stuff too. That's also like blank. Just hold first. And that basically just says this is a little symbolic symbol. I'm going to call it X. But you know, that's, that's formal, basically. Yes, but, but I know there's formal X, but. But is there actually a head formal that you wrap around X?

No, but there maybe should be. And that's what we were, we had that discussion about variables a few weeks ago, which we need to come back to. I don't know. Who is that? That was an LCC discussion, as I recall. Yeah, right, okay. Which we need to come back to. And by the way, when we come back to it, the idea of a formal wrapper. I think we discussed that in that meeting. Formal of X.

Yeah, I mean, I don't think we discussed it in that.

Specific meaning, but it's something we've discussed in the past.

Okay, well we need to come back to that topic. And when we come back to it, we need to talk about estimated distribution, estimated function. But I think the solution right now is this fourth argument to find fit fifth document. Is it really the fifth document?

Well, no, I think it's a rule syntax, if I recall correctly. I don't.

Okay, fine.

It's, it's like what we did with fit, where we, we added a rule syntax to, to give.

Okay, well, okay, so let's, let's come back to our. I, I mean, Christopher is claiming, by the way, that the X here is as irrelevant. In other words, that I don't really need. And do I in fact need it here? Do I need that second?

Oh, I mean you, you need that X for sure because you can't distinguish X from a. Oh, yeah, yeah.

How do you know what you're minimizing?

Yeah, right. But so though I'm claiming it's not the same thing as, as the case of, of as this find fit case. Because the find fit case, you know, you are specifying the X, you can't get around, around that.

But you.

By exclusion, you know what the parameters are.

Well, suppose. Okay, so here's a case though you might imagine we'd have a symbolic find fit.

Yeah, right.

Where you can spit to symbolic data.

Yes, yes, yes, yes.

Which we don't have right now calls minimization. So it could call minimize in principle.

And which will be an interesting thing to explore. I mean, I'm not sure that that's a realistic thing because the fact is the, the chance that you have so little data that that formula Isn't just going to be absurd. Seems very low.

Would probably work. It would probably work in the linear case quite well.

But.

But that's. That's pretty. Pretty uncommon. Right? So.

Well, actually, another thing, another reason why it makes sense that find fit takes this argument. Is that right? I mean, it returns a. A list of rules with these things. Right? It makes sense that you ask.

Oh, sure.

Tell it, you know, tell me, you know. Yes.

If it's returning that list of rules. But that's why I'm saying we need an estimated function thing that doesn't return the list of rules and just returns the estimated function.

I just worry that. That there's going to be, you know. Oh, I didn't real like there's going to be. If I have a bug and I have some, you know, I meant to just set X to 12, but I forgot and now there's an unevaluated, you know, or an undefined X in my expression. And now all of a sudden you're fitting it like.

Well, that. That can actually. That actually does happen because we. And find fit localizes X, does the computation. Then when returns the thing. If X had a value that gets substituted when it reevaluates.

That's the opposite issue.

Can we, can we go back to.

Yeah, yeah, I was about to vote to go back to tabular.

Right. Okay. Transforms. Transform missing. Right. It's not transform missing.

Yeah, it's missing. Sorry, typo me.

Okay.

All right. Okay. So basically here's the idea then. Let's table this discussion about normal distribution, Mississippi. We should deal with this together with the rest.

But I'm just trying to understand what this looks like. Column A, column B. We want to be a random variant. Exactly what we need to say what kind of random variat. A random variant like that is kind of silly on its own.

This is why I bundled that with random. So it is the random sample. If you. I know random choice. Yeah, yeah, yeah. It's only the first part of random element or random. Random variant. But yeah.

But that's really random choice. Actually that will be the correct name for that.

Yeah.

Is a random choice.

It is a random choice on the column every time. Yes.

Well, except no, it isn't random choice because random choice. It might be a discrete distribution. We might want random choice. So I think we do want to call it random element.

But there is a distinction between random choice and random sample. And maybe we want to have that here. Like you. You don't.

Okay, but that's a different issue. I mean, this Is, look, this is this thing here, right, for example.

Yeah, yeah.

Okay, so the issue is if you want parameters in the random variant, you could say something like this and then you say random variat list, random variate, normal distribution. And I absolutely hate giving parameters there. I hate it, it's really horrible. But you're saying fit those parameters and then use it as opposed to saying something like.

I don't know, 10 or whatever. Yeah, no, yeah, once 10 doesn't work. 0, 12 or 1 3. Yeah, don't do 1 0. Yeah, but in this case you're right. The syntax just with only random variate wouldn't make sense because you don't have a distribution.

Well, unless we assume a distribution, unless we assume a normal distribution or unless.

We assume distribution or not, it's going to be like something like fine distribution, but that's very slow. I wouldn't, I wouldn't go there. I would assume the normal distribution.

Pretty horrible. But yeah, yeah. I mean, the truth is, most of the time, you know, you could imagine. You could just do that. I mean, if we're going to make it simple to use, this is going to be a common thing to want to do.

Yes. If the distribution has no other forms, because depending on the number of parameters in the distribution, you may have a different distribution.

Okay, reality check. When people do this in the real world, what distributions do they use?

Well, that could be normal, of course, that could be uniform. The question is whether you want to uniform on the min max or some other domain that you know for the data. Because it might be that, you know, that your data was sampled from some interval and you want to put it in, you know, and then, you know, there are other distributions like gamma, for example, where we have, you know, different number of parameters, specify slightly different distribution or maybe something, you know, completely different.

I don't know. I'm wondering whether we shouldn't just have, you know, in terms of specifying, what do you do here? I mean, why not just have something that says, You know, something like. Well, this might be crazy, but something like this where you can just say. Where you can give something which is to be applied to the whole column, where you specify in an association what you want to do.

That makes sense as a last resort type of thing.

Yeah.

Because you surely, surely want these easy cases first, what you actually use.

Oh yeah, sure, yeah. And I take Goa's point that uniform distribution doesn't isn't very well defined.

So what are the things that you should put in the association here?

I don't know, potentially a function to apply to the whole column, but I'm not sure. Or just you know, element function.

But also you could delegate it to other processing functions. We have.

Have we talked column wise here yet?

Today we're trying to make transform columns work so that you can do that sort of thing just naturally.

This was my initial, initial point. Like at this point it makes much easier to try this with transform columns. And we should be working to make transform columns easy to use in this case because you're really going to your custom thing. So let's not create two functions that can do custom things.

Real world reality check. What do people mostly do? They mostly want to set it to constant values. That'll be the number one thing to do. Number two thing to do is probably mean value maybe.

No, I think like random element. I think random element is usually the most principled thing to do.

Yeah, I mean I put like the families that I put here are.

I think another common one is just to use the previous.

Yeah. And I have it there like the next previews. Yeah, here I over interpolate. Exactly. So there's really four families. Here is constant statistics in terms of mean, median, random and the end positional base. The one before, the one after. The idea here is let's keep it very simple because this stuff will cover a lot of uses with a dedicated function. And missing fallback is I think the way to go to basically turn transform columns into the powerful missing imputer that you were trying to design with the association.

Stephen, how does it remind me how missing fallback works?

Missing fallback. I mean you basically in transform columns you say missing fallback of a comma, whatever you want to do. And that function will return a if it is non missing and use your function if it is missing. So it's basically the only thing that you need to turn a normal function into a function that only works on missing elements.

But wait a second. But that thing cannot globally pick up a column. I can only pick up the.

I'm not gonna. I'm not gonna answer this because there's a need for the next session in the discussion.

That's. That's the next point of discussion.

Exactly. So let's. Let's stable this consideration because it relies on.

Okay, but I. I'm not really keen on. I mean I think this is really pretty cruddy the syntax given that there's a perfectly reasonable. I mean the problem is some of these require very supplied like information about the local environment.

Yes.

So Stephen, what about let's, let's move to the next part of the discussion.

All right. Okay. And then maybe, I mean maybe what we do is we do something like this. Well, there's nothing. Yeah, I mean then this could then make use of whole column information. I mean that function could, could be given whole column information here.

Yeah, this was the original design that, that would have been a function that takes the whole column. But then you, you are starting to design something which is exactly like random column. Sorry, random column. Like, like.

Yeah, yeah. So at that point, why not use transform columns?

Exactly. Because transform columns doesn't get the whole column. But that's what we're going to discuss now, according to your suggestion, Stephen, it's your suggestion.

So let me add one little bit about anomalies because it's kind of design directions here and then we'll let Carlo and Rob speak about that also because I have to run away today. Sorry. So anomaly detection, current functionality is the symbols that you see here. We have find anomalies that will return anomalous data or with properties and non anomalous or stuff like that. Delete anomalies, obvious thing. Anomaly detection is the one that trains the operator form of find anomalies. And then you have anomaly detection function which is the result of anomaly detection and anomaly detector which is the option for predict and classify. So this is the state of things right now. Now I say train. This stuff is basically a single method function because it relies on learned distribution which has in truth multiple methods inside like multinormal Gaussian mixture and stuff like that. But basically the whole system is based on learned distribution and the concept of rare reprobability. So that if you have a distribution then you can check a data point. It's like a p value thing. If the probability of extracting that data point from the distribution is very, very low, then that's an anomaly.

I remember, yeah.

So this is the current system. Now we want to quote unquote, generalize this thing to simpler things like missing, like the transform mistake. So there are like few very, very common methods that don't require, you know, they do require training in a sense, but it's more like fitting that are like Z score, which, you know, name open, which is based on standard deviations. Exactly.

Yes.

Interquartile range mean absolute. Sorry, Median absolute deviation from the mean.

We have functions, right? Z score. We have a function for computing Z score.

Yes, yes, yes, yes, we do, we do. I mean again, this is for the sake of, of speed that you do, you know, operator form fine anomalies. Z score piped on the tabular and that you get the mask out. How do you know? It's called standardized standardize. And then you have to.

No, but the.

No, no, no.

Z score is relative to a distribution.

No, it's relative to the normal distribution fit on the data.

Right. What is the value of this thing? I mean really, what it would be is something more like.

Is like absolute values of standardized. Of data less than 3, for example.

No, but standardized doesn't do that. Standardized actually messes with the data. I mean, if you say Z score of data comma X it should tell you the Z score of X relative to that data, right?

Oh, oh, C points.

I don't know why we don't have that. We must have that somehow, somewhere. Maybe we call it confidence interval or something. What is that? What is that function? Statistics people. How do we get that?

What did you ask? What is it?

This thing here. The Z score with respect to this data fits to a Gaussian of this value. How do you get that?

Standardized with respect, you know, centralized. And then divide by the standard deviation.

No, it's not getting that. It's not doing. Oh, you mean that Standardized gives you the Z scores for every variable.

Exactly.

Every element. But this.

Is it.

No, I'm asking for a particular element relative to this distribution. What is its Z score? How do I get that?

Right? So it's like mean of. It's. It's. What is it?

It's like minus mean of data divided.

By standard D exception of data.

Exactly.

But yeah, maybe there could be a function for that.

I think that will be a useful function. Can we bring that to an ILD meeting, please?

Yeah, I'm taking note.

Yes, I'm taking note.

So you want to have X minus mean of data divided by the standard deviation of data.

Pretty much, I think.

Okay.

Seems useful to me. Okay. In any case, what are you showing here?

Yeah, I mean, that is basically a way to extend the design of find anomalies. So if you open find anomalies new, that is the link for missing data.

We already know it in the tabular for outliers or anomalies, we don't know it. So the first thing is to find it, you know. And how. How do you define it? So that's what this one gives. And it can give the same bit vector back as we have for missings and stuff. And we can now. We now add it to select.

I'm sorry, I'm confused here. What are we.

Open. Open the documentation for find anomalies new. There, there's a link that you can just click on that. No, you will ignore my.

I'm going to have to go in like 15 minutes or so. So I, I think it's important that we get to this other stuff.

Okay. I mean we can cycle back to this if you want. So this is the current find anomalies.

Let's put this on the agenda for Monday and look at column wise now.

Okay.

So that's fine by me.

Okay.

Okay.

Okay.

Last time, remember, we talked about transform column semantics and we came up with one idea was transform whole columns. A completely separate command that had evaluations manifested tabular. Another idea which you'd suggested looking at was this. You called it column wise evaluated. Colin and I talked and we thought, well, maybe why don't we just make column wise inside the function. Just do that. So it basically does a single evaluation and it works. And so that's what lexically.

So it does. It changes the meaning of the slots within it. And it also like does. It works like once, like basically it replaces the value of the mean so it doesn't do it any time.

Right.

So that solves that N squared problem.

Yeah, it's lexical, it's simple. It works also with normal evaluation.

And how does it relate to element wise?

So. Well, this is by default element wise. So we don't.

A problem you didn't like with transform whole columns is say, you know, instead of we're, if we're doing instead of the minus there we say had a greater than or something. You had to use an element wise in the tabular whole columns and this, this, this avoids that. So, so it's, it's sort of a different, it's the opposite direction of element wise in some sense.

Now I know, but I'm asking, given that it is similar in functionality to element wise, is it semantics the same because element wise is a headdress.

No, this is not, this is a.

No, this.

I know. This is not a headdress. I know. So I'm complaining about the fact that it sounds similar to element wise because element wise is a headdress.

Yes. We could have it, we could have a completely different name for it, you know, by, by columns or something else like that. I mean, maybe column wise may not be the right name. Yeah.

Isn't the right name. I don't know. But one of one of the two. It can't, you can't have one of them. One of the Y absolute.

Yeah, absolutely. I, I understand your objection. And that's a, that's a, that's a good objection.

Okay. And then we thought that there are some situations where column wise doesn't return a scalar. And so you need to do something which is like the opposite of threaded to redistribute that. So we propose another symbol we like tentatively called weave.

Oh my gosh, the boat is sinking. We got thread and we've got weave and then we'll have knit. I suppose.

I mean weave, it's kind of a pun while we wait for a better symbol. But the idea is that weave is basically just part with.

I don't understand. If column wise returns something which is a list, why not just thread it?

Because look at the example just below. We knew you were going to ask that. There are some situations where you can't like, like it's fragile to rely on the fact that it's a list. It might not be the right length. It might be the right length for completely random reasons.

So the example that's problematic like that are part semantics. So if you do, if you have a list and you do, you know, part.

Look, the only way I can see this working is to have two different functions here. One that expects to get a scalar and one that expects to get a list that it will, will be threaded.

Yeah, that's, that's, that's what actually suggested as a possibility. So.

Yeah, as a matter of fact, you can't use weave without really using column wide, right?

No, no, it's. Yeah. So two, two separate heads. We need to come up with names. But, but I think in principle this is actually it's, it's quite workable.

I think it's quite. Will be quite nice to use.

Yeah. So we just had to figure out the names. But I think this is nicer. Honestly, I think this is nicer than the transform whole column solution.

I do too.

Okay.

I think it's more flexible and I think that the. I mean, I don't know what happens if you do weird stuff like nesting. Okay, so.

Okay, so I think nesting is an error.

No, nesting, nesting of just column wise is probably just. It's item potent.

That a possibility of doing something like that.

I think it's more like columnar block, something like that. Because it's like it switches to columnar mode.

Yeah, temporarily.

But I like the list, the list, the list and the value. Those are good. And we'll.

Hold on, hold on, hold on. I think that's very weird. I think that's very weird. Obscure seeming. Well, I mean I wonder if we could get away with using column wise list, column wise value on the grounds that then it's a different thing from. From just a pure.

Then it does feel different than element wise a bit.

Well, let's just see. Column wise value. How about. Maybe this is wrong as well. Is that the wrong meaning as well? I'm not sure. See. Yeah, because it's crazy because it changes.

The way the function is applied in some sense. So.

I see. Yes. Right. It's like reverse supplied. Yeah, it's really.

If you want to put it that way, it's really transposed applied.

Is it?

Not really. Because it's. It's. Okay, so reverse applied is a headdress, right?

Yeah. The problem I have with applied is that we're really talking about. In either case, we're just talking about a single argument, which is the association.

No, it's not a function. It's really a block.

What do you mean by being a block? It's not a block. It's a. It's. Look, it's the. There's hash A which is a, A A row element and there's the other hash A inside that thing that is supposed to be a column element.

Yeah, but it's. It's. It's not a function. Like a function is the first. First class citizen in language and it's not a function. Right.

Yeah.

And. And also. And the other thing is that if the outside function has a first and second argument, you don't want to repeat the first argument just for that.

Nice.

That's how it was. Yeah.

I think the nicer in this list are convince list and con. Nice value.

Yeah, but let's see. Column wise list.

I like the fact that list and value in the name kind of imply that this thing evaluates only once.

Yeah, A little bit.

Gets replaced with the value.

Yeah, but I mean the typical thing you'd see is X arrow. Column wise value mean of hash. It could even be hash X. Yeah.

That would be hash X. And typically, you know, a common thing is to do, you know, minus hash X.

You know, another. Another cool thing about this is that you can do missing fallback of hash A and then you get the thing we were talking about.

Hold on, hold on. I don't understand here.

You put the hash A outside the column wise value.

Oh, yeah, yeah. Right, right.

Because that, that you want element wise. Exactly.

Yeah, yeah.

That's not bad.

No, I think that's. And that allows you to. You could change the minus to a less or greater to compare whether it's and, and it works the same way. You don't have to do any specialized special things with element wise fallback. Yeah.

Right, but then, then column wise value would apply to all variables inside here, right?

That's right.

Does that look weird? If we have for example, a function. Oh, it looks fine. Right. There's nothing wrong with this. That's just fine.

Yeah. So it uses, it uses the same function replacement. It just does. It, it does a replacement before evaluating the function to do that same function replacement. But just on those parts.

Yeah, I mean that model. I suspect there are other uses for that model in other parts of the system.

One thing that is a little bit, one thing that is a little bit confusing, I think about names is that if you use column wise value and you have something like accumulate that returns a list, then you get a list in the values. If you use Commonwealth's list, then you get each value. So maybe they are. You see what I'm saying?

Like, so that's my question.

So what happens when somebody says column wise value quantile survey and they're going.

To get, they're going to get that thing of three elements and that's, that's, that's what they're, that's what they're asking for.

Just above the cut in the screen where you get 2, 9, 14, 18, 22, 28, 33.

Okay, so that means that you kind of treat it as a value and.

Otherwise you kind of treat it differently.

Right.

Which is a bit confusing in the names, though. So.

Yeah, well, if you give a list and it doesn't have the right length, then that's going to be an error.

Yeah. But I wonder if it should be something like column wise whole and column wise part, something like that, rather than value and list.

Sorry, I spaced for a second. Could you.

So the question is like the list, if you pass to column wise value something that returns a list, then it will plug in a list as the value. And so that's kind of confusing about the name.

I'm sorry, say it one more.

First example on your screen on top with the one which is cut off, that's a case of column wise. The first one is column wise value and the second one is column wise list. And the column wise value puts lists and column wise list puts value.

Column wise value. There.

Yeah.

Would do accumulate. Right. Because it's making just a value. Yeah, yeah. Okay.

And this, that would be conwise list. So the one that turns list is commonwealth value.

You see the confusion.

Locally this value here, I mean for every element wise element. Yeah, I mean this is a. This one is doing it element wise here. The second one is threading the values back in, right?

Yep. Right. Exactly.

Could it be commonwealth and commonwealth thread?

I don't like column wise on its own.

Okay, well, column wise value and column wise thread.

No, I don't think so because. Well, it's not a list. Yeah, you're right.

Yeah. Thread is not terrible because then we could have things like, you know, the, the errors you get if it had the wrong length would be like thread errors.

Well, so what would it look like? We could say, let's imagine that we said.

But that's, that's wrong. The function goes outside convoy.

Understand? So minus column wise, value of mean of hash A or total of whatever it is. That's not terrible.

Yeah, this seems. Seems reasonable to me.

All right. Okay.

Then there is another proposal which is kind of orthogonal. Like we grab both, which is what if we use the extended key syntax to have some kind of reserved names for things like old tabular row number, like ambient values that do not get displayed, but they're always there.

I don't know.

Because the cool thing about that, I.

Don'T know why that isn't supplied. That seems very much. That's basically supplied territory.

The good thing about that is that.

You can use hash. You can use hash.

Tabular, unless there is collision, then you have to prefix it. And so you don't have to invent weird new keys. You use the extended key mechanism.

I can't think about this right now. Okay. I really think this seems like supplied, but I might be wrong, which is bringing in ambient information into the middle of a function somewhere.

Yeah, but this, we can do it basically for free. Like free for every call to transform column or reduce tabular and so on.

Yeah, but I mean.

So, okay, it's a bit like making so supplied is a thing, you know, you do supplied of blah, blah, blah. Maybe we could have supplied work especially for transform columns that would bring these things in. Maybe we. We can look into that.

In the meantime, the good thing about.

Having this kind of extended key is that you can get all of them without asking them and just have them be ambient and they don't show up. That you don't have to say supply, they're just there all the time.

Yeah, I need to run. Okay. Yeah. You guys.

Okay, well, good. I think, I think we got. We got somewhere on this finally. So.

So that's.

That's good.

Okay, thanks. Good stuff. All right. Sa.