Live CEOing Ep 836: Language Design in Wolfram Language [Tabular]

Hi everyone. Welcome to yet another tabular design review meeting. So I'm not a believer in this extended key system thing. If we're going to do this, why don't we just have a thing called tabular key?

Are you proposing to rename extended key in general?

No, not at all. Extended key is a thing that relates to user defined keys. This is something that has nothing to do with user defined keys. These are reserved words. Right.

Only the prefix is reserved. That's why I was proposing to use extended key, basically so that there can never be any clashes and most of the times you can just use hash whole tabular.

So because the semantics is that hash will pick up things.

There's only one whole tabular it will pick up. You don't need to specify prefix.

I think it's a hack. But I mean in.

Well, that's the same hack we use for symbol names.

Yeah.

Well, the other thing we could do would make it more palatable if we could say something like that first element could be something that is not a string, for example. I'm not sure if that necessarily helps us, but I think it makes it a little bit less obscure. Like, for example.

I mean.

Yeah, I mean. Right. It's not really the same hack that we use for symbols because. Right.

It.

It's. That's a real context. Right. And they're still symbols.

Right, right.

I mean it's.

I'm not saying the same.

There's no key. There's no key called system in the tabular.

Well, the proposal of Carlo is that.

There is always a key call system in the tabula that contains.

It's not a keyhole system, it's a prefix call system.

Yeah, it's a prefix.

Yeah, exactly. There's always a prefix call system.

The key is called whole tableau or row number or whatever. But yeah, it doesn't have to be. It can be any. Like the prefix doesn't have to be a string. It can be any symbol or any. It could be a graphics for all I care. Like it's fully symbolic. Right.

Look, the main objective here is to be able to use hash whole tabular. Right. That's the main reason you're proposing it this way.

Yeah.

Plus the other. And the advantage of this is that.

You don't have to change the semantic at all compared to referring to any other thing.

Well, I don't know, it seems like a hack to me. I mean, maybe we shouldn't say extended key of all comma, whole tabular.

Why all. Because it's a symbol Anyway, why.

Why I would use tabular at this point.

Yeah, I would use tabular.

It's about.

That's terrible.

The whole.

We can't do that.

Okay.

I don't like this very much. I think this is going to get us in trouble. I think it's just like supplied. Basically. It's a thing very much like what supplied is trying to do, right?

Yeah, but we could use something like that for supplied as well.

Something like what?

Like this extended key. Extend the key working associations as well.

What does it do in a generic association?

Well, it works the same like if you. If you use extended keys in a generic association and there is no collision, then you can just use the last part of extended key.

No, that. That if I say it's not been done for association.

So that's what you would. Guys were just saying.

Carlo was saying not.

It's not yet been done. But we. We approved it.

Okay, so you're saying if I say extended key foo comma, a arrow X extended key. Is this the syntax? Do I need any other syntax here?

No, but I don't know if it's been done yet.

No, it's not.

I'll just check. It's not implemented yet. But the idea of it is in tabola you would just have to say A or B.

It's much harder to do for association than for tabular. I mean, you know, it's. It's the same thing. It involves changing the fundamental data structure for association.

And the point for association is then that rather obscurely you could end up with two A keys in association. Right? Yep, yep, yep. So what do we do?

Let's say that you.

That you merge two associations. You could merge with some function that creates ascended keys instead of.

Okay, so what do we do in this case?

Oh, well, in this case A will not resolve.

You have to say X A.

No, what do you mean?

And you could get.

What does it do if I do this?

Yeah, I think. I think in that case if you just do A. You'd get a missing. That says ambiguous.

Yeah, that.

Okay.

Instead of not found, like not, not. Not available, whatever it is. Missing. Missing key, whatever. This is like more ambiguous key.

Although it's a little weird to call it missing when.

When.

When the problem is that there are too many.

I think this is not properly thought through. Look, I think you should call it tabular here. If it's anything, could be internal.

Something we. I mean, we have internal backtick.

Well, what if. I mean, what if we made. I mean, as an alternative, what if we made supplied work, you know, so you do transform columns of blah, you know, of tabular, comma, you know, new column goes to supplied around your function.

Well, what would that actually look like? And say supplied.

We'd have to give the elements that we'd want supplied. Right.

I. I don't think we should be designing supplied in this meeting.

Yeah, me neither.

And I also think that supplied is not really. I mean, I was looking at the, the breath page for it, but it's not really designed yet.

Then this is the answer.

Okay.

I would favor this with an all here or maybe an automatic there.

Yeah, I think. Do we want something slightly more obscure thing like inherited? I don't think either all or. I mean, automatic maybe, but it's still kind of weird.

It is weird. This whole thing is weird. I mean, there's bowl, for example. It's another obscure one.

Well, I think inherited would be better than either.

Well, certainly better than full, because the.

Whole tablet is certainly all or full, but the next row. Why would it be all?

I mean, on some level it should be like information, but you probably wouldn't go for that.

No. Yeah.

Inherited makes. I. I kind of agree with you, Itai. It makes.

Yeah, because inherited, like kind of. I mean, we haven't used it that way because inherited just means take it from the outer scope.

What about none? Right, It's. We're. We're looking at the. Not any part of it.

No, I, I think none is wrong because, you know, you might expect that something like none behaves like nothing maybe there and then gets deleted.

Huh. That we have a symbol called anonymous.

But that's an attribute for. That's an attribute for variables, isn't it?

No. Represents an option or other value that indicates the absence of a name.

My God.

And the one example is using fine channels. There are no notes. And the one example is using fine channels.

You know, I've started to like, inherit it more and more because this is something that you get without having done any work for, so you inherit it. I guess maybe it's too much of a political commentary.

I liked automatic.

It sounds like kind of a default behavior type thing.

Yeah, you get it automatically all the time.

Yes, I kind of agree. And you'll probably never type this because you'll just type hash whole tabular.

Yeah. And the likelihood that, you know, this collides with something that comes from an external data set is very low. And you probably want to rename it right away. You're using both.

Well, I mean, things like length. What the Hell, does length even mean? I think they've got to be whole.

Well, length is constant.

You get everything. Like for all rows, length is the same. It's the length of the column.

I mean, it might help if we said tabular row number. You know, if we called it that will reduce the chance of a collision and make nobody ever look under the, under the hood to see how this is actually working, more or less. And I don't even know what length means. What does it mean? It's a number of rows, number of rows. Okay. Also, why don't we say, I mean, tabular row number. But now what is the tabular total rows or tabular row count?

Row count?

Yeah. Okay. Okay. What's row fraction?

It's row number divided by row count.

Okay.

Possibly minus one. But you know, there's two versions. One is zero index. One is one index. But can decide which one to pick.

I think it'll be okay if we add the word tabular here and treat this as quite experimental.

Yeah, I mean, we, you know, the thing is, I think it will be pretty useful for us for implementation. We might even decide not to document it right away.

All right, fine. Extended key, automatic. And then everything has tabular in the name.

Okay.

And this business about caching them, that's an internal implementation issue and we should make it right.

Yeah, no, but the, the, the column. This, the point about ordinance squared is, is that if you start instead of using column wise value, you use these. You have no guarantee that what you're typing is not N squared because there's no automatic caching of the things you compute out of this stuff.

Well, why don't you just compute them? I mean, I understand if you say whole tabular cache, it.

No, no, no, no. That's not my point. My point is like for the thing, remember last time we were looking at. You compute the mean of a column, right? If you use column wise value, you have a guarantee that that mean since it's constant, it will be computed only once. But if you write your own mean by using whole tabular, blah, blah, blah, divided by length, whatever, I see no guarantee that that's going to be.

We agreed on column wise value. Is that right?

Yeah, yeah.

This is another thing, right?

Column wise value and column wise thread. Yeah, they're already. Which. Okay, good. Okay, let's go on. Ready to go on?

Yep.

Okay. Delete missing.

Okay, so this is something that we.

Left out because not a new function.

And we realized early that was difficult to extend. But then we also, while talking to.

Carl, today we thought that it would be weird not at least to think about what you want to do with this. So delete missing has a problem for us with tabular. It is designed with an expression in mind. So it's thinking about a tree. It has, you know, by default it deletes what is missing at level one. So if you nest missing in they're not deleted, which is my second example.

You can decide at which level of.

A tree you will be deleting missing.

And you can also say if you.

Have a missing in the subtree, delete the whole subtree at a higher level. Which is my last example. Delete missing, blah blah, blah, 1, 2. So basically the 2 comma missing gets deleted because there's a missing at level 2.

So none of this is useful for tabula because tabula you never want to.

Treat to basically destroy rows. So you only want to look at level one.

And what you want to do for tabular is typically mainly two things like delete a row if every element is.

Missing or delete a row if any.

Element is missing with some special case of. You might just be looking at some.

Columns that are more important that you want to specify.

And it's kind of clunky to kind.

Of fit that design in the current delete missing design.

So yeah, one possibility is that we.

Kind of use special symbols or strings to extend it.

Another one is discard. There is just no way, no way.

What is discard is already in. It's the opposite of select. You approved it. It's the opposite of select.

It is, yeah.

About three weeks ago we had the longest.

That's fine.

So maybe we should have discard missing. I don't know if we can't have.

Because delete missing tab all goes into.

The level spec and it's not a level spec at all. It's more the third argument of the it's missing.

Okay, so the real things you want to do, you know, remove a row if it's all missing. Do you ever want to remove a column?

No.

Are you sure?

I mean that's not the way you do it. But I mean these, these functions, how do you do it the row wise? I mean if a column is all missings, you remove it by saying a goes to nothing or something like that. It's not.

It's not, no.

But these are stupid question actually.

Like you could say delete a column which is all missing or the delete.

A column that contains some missing values. I want only to deal with the column.

The reason why Is strange to think.

About it this way is that yeah, you are basically destroying the shape of your data. You are not destroying examples like column are more dominant in terms of semantic. But it's not something completely impossible to be able to be me wanting to do that. It's just not as common, I think.

Okay, let's look at. I see Ethan on our livestream mentioning deleting failures as well as missings. I just wanted that Literally yesterday I was using confirm and close. It was beautiful. But I wanted to delete the failures that had come from the confirm. Actually I was going to look up discard.

It's actually the nice way to do that. Yeah, you just replace missing queue with failure queue.

Yeah, yeah, but still delete missing is a very nice function. I use it all the time. I mean I should know and I don't for enclose if I wanted to any failure to say nothing ampersand. You just say failure, blah blah blah, nothing ampersand and that will. If the thing that is enclosed. If the. If the resulting thing. This is a list. If you say how do I generate a failure? Well. Oh no, that's not what I want. I want enclose for each. I need to confirm on each of these. Right?

Yeah.

What the heck, why didn't that work?

Yeah, you remove.

You remove confirmed. You should wrap the if should be wrapped in confirm probably to simulate what you want to do.

Also I don't think you need the ampersand with nothing like applied to things.

What's that.

On match brackets.

Yeah.

Yeah.

This is throwing. Yeah, you, you, you confirm.

You probably want to confirm inside of the.

You also want the enclosed inside of it.

Otherwise that's going to work.

Oh yeah, yeah. Okay, fine. I'm writing a silly piece of code. Okay, but that's the way that you can get rather than generating it and then removing the failure. I mean there's a certain argument that we should have a function delete failure just like we have delete missing and have it be a bit more general. I'm not sure mean. Look what I'm Otherwise I'm just doing delete cases of underscore failure. Yeah, okay, but what you're saying is.

Or discard failure two would depend. You know they both have an operator form and they can both be applied on a. On a list.

Okay, so how do I delete? I have it. There's a delete rows function. Yes. No, what is it called?

Discard. We designed that for this reason. Yeah, or you want the positionally to police position I want to.

Discard. Wait a minute. This. At the top level, select can select a row. Correct.

Exactly.

Okay.

I mean, select with the function.

Yeah. Because you need. Select will basically scan through the rows.

Select is applied to the rows. If the select of a tabular tests the function on each row.

Exactly. So the tabular is.

Is seen. It's in its normal form basically as a list of associations. You can imagine it that way.

Okay. All right, so this. I don't understand. I mean, in. In it will be totally bizarre to have a discard missing that basically does the exact same thing as delete missing. Right. So let's just try to understand what. What you're talking about. You. You're saying I want to delete a row if there's anything missing in it. I want to delete a row if there is. If it's all missing. Only if it's all missing. So, I mean, clearly, you know, this will work. How common is this operation?

All true.

Not. I mean, I'm now speaking about data that comes from a relational database. I can't speak of other imports, but usually, you know, you're not going to have multiple rows that are all missings, otherwise there's something horribly wrong with the data. It's relatively common to say delete any missing or any missing out of these three important columns, which are. The first can be done. The second no with it missing. Like.

Okay, so let's. I think that case seems much more important. The case where if there was no name in that field, for example, in that record, then just delete the record.

Yeah, right. Yeah, yeah.

The.

Any true is also useful if you are piping with some analysis and you know that your algorithm cannot handle missing. Yeah. So that's the safety step.

Yeah. Okay. Okay. So let's see where you know, the things of elements are. Delete any row that contains any missing. Etc indeterminate, you know, whatever else.

Yeah, yeah.

Okay. And the problem is that we've kind of cooked our goose because we're using transform missing. So the natural thing would be a delete missing. But we already use delete missing for something that works on ordinary lists.

Yeah, yeah.

Which again, there's nothing wrong with ordinal.

Lists like select discard. They generalize. Fine. Is more like tree versus the data sets. Yeah, yeah.

The level spec.

That is annoying here.

Well, the level spec is annoying because it gets in the way of having a something. I mean, so this is a possibility. Right. Is to put a special second argument, that's a bit of a hack.

Yeah. Which Is what I had there at the top. Yeah.

And what Carlo was saying, like it's a hack.

And what would delete missing do by default in that case?

Well, nothing, because delete missing by default is delete missing comma 1 and tabular.

Is always a list of associations so.

That you cannot have, you know, an.

Association with missing in every key is not equivalent to a single missing in place of the association.

So you can never fit the bill.

For the one argument for.

Okay, all right, so hold on. So what other functions do we have that deal with missing? Just transform missing.

Right.

Especially with missing, we have synthesized missing values.

But yeah, that is completely different. Beast and missing fallback is a new one that introduced. That deals with missing as an expression, not as a stuff contained missing.

Okay. And transform missing is something that replaces the missings with something else. So we've really got two cases. We've got to, you know, if there's a missing, just get rid of the row. I mean, There's things like this I suppose we could consider.

Well, at that point it's probably faster to add a new argument, a new.

Value in the second argument of transform.

Missing so that the operator form is a transform missing nothing or transform missing, delete, discard or whatever.

What is it doing then?

So we could have transform missing kill the whole row as a special operation.

How? Because this is just saying what to do with a column with something that's missing.

The C missing by default is using the same operation of all the columns. So I guess if you do that.

With transform missing, it will be basically any. Any true. Unless you, you.

I don't understand. I don't understand the. If I say transform missing. And then I say a tabular here and what would you imagine was happening here?

Let's do a simple case.

Let's say A goes to delete. For example, let's pick a random string here, like string a rule delete. Yeah.

So any other operation would be like.

In the column A, replace the missing with this operation. So I guess this operation would be. Instead of imputing it, just remove it.

And given that you cannot remove an.

Element without removing the row, this will cause the row deletion.

So basically this row deletion, it might only be in a particular operation you're doing. I see. Because this is transforming the whole tabular.

Yeah, yeah.

You cannot punch a hole in the tabular.

Yeah. So you need to remove the row.

Could we combine this with a discussion about like deleting the rows?

Yeah, that's what I'm thinking. That's what I'm thinking.

Because, you know, we have, you know, we could kind of think about, you know, delete rows and then you can either say, yeah, I can get a, you know, if it has missings.

Yeah, exactly.

Or if it's greater than 17 the condition. But that discard. It's exactly discard with a slightly longer name. And the condition you would put in the second argument is exactly the same.

As discard because it's the predicate. What you were saying. Yeah, greater than that.

Why is that? I don't understand. How do you discard.

Discard, hash a greater than true.

Hash a greater than whatever. Okay.

So I put, I put this card here because I mean, if the cleanest way we have to do this.

Is discard, maybe just let's, let's use this card here.

I mean, discard and then.

Yeah, same tabular.

All missing. Yeah. See, if we, if we did that, if we had functions any missing and.

All missing with a Q at the end. I mean, is this returning true false?

Well, it's like all true and any true.

Yeah.

Okay.

Yeah, so I don't quite know how that would work. Hold on, how would that work?

Well, any missing would just expand to any true of missing Q. I mean.

Okay, so what would it be? Any missing of a missing B would give true. But what does it all go ahead?

No, no, exactly.

And all missing of that list will give false. Right, but teaching two new symbols is.

Enough added value compared to not expanding all true missing. Not having a shortcut for true missing queue. Yeah.

Especially considering that if you teach.

People all through missing queue, then it's easy to teach all through failure queue or all through auto queue.

I mean it generalizes very well.

And the one other case I've seen, and this is in, we had Truman working as an intern showing his data science stuff. And one was delete rows with more than say five missing. And that works with discard. With, with count. It's more complicated, but you can still do. You can do basically anything you want to do with discard.

All right, then let's just do that for now. We'll. We'll pencil in talking about this further If. If it seems like one's doing this all the time. Okay. Okay, what's next?

I want to do a quick review.

On what we have yesterday for transform missing so can hopefully close the design.

So I put here all the string that we have. I mean, median mode was kind of.

Uncontentious if I remember correctly.

I had a random string that you proposed to Replace with random element.

Yep.

So after discussion with Carlo, we go like, do we want to actually split.

Between random choice and random sample?

Yes, to distinguish with replacement.

Without replacement.

Well, I mean, random sample, I don't understand. It's going to use up other things once.

Exactly. And so that has the implicit problem.

Like what do we do if we run out of elements?

If you have like ten missing elements.

And two missing each place. Missing. Okay.

And the point of that is.

Is it the same missing or a new missing?

Probably a new missing.

Well, every lecturing is the same lecture.

These missings for which you don't have elements will stay missing. And you may just issue a message that we ran out of your options because there were too many missings.

Does random sample actually make sense? In other words.

Now that I'm telling this to you, I'm thinking about that. Do you get any beneficial properties of.

Not putting a replacement in? I'm not sure.

It's basically a data set with maximum.

Two copies of each element.

I don't think it makes any sense.

No. Yeah. Now that they say this allowed. It was more like a language consistency than.

Actually, I don't think it makes sense because you're, you're, you're, you're saying is, you're calling a random sample. You wind up with multiple copies of various things, which is not really a sample, then. So, so I think it doesn't make sense.

Yeah, that's, that's, that's true. That's true.

Random element.

Let's stay with one random element. Okay.

Random variate.

Random variate.

Yes.

There is the one with the distribution.

After it is completely fine.

We were floating.

Do we want the naked string?

I think.

And if so, what do we do?

Christopher was suggesting sample from the normal distribution.

Which is fine as a fallback, but then we have a type problem. Like what if it is a column.

Of strings stays missing? Do we define a default distribution for every type?

I think we should try to define a default distribution for every type. Yes, I think we should try to define a default distribution. But for strings that's tough.

Well, if it is a categorical column.

Then it's tough with the category list.

Then it falls back to random element.

I guess maybe string should just be random element.

No, I don't think it should be random element. I think it should be random element with the inferred distribution. See what I'm saying?

Yeah, but isn't, isn't this in distribution.

Exactly like calling random choice? That will respect.

That's right. That's right. In that case, I think yes, it is. Yes, you're right. So yes, I think random variate should be type dependent and should try and do something sensible. And I think it will be widely used.

Okay, I'll make a table.

Yeah, I mean in general, I think. Yes.

If we can find these pockets where like maybe a string setting can be beneficial.

It could also suggest functions sometimes I think like when we have stuff that does the right thing for every column type.

I think tabular is going to be.

A good test case for these scenarios.

But yeah, I can come up with.

A table for every type. Type of something here.

Okay. Then at the end, what do we.

Do with the estimated distribution syntax? Do I put it in? Yes.

No, the naked variables.

What naked variables? What?

Oh, the random varied normal distribution.

Miss.

I'm not a fan. I would prefer. Prefer to see blanks there, if anything.

I think blanks.

It's easy to implement.

Let me try.

Okay.

And then the rest.

Maybe you could treat it as equivalent and just let's not document the. The symbols there.

Because I. Yeah.

Where you say a such and such, you know, such and such distribution with 18 degrees of freedom and you put the 18 in there and the other things are blanks.

Yeah. Okay.

Then we canonicalize to next previews and.

Not backward forward in terms of look at the previous and the next non missing element, next element.

Then let's call it next element, previous element.

Okay, fine.

Along with random element.

Okay, yeah, that goes well with random element. Yes.

Interpolate.

I also think that's not.

Not problematic.

There's going to be probably interpolate. I think it should be called. Shouldn't it be called the function as inter.

Interpolated.

Interpolation. Can we call it interpolation?

Interpolation.

Okay.

Interpolation.

Fine.

Kind of like it's a random element or it's an interpolation rather than.

Yeah, yeah, no, no, that's true.

We're always using a noun here.

These seem useful. Sensible.

Okay.

The only thing that I see here in terms of extension for interpolation is.

If we have something time series like.

Where one column of a tabular becomes.

To be honest, I think we'd be better off putting the interpolation order as an option to transform missing rather than putting it in that weird list form there.

Is it used in any other case?

Not interpolation order. It's used in interpolation.

No, no, no. Yeah, yeah, it is using any other.

Case in transform missing. Except for interpolation.

Not. Right. If we had something that we could Use for the distribution here, Go ahead.

The other thing, you could also use a method for interpolation. So having this in, you know, in that list maybe makes sense to group them as options for interpolation together.

I have. I see what you're saying. Yeah, there are other options for interpolation.

But Julio, apart from the. You were saying time series. But in general, if you have something that is like a scatter XY scatter, can you give columns to interpolate and mean only interpolate Y based on X or something like that?

I mean, what would be the interpolation function? Yes, in this design.

In this design, I mean, Right.

So you're doing column wise, all the missings are being demysified column wise without regard for other columns. That's the basic point, right?

Yes.

We need to think that could be.

An option.

Piggybacking on what Roger was saying. Instead of interpolation order, we could have.

Something that is telling you what is the reference column.

This is getting crazy. I mean, okay, there is a missing in column seven. It obviously makes sense. Why isn't there a mechanism for filling that in from column six? In other words, if you know, if the location, if the city location is missing, fill it in from the country.

Okay.

Instead of creating a crazy syntax that's exactly missing fallback. So if you do transform column with.

Missing fallback, you get exactly this behavior plus any other special sugar that you want to put on top.

And I'm going to have examples, I'm.

Going to have examples of these in the documentation page.

We just want to avoid duplicating the design.

No, I understand that, but I wonder if transform missing should allow, I mean, if you say quotes a what could there be a function on the right hand side there? Yes, there can be. Right?

There was. And we remove it to simplify the design. Because then you have the thing. When do you apply the function to the whole column? When to the whole tabular, when to the single row? When do you evaluate the function immediately or not? There's a lot of questions that come with allowing the function.

I see.

And I think the design is much tighter now.

Then let's not try and do multi column stuff for now. Okay?

Okay.

Let's have it be local to that column.

Okay, cool.

Okay, now we have a tight design. I think this, this is also mostly implemented.

Okay.

All right.

I have a. Sorry, Julio, but like, I have a couple of questions about, like, do we treat in all of this the date columns as the same as the numeric columns? Like, is There any kind of basically random variants with using UNIX times basically and then converting it back to date?

Well, I have to check in the statistics.

There are lots of statistics functions that work now on the dates and also you have random date, random time, if you would like to simulate, you know, dates and times.

Yeah.

Are distributions working on. If I do, can I pick a.

Distribution on list of dates?

No, there's an object called data distribution and an object called time distribution which you can distribute the dates and times. And you know, if you want to use estimation, you can also estimate, you know, over the list of dates and find, you know, automatic selection for that.

That looks like a sensible default for random variate.

Yeah, yeah, yeah.

And the other thing I was wondering is like for strings, I think not probably not terribly useful, but something like random word might come up.

Look, here's what I would do for strings, okay. I would look to see if there are, I mean, with some heuristic.

I.

Don'T know, I mean, arguably the heuristic could be dangerous. I mean, with strings, if they're like a big thing with two different strings and that's it.

I would argue that in that case we should be smarter with the type system and have that as a categorical column, not as a string, like a generic string column.

So maybe random verit just doesn't work with strings initially. I think the idea, I mean, it could be a genome string, it could be this string, it could be that string.

I mean, for data from the wild.

I would treat string same as expression.

Something super wild because this is also.

How people use it to store arbitrary stuff in another.

Yeah, and I think by the time we're making a random variat of that arbitrary stuff by training a learned distribution on that thing, it's going to be just garbage. So we should.

I'm not saying just random, I'm saying like as an option for strings we have a function random word, so we could have the string random word.

I'm aware, I'm aware of that function and that's it.

I'm not saying estimate distribution or strings.

That's.

That seems crazy to me as well.

Yeah, but random word is not going to cut it because it's.

I mean, how often a string is going to be a word compatible with random word?

Not do it. We don't need this. We've got enough features as it is.

If somebody wants random word, they can, they can, you know, insert column.

A column, yeah.

MF Callback.

That's a callback.

Okay, now can you open the Last one before anomaly detection.

I think it's in term of time is more relevant.

Okay, here.

Yes.

Do we want all these functions and I think yes.

To support a tabular column.

And in general, are we going to.

Push for functions to support tabular column?

Because I'm putting here some example about.

How you could use these together with column wise thread which we just added to, you know, pipe stacks.

You're already inside.

This is not a reserved word. Right. This is just. You just made that up, right?

Yeah.

I'm seeing how can you pipe these.

Functions in transform columns.

And with column wise thread, you know.

This function can easily be fed individual.

Columns and you can create new. Because the point is this is the.

Syntax to create new columns in the tabular.

I agree.

So I think, I think that these, I think we need a lot of list based functions capable of taking a, a tabular column. I mean I would even argue if it was possible with UP values that we should just have the thing, you know, look at a whole, you know, UP values for tabular column basically listify itself if it doesn't know what else to do.

Well, yes, but it's, it's, it's. That can be dangerous. And so we have to figure out cases where it should not. And I've set up, for example, a good example of that is free Q of some expression comma underscore tabular column.

That will fail if we.

Well, if you listify. If you listify the tabular column.

No, I understand. Then your goose is cooked at that point.

Right?

Exactly.

Exactly. So that's, that's, that's where you have to be careful is for stuff like that.

Yeah, but I don't think that Simon.

Was saying let's, let's do it for every function. It's like let's have a list of.

Well, I was kind of saying that, but I think just inventory what, what functions this makes sense for.

Yeah, well, we said there, there is an inventory of that and, and you know, I'm adding to it continually. But there's, there's. There's a fairly big inventory and I've, you know, we'll, we'll have a way of, I'll set up a way of registering functions that should just listify in the upcode for tabular column. That's not too hard to do.

Oh, nice. Okay.

Okay.

So. Well, you have four minutes.

So I don't think this is why.

I left a normal detection for last.

If you want just to tease your brain with what's to come, you can Open that. So last time I was explaining also.

In the last four minutes, what's our.

Current stack for anomaly inspection?

So the whole idea is that right now we have. Speaking from a tabular point of view, we have a single way to deal.

With anomalies which is you train a distribution with learned distribution on the whole tabula. Yeah. And then it's a probability based way to discard them.

I know, but you don't want to have to train it.

Well, you kind of always train it. Yeah.

Because if I say interquartile range or Z score, you have to compute the mean and median.

Yeah. It's just a very fast stupid training.

Also there is here like let me.

Make a point of distinction between the mindset of machine learning and statistics. So statistics is like you have a single way to apply it.

You do, you know, find the appliers.

You don't make a function.

Exactly, exactly.

The Z score applying function.

Exactly.

While the machine learning point of view.

Is you train it once and then you apply it.

Well actually to be fair, you know, in statistics if you're trying to work out sort of P values based on some you, you would find the distribution and then work out p values for observations. So it's not actually that different.

No, no, it's more likely that is done in place. Yeah, like the statistic point of view is you do a computation and then.

You use it immediately.

The machine learning is. That is a special case of training and applying.

I know, I know. What do you need to do here? Yeah, go ahead.

Exactly. So here we have, you know, the current design of find anomalies is this.

With the, you know, one argument form is training and then applying. So this is the one in place basically then you have two argument forms which is example prop and function examples. I don't know why it's called data.

Here, which is the one that lets.

You use a pre trained one basically.

So barring the weird argument order here.

My main issue with design here is.

That it feels weird in the experiments.

I've made to stack together univariate and multivariate paint here, the new one that we care about that are very simple and very used are applied on the single column. Now what happens when you have multiple columns?

Do you apply the univariate one to all of them?

And you do an and an or like any true, all true.

Do you switch to the multivariates for the method that allow that, you know, like you could have a circle, like.

You know, an interval becomes a circle or an ellipse.

Hold on, hold on. First of all, we've got. I mean, what would we naturally call find anomalies if it was. It's finding anonymous rows for a given column. Is that correct?

Yes.

I mean, on a tabular column, again, implementing this on a tabular column is trivial. You have a method, you find out.

Which element of the column is anomalous based on the method.

So again, using finite anomalies in transform.

Columns is also easy because you plug the row or you add the column with all the binary flags based on the specific row value compared to.

What are you going to do with the columns with rows that are anomalous?

Okay, so this is where it becomes.

A bit wilder compared to missing.

Because depending on what you do, I mean, anomalies unlike.

Why aren't we calling it transform anomalies? And why aren't we having it define.

We have a tentative design for transform anomalies that carries yet a larger syntax problem. Because then you have to specify column.

Where and what.

Like column A.

Then I know. And what makes.

Where are the anomalies?

Exactly.

And then what you do with that.

So the general case starts to get big.

I left this as a final thing, because I'm still working to find good proposals and good ideas here about the design.

Giulio, I think that the what column could be solved if it's not something that you ever apply to a whole tabular.

To what?

Like to a whole tabular? If it's something to be used either with transform columns or insert columns, then.

Yeah, but okay, first, it's weird. Second, I'm not fond of having too.

Many new symbols that only work in plus one columns.

I'm not saying it only works on transfer columns. I'm saying this in general would work on lists of things. And so you plug it in.

Okay, hold on.

If it works on this, it works on.

Very basic thing is this. I would like to take a tabular and both find those things which are anomalous in the tabular and return a tabula, which is the anomalous rows in that tabular. Or I would like to just say, hey, anything funky going on, just throw it out and return a tabular.

This is already in the design of find anomalies. If you open it, the prop argument.

If you have the current one, it has.

Yeah, anomalies and then non anomalies.

Which is basically the two things that you wanted.

That the Boolean list could also be.

We could also have an anomaly mask, which is now the bit vector, which is a more modern way to Return an actual bit vector than a list of truefold.

Okay, listen, look, I don't understand. Why find anomalies of a tabular with perhaps some additional properties, there wouldn't be a fine default. Now, the issue is what's the method of finding the anomalies?

Right, yeah, yeah.

I'm just worried about the messy design, the clash between the single column and the default, the current default, which is the.

The whole tabular thing, basically the multivariate distribution.

Well, I mean, why not do this? Why not say, okay, why not do something like this, find anomalies, Tab arrow A, for example? Because you're going to have to specify the column, aren't you?

Instead of a second argument, you put.

It in the first argument.

That's what I'm suggesting. So you say which thing you give there, and then you say, okay, then it's the prop. Okay? Now, were you to give a function, what does the function return? I mean, you might be able to say find anomalies. And then it's a whole elaborate function that applies to the whole tabular and then the tabular and then the prop.

Yeah, but I mean, there's a reason why the only function that you can put here, a normal detection function, is.

A wrapper around learning distribution.

Okay. I mean, we could put a function that says. I mean, that's where I think we need to look. Because that, I mean, if we had a function that basically means, you know, that is the Z score thing. Okay. How would I remove elements from a list that are above some Z score value? Okay. Do you understand my question?

Yes, I would say I would do.

Pick of list, comma, standardize of list and then in the pattern, you mean or like unit step of standardize of list minus something. That's the efficient thing I would do. That's probably a less efficient, more easy to type thing that you can write.

Well. But I would like to suggest that this is a function that might be useful anyway. I mean, this is like standardize, except it isn't standardize element Standardized, I think.

Is the absolute value of the element.

Of the list minus the standardized. No, sorry, is.

No, it's not mean invariant. 01.

Yeah, but it.

Right. So it's trying to make the variance 01. So this would be.

Yeah, but it's exactly the element minus.

The mean over the standard deviation. Yeah, Goja.

Yep.

Yeah, it is. It is. But that. For that case, what I'm saying is.

By default, yes, by default, I, I.

Would like to say something like outlier, let's say remove Outliers, okay. Of a list and then it's something where it's a Z score thing or something else, some other criterion. How do I compute? I mean with respect to a list. It's very similar to the anomaly detector with respect to this list I learned the distribution. So I know what, how to define a Z score because I know the distribution in that list. You get what I'm saying? And now I come in with my anomaly detector and I'm saying my anomaly detector is I've got this list. I know what a Z score means. Any element that's more than 17 is Z score greater than whatever it is. See what I'm saying?

Yeah, yeah.

I mean I think that function is.

Pick like select from a list based on another list.

No, I know, but that's not what we're doing. We don't. You don't have to. So you want to.

You want to remove them or you want to change the.

I'm not sure. I'm not. I don't know.

I mean because it's either, you know, they're this kind of two concept like trimming so you can trim the list, you know, if it's above something or windsorizing and in Windsorizing you operate on the quantile, you know, the quant which is, you know, related to this values.

Well, I think what we want find anomalies normally returns a list of anomalies or something like it. They're based on that prop argument transform anomalies. Is something different relevant for tabular where you're doing things with the anomalies and transform anomalies should also work on a list. I think.

Yes.

We could also have deleted anomalies and.

Find anomalies are like kind of sub cases of transform anomalies.

Okay. I put together transform anomalies and then you see with this idea of the.

Top goes to a then CP design holds.

But find anomalies find anomalies for tab.

Goes to a. Yeah, I already have a tabular there.

Yeah.

And I think that having a function other than a machine learning anomaly detector as the first argument or anomaly detection function as the first argument of a find anomalies makes perfect sense.

In that case, does it have to be a.

Probability distribution to respect the thing with, you know, acceptance threshold and whatnot?

Perhaps. So I mean what does.

If you put a normal distribution.

Okay, hold on. What does it return? What does it return? Function return True, false.

But you can also put in a.

Learn distribution in that. Yeah.

So I mean I don't claim that.

The Design is the best design here. I don't love it. It's very ad hoc around anomal detector function and their distribution.

No, but this true false thing is also, you know, that's like a. A given. You know, if you have a look if what you have is the first argument of find anomalies. Is P value. Do we have what. How would we call P value? Is it called Z score?

No, we don't have P value.

But we could even go simpler at this point.

We could have height greater than like zero. Exactly.

Well, if for tabular is going to.

Be like hash height is less than zero is an anomaly.

Yeah, I understand. But for just a list in general it will be something like this.

Yes, yes, correct.

We can generalize this a bit.

I have no idea why we shouldn't allow this. I mean this seems like an obvious thing to do.

Yeah.

In this case a septon tripod will not make sense.

But we always have the true false. Yeah, so it's not a soft thing, it's a hard thing.

Sorry, what? For what?

Which the final anomaly is being built.

Around anomaly type of function which is.

In turn built around render probability.

It returns true false by cutting at the distribution at the probability.

I understand, I understand that.

So in this case we will just.

Ignore a certain threshold and it will be a hard. A hard problem.

No, I don't think so. I think what we should do is if you provide those things, there are additional things given to this function, just as there presumably are for anomaly detective function. See what I'm saying?

See this?

Look at this.

You know, but I don't understand your.

Function, what it should be provided or. Or return.

No, I'm saying when there are options to the find anomalies function. Okay, When a prop is given then here instead of just having a hash one that is the data that it also has.

Oh, but no, but that function is.

One that is producing the property that is produced in the rare probability. So it's not. It's not a matter of what is fed to that function. What is fed is always the data. That function instead of giving a true false, it gives you. I don't know, a curve and you get truthful by cutting that curve by thresholding.

I understand, I understand, but I'm saying that, look, just. I think that you can flow through from find anomalies. The prop argument of find anomalies can be flowed through to whatever is there in the first argument. Whether it's an official anomaly detector function or whether it's an arbitrary user function.

Okay, I understand what you say, but.

I don't think that lies. Let me. Let me circle back to this at the next meeting because we over time.

All right.

I will come with the implementation.

Sounds good. All right, Thanks a lot, guys. See you all soon.

Thank you.