Live CEOing Ep 812: Language Design in the Wolfram Language [Tabular]

Okay. Hello everyone. Welcome to another OR from Language Design Review. This one is another one in our series about tabular. And the topic for today is how what to do when things don't fit in a tabular. As in, well, let's look at what this is doing. This is saying map to string, string to number. Okay, that's. That's the strings. But now if we want to make this a number that's not going to fit in a machine integer. Right?

Exactly right.

Correct.

You've got to have something inside the missing there. In my opinion, to say, you know, missing. You know, in other words, I don't see any reason our missing processing stuff is just looking for the head, is it not?

Yes, yes. And that's, that's sort of. That's all we're, we. We talked about trying to, to set up something so that we can, you know, carry different tags around. It's possible, but. But the. But. But it probably won't propagate well through operations.

So what. I mean, yeah, you know, missing plus missing. You know it's going to be a missing. Presumably.

Yes.

I. Should we generate a message.

What's missing plus two. Is that also missing?

Well, that's one of, that's one of the questions for these control parameters is in, in the kernel it's just two, which is, you know, and that's pretty disastrous in tabular. If you do that, it's just going to be another missing.

Then we've got to have a way to do that at the level of kernel computation.

Well, that's. Yeah, so. So the way to do that is you can convert those things to tabular column and do the operation and then convert back.

That's weird.

Yes, it's a bit weird, but.

Well, is it obvious that we should not make two plus missing be missing?

No, it's not obvious. And I thought about that. You know, fiddling with the basic arithmetic in ways like that is a little bit dangerous. But I think it might be possible to write. Since we sort everything with plus I think we can do that. The, the, the complicated case is, is when we get to listability.

Yeah. What's the issue? So you say A B plus C missing, is that right?

Yeah.

What should you do in that case?

Yes, Well, I think, I think the, the second element. I mean, you must have had some things defined before. But you know, it's. Assuming those were just symbols then. Yeah. That. That last element I think should just be. Be missing. But, but again, that's something that we want to have a control parameter For. Because people may be depending on the old behavior.

Just like we have a function delete missing, maybe we should have a function called conform missing or something or compress missing or something like that. Because I'm not sure that it's. I think this could be useful and we've had it for years like this and why not leave it and then have some, you know, processing function that is essentially called automatically inside tabular that will do things like turn this. Because it needs a choice. I mean, it's not totally obvious to me that this should just, shouldn't just be B as opposed to B plus missing.

I think, I think that it's pretty obvious to me that that's not the answer.

Fine.

Yeah. Right. I think you'd want to replace missing with zero first to do that.

Now here's an more interesting question. If you had a. The list a B + missing, is the result missing or is it a list of two missings?

Yes, that's, that's where it starts to get very tricky. I think it has to be a list.

What does indeterminate do?

Well, indeterminate known. Indeterminate is known to be a scalar. Right? It's.

It's.

It's nan. Right. So it, it should, I think that behavior is pretty obvious.

Okay.

I don't think we should break the list ability for the fact that there's a missing there.

Okay.

No, no, I think we have to, we have to keep the list ability. And, and you know, if somebody wants that to be a vector missing, then there's Adam's whole branch with the non scalar stuff. That, that would give that possibility.

That's threadable, right?

No, it's non threadable.

Non thread. Okay, that's. I can't remember what, what the name of it's become, but that's. Yeah.

So this would be.

No, it's an attribute. It's an attribute.

Attributes.

Yeah.

So you, so you would need to. You know, I guess we don't.

We can't really do that.

Right. You'd have to muck with the.

Well, yeah, yeah, we don't have a local attributes. We don't have block attributes mechanism, which perhaps we should have. How would this work? I mean, it would usually be a set attributes.

Yeah, well, you could block missing and set attributes locally on it. Right?

Yes.

And I, I won. Well, here, let me test this in Adam's branch. I want. Well, this is the prototype build. What happened? I mean, this is kind of crazy, but what happens if you do a list AB plus missing of you know, vector symbol of X.

Yes. That might inherit properly.

This would require set attributes of missing.

Right, yeah.

Set attributes of missing command.

No, the vector symbol thing has a certain propagation through various things. So. So it might just have to be set to propagate through missing.

But yeah, that's.

Look, my suggestion is that I don't think the name is right but that we have a function like conform missing that essentially resolves what are otherwise missings that are treated purely symbolically. It's kind of the opposite of delete missing which just gets rid of all the missings. This is something which infects things with missings by default. Now.

We have an option missing behavior which with value automatic makes query and data set. Do clever things with missing.

What kind of clever things?

Well, depending on the function like mean or total, it will do the what you expect replacing by zero or whatever.

Well, right, for. I presume for mean you'd replace by zero and decrease the denominator by one.

Yeah, one would hope. Oh God, you know, this is just so hopeless. You know, this is a totally under documented function. Okay.

Yep, yep, absolutely.

So I mean, I agree this is a reasonable thing to have for tabular presumably as well.

But yeah, I mean it, I think it's generally useful, although it is very mystical. Not just because it's not very well documented, but because it's just. I feel like there are a lot of situations where I don't know where. Where there's complicated stuff of missing and this kind of just magically handles it in a way that's different from how it handle, you know, outside of query.

Yes, yep, yep.

But I mean that suggests that we want a wrapper.

I mean look, if we have, if there's a sort of a single behavior for missing that we think is like you either want to have the current behavior where it basically does nothing or you want to have this other behavior where it does something like what missing behavior does with automatic. Then I mean it sort of sounds like maybe we should just have a different symbol that just has this behavior everywhere all the time.

I see. That's an interesting idea. Christopher. Yeah.

How are we generating missings? I mean, other than just what generates missings? Some of the entity value stuff generates missings.

Yeah. Missing keys, associations.

Yeah.

And you know, the examples I have here is when we can't convert, we're generated and missing or an. And instead of doing an overflow and converting to big nums, which would be disastrous for a huge table, we generate a missing.

I don't understand why we would ever want missing. I mean, if you say mean of 2 comma, 3 comma missing, I just don't see why that would ever want to be something like that.

Well, I don't think it would. It doesn't necessarily want to be 2.5 because that could just be wrong.

Right.

It depends on what's. What stage of the computation you're at. Right.

Yeah.

And how much you want to fudge your data.

There's some interesting examples and I don't remember exactly where I played with the entity framework a bit. And you know, you do stuff with country data and, and you, you get these, you get these bizarre means with, you know, nine times missing plus this other stuff. And I guess it tells you how many countries don't have the data, which is useful in some sense, but doesn't actually give you the results you always want. So that's why I think this needs to be. Yeah, I mean it needs to be controllable somehow.

Well, that's why I'm suggesting a propagate missing.

Yeah.

Type thing because. Okay, so the real question is at the source of the missing. Do you know how you're going to want to handle it or do you better know that at the end of the computation when you're kind of trying to pull everything together? I claim it's more likely you know it at the end of the computation.

I think I agree with that.

Yes, yes, but, but on the other hand is if you have a whole chain of. For, you know, for example, for doing stuff out of core, we have a whole chain of, of commands. I guess if we can know ahead of time that we're going to propagate missing in a certain way that.

Well, yeah, we can always have a missing behavior. Ish thing. But that is a, an option. I mean, so long as there is a default behavior and it's changeable and you can get the pure symbolic behavior somehow, I think it's okay.

Well, no, wait, wait, hold on. There's two, two sort of points. I mean, one is, I think maybe this is probably what you were already saying, that if you're doing big operations, you, you really, you can't have it like a tabular column of, of integers. Of integers where some of them are missing. You do an operation with it and now you have a tabular column of expressions because, well, some of them are now two plus missing, which is not an integer or a missing. Right. Like that's, that would be very inefficient if that's sort of how it worked. The other thing is, I would say recovering, you know, saving yourself from missings at the very end, I think is generally bad because there are lots of places that missings could have come from. And when you just have a result where it's, you know, 1/3 5 + missing, it's really hard to know like, well, what actually is that missing? Should it be zero? Should the whole thing be missing? And you know, it helps you if you knew that, oh, that came from when I called mean. And then I have a sort of certain missing policy I'd want in that particular function that's different from if I was, you know, using, you know, times at. At. I probably want one as my missing substitute.

So I think it's the horrifying case.

Right, I see. I, you know, when you asked earlier, are you more likely to know at the beginning or the end, my answer was going to be in the middle.

Right. Yeah. Like when you're actually doing the operation that's going to create some ugly missing combo expression. That's when you know what you want the missing behavior to be.

Exactly. Because what. But in it's, it's sort of, you know, missing is sort of tied in a little bit with the types.

Yeah. Actually, I think does. What does indeterminate do with like indeterminate over indeterminate or indeterminate minus indeterminate, etc. I think I remember seeing some cases with this that were sort of surprising. But.

Yeah, actually, I mean, it should be.

In any sort of purely numerical thing. It should just resolve to indeterminate. Yeah.

Okay. That's what you'd want.

Yeah, yeah. So in some sense maybe what the semantics which I've got for this missing stuff is really, perhaps indeterminate is a better way to think about it.

And maybe actually one other argument for something like indeterminate maybe is that it sounded like, at least before Rob, it sounded like missing. At least for now. You don't have a good way of handling the. The argument. Like if you have some of them have missing of quotes, blah. And some of them are missing of quotes and so on, there isn't a great way to handle that. So indeterminates are always just indeterminate.

Right. And in some sense when we can't convert, we, we can't convert to a machine integer, it is sort of indeterminate in some sense. So.

Right. But of course, you know, indeterminate is only unambiguously indeterminate in a purely numerical context. Right. I'm pretty sure if you did indeterminate plus Today you'd get indeterminate plus a data object and not indeterminate.

Oh, I think it determined it's just an up value.

So indeterminate is very infectious.

Yeah, I mean, I think, yeah.

F of indeterminate is.

I mean, what happens if you. So, so I mean, there can be other types, like strings that are not numbers.

Oh yeah.

But those length of indeterminate.

Yeah. Oh my God.

Yeah.

Okay.

I think indeterminate, if I remember right, it's just an up value. So if the function calling it is like has the numeric, some numeric attribute, then it, it returns indeterminate.

Yeah.

Ah. Which means if, if you cleverly call it on, you know, like one of the integer functions which doesn't have the numeric function attribute, it won't.

So we have another one that has similar poisonous behavior which is called undefined. So indeterminate means it exists, we just don't know what it is. Right, so means that it doesn't exist.

Right. So like indeterminate and you know, and A is probably. Is indeterminate and a. But undefined. And A is undefined.

At least.

So this is like the, in computer science that's usually done as the bottom element, the bot.

I mean, one problem, by the way, with things like indeterminate and undefined is that we have a little bit of machinery around missing, like delete missing, I guess maybe replace missing or something at some point. But we don't really have any good machinery for stuff like undefined.

Well, maybe, maybe we could have a missing with a tag indeterminate. That would, you know, it's sort of a workaround a little bit. But that would be, that would be. And, and missings with that always propagate.

Oh, like missing of indeterminate has indeterminate semantics or something.

Yeah, with a tag, I mean, I think that it would be, quote, indeterminate, you know, as a, as a tag.

Okay, that would be very funky.

But it would be very funky. Yes, but, but maybe.

I mean. Right. I mean, basically the point would be you're defining the behavior, how you want this missing to behave. Like you can give an argument to the missing that defines how you want it to behave. Yeah, I mean, there is something nice about that because, for example, if you know that, I mean, you might imagine we have more than just indeterminate. So maybe if, you know, it's an indeterminate integer versus an indeterminate string, it can maybe be a little bit different.

Yeah, and that. So, so, so instead of having the blank missing, maybe we could have missing of indeterminate. And then, you know, if people have different missings, we can try, you know, I think on propagation those will. Will go. But it's, it's just a. Yeah. I'm not sure if that's the.

Look, we already have this concept of missing and reason.

Yep.

We certainly could have some kind of option to missing which would determine its propagation. But I'm still understand. Trying to understand what. You know, for the mean case, for example, what do we actually want? You know, ignore. Take the data to be just absent and mean everything else without the data. You know, you could say invalidate the whole result. You know, what do you want to do? So, so I mean, maybe you had a list here.

I have a list. So.

I think the most common case will be nothing. Semantics rather than index.

Just, just drop it.

Just drop it. I think that's, that's, that's the, the common case.

Okay, so. So let me, Let me, Let me just. Let. Let's make some notes here. So what to do, what semantics to have for the missing when used. And so one possibility is indeterminate. Which, which just infects everything.

Not, not everything. There's undefined is more powerful.

Okay.

They have different semantics.

Okay. Undefined. And then there is nothing. Right. So nothing just shrivels up and disappears. Undefined infects everything and determinate affects arithmetic. Right. What other cases do we want for missing? I guess we want another possibility is.

Zero.

Maybe absorb into. But that's probably nothing absorb into one missing. So in other words, if you have seven missings, if you have a sum of terms and I don't know, sum of terms, maybe that's not right. Seven plus missing. A bit of weird. But seven, eight times missing is even weirder.

Yes.

And. And almost certainly wrong because that means missing minus missing is zero and all those kinds of things.

Yeah.

What other behaviors can one imagine wanting? So let's imagine that missing behavior could by default, missing behavior arrow indeterminate would make the missings behave like indeterminate. It might be crazy. I don't know. See what I'm saying?

I think that's, I mean that's a little. At best it should be quotes and determinate because otherwise it just sounds like it's unclear what the missing behavior should be.

Yeah.

Yeah.

Okay. But now where would we put that option? I mean, in other words, that could be in any of these things, like tabular, you know, like this tabular column, for example here could have a missing behavior.

So, so we had an idea for that which we put at the very. It wasn't just with missing behavior but the other ones. And that was, you know, that we discussed this morning. And I don't know if it's the best idea, but it's a possibility. So we could have a. So we could have a missing behavior option which if you give it to tabular, that inherits to transform columns and stuff. But we could also for globally have a dollar missing behavior variable that would. That would set it, you know, as a global thing. And I know that's not ideal to have global things, but there's some cases where you can't avoid that.

I don't see how that would help us. Wait a minute, what? Why would we want that?

Okay, so suppose I have a tabular column and we've so. So tabular column. And you take add that to another tabular column.

Yes.

Okay, so those. So, so how would you control the missing behavior on that sum?

You'd have to have a wrapper around the whole thing.

Yep.

Because a global variable just seems horrifying. Just going to be completely non reproducible and you know, just a mess.

It could be the default value of the option. Yeah, so I mean we have some triples like in assumptions. They are assuming to control from outside the option assumptions and the dollar assumptions.

Yeah.

And they play well together. You choose what's the method you want to follow.

Yes, I agree.

And the default of the dollar variable would be automatic. And so somebody could. I mean what's nice about that is what the dollar variables are nice is you can locally set it in block.

Fine, fine.

I mean I'm not saying it's the best way to do it, but it does have some advantages.

Yeah, okay, fine, but. But then you're saying that if we have a missing behavior option that we can have dollar missing behavior, blah, blah, blah.

Yeah.

Then what we would end up with is something like with missing behavior. Or maybe we need a different term for this with missing behavior. Blah. Or we would have the missing behavior option or we would have dollar missing behavior. Okay, but so now. I mean we could call this. Combine or something we could call this. Always propagate. I mean just. I mean that's disappear basically. No, I'm not sure what the right terms are but I mean there are these different cases and it might be.

Ignore is is another thing instead of disappear.

That's better. And I mean you're claiming there might be just a plain zero. Set it to zero. It's not necessarily zero that you want to set it to. I Mean you might want to set. Go ahead.

I was thinking that it will depend very much on the operation we want to perform and something that is fundamentally mathematical, the zero might be useful.

Yes. Look at these here.

For example, in mean, in general, we might want to make it zero and count one less. But what if the user wants to count the presence of the missing but still make it zero?

Sorry, what do you mean by that?

That if you have a list. Exactly. So if you have a list of three elements and one missing, you still want to divide by four.

Wouldn't that just be replacing the missing with zero?

Yeah, that's doing missing imputation.

Yeah.

Yes, yes.

I think if you want to treat missing as a specific value, the proper way is to impute it.

Yeah, I think that seems overkill.

Well, you can impute it as zero, which is what that is.

I mean, how do you do that?

I mean, you could just say what you proposed last time was that transform missing. We haven't implemented that yet.

Transform missing.

Yeah. If you go open the recap, we look at that last time. Transform missing, the second one. So this one was the proposal. And then if you scroll down, that's what you propose at the end. We probably have to do some more design rounds on this.

Yeah, yeah, yeah. Okay.

Well, so, but does the, the problem is, does the missing see the missing behavior? Does that control that case of converting strings to numbers? It almost seems like we need another, Another thing which Jose suggested is what is it? Failure action or, or action. I can't.

Yes, the. The current option in, in query is failure, failure action.

See this here? Instead of doing some of regenerated and missing from something that doesn't work, which is a little bit different than propagating missing.

What does the compiler do when, when you try and just cast.

Oh, it just, it just fails. It did. It just returns an exception.

I see.

I mean, you can, you can use the, the undocumented unchecked arithmetic and will wrap around. But, but the default, the failure. And, and, and I've worked with a compiler to set up this arithmetic so that we can catch the, you know, catch the exceptions and, and continue on with the computation, which is typically what people want to do.

Right, okay, so, but what you're talking about here is the conversion. The first section here is all about data conversions for column.

Yes. Yep, yep.

And what it should do when. Well, what do you think it should do? I mean.

Well, I, I think, I think we want to allow people to be able to control that because in some cases people do Want to get big nums and actually do the, you know, continue the competition with big noms. But in a lot of cases for BigTable, they don't want to do that. So, so we proposed, I think at the very end there's a proposal for, for what we might do with the failure action. Think Jose wrote. Yeah. Look at the compare to failure action.

Yeah. You know, why does this remind me of the whole, you know, error handling discussion? You know, in other words, what to do if some particular kind of thing happens.

Yes, it's exactly that.

And the problem is that what we need. What's that?

It's a friend.

Yes. But what we needed in error handling, let's remind ourselves the problem of error handling is if you were going to write everything precisely, you know, then you don't even have any errors. You're going to have code that just checks every case. The whole point of error handling is it's a safety net. It's a broad safety net that tells you roughly whether you want, you know, the analogy is falling apart here.

Well, yeah, yeah. I mean, there's already a difference though, which is that there are, well, there are cases when you just, you, you know, you're going to have missings. It's not a matter of, oh, I didn't consider it. You just, you know, you're going to have them.

Yeah, fair enough, fair enough.

Like they semantically mean something, right?

Yeah, there's, there's cases of generated missing, like some of what I'm talking about here. There's also missing pieces in the data observations that just weren't made.

Right.

I was going to say, doesn't numeric array have like a way of, you know, maybe it's a method option of like saying how you want to convert.

Things like that, that example, the top of the screen there of the top of that notebook conversion method goes to round. That's a numeric array conversion method. Okay, so there's a whole set of. And, and that, that I've used those methods for the vertabular column conversion with, with sort of directive opt.

So hold on, let me just understand here. So the first question is, okay, let's say we've got an importer and it is importing from a CSV or a spreadsheet or something, and there is simply stuff there that is like a null string in place in place of a number.

Yep.

And then later on we compute the mean of that column. You know, I think that the civilized thing to do is to make it clear. That, You know, there was something Wrong. Because otherwise people just. I mean, I'm sure there's a million papers in the medical research literature, for example, where people had missing data and they just went and computed statistics on it. They had no idea there was missing data, how much of it there was, and they quote a P value of whatever it is, and it's all nonsense.

Yeah. Honestly, I think the conservative thing is just to return missing in a lot of cases, just have it propagate. And if you don't want to propagate, then you have to do something.

Yeah, yeah, yeah.

I mean, I do think this. It reminds me of numeric precision where we have done a good job over the many decades of returning only, you know, in arbitrary precision, at least returning on the digits that are justified.

Yes.

So, for example, let me. Let me give an example here. So I could imagine in a fit, you might want to say my missing is certainly between 0 and 10, though I don't know what it exactly is. So make me a fit that allows all my missings to be anywhere from 0 to 10. Don't ignore the missings. The missings, in fact, generate uncertainty, but bounded amounts of uncertainty. Does that make sense?

Yeah. So that would effectively be doing the transform missing. You transform missing to around. Blah.

Yeah, Yeah, I think so.

But around has a definite distribution. I mean, it thinks it's a normal distribution.

Not really.

It does.

Well, no, it. Right, it really does. But I mean, that's probably what you'd want anyway if you wanted to have something weird like this, where you want to say that I have this value, but I know it's between x and Y. I mean, around seems like it is like kind of the formalism we.

Have for that or answerable.

But, but, but either way, I mean, it's like this is a. This is a separate. I think this is kind of a separate step. Like when you call meme and there's missings in it, you'd like it to. To return missing.

And then.

And then when you see. Right. You like to return missing. And then when you see. Oh, well, actually I have a policy for. I know that all these missings are between, you know, two and four or something like that. Then you use replace or transform missings or whatever. We have to transform into whatever you want.

Right.

And I realized another way to think about it too, is to think of that as weights. So let's say that you want to compute a mean and you have a missing element. If you set the weight for that element to be zero, it will effectively not be counted. But you still count it as an element. So you're gonna still divide by N if you had N. See what I'm saying?

No, you wouldn't.

Well, in general, this is a complicated problem.

You divide by the sum of the weights.

Right. This is actually something I've encountered, which is if you. I mean, this is like a nice feature we could have, which is if you do mean of weighted data, if you have something with weight zero and the value is something silly like missing, then, or like infinity or missing or something like that, you'd like it to just ignore it. Right. But instead you end up getting, I think, weird algebraic stuff.

Yeah, but if the weight is zero, that should be. I mean, one practical example is if you're doing filtering on an image or an array, like do you have the usual boundary problem? And we support padding none. Which what it's doing is using a smaller kernel at the boundary, which is effectively setting the weight of the, of the stuff beyond the boundary to zero. And those are, you know, stuff beyond the boundary of an image is practically missing elements.

Good point.

Yeah. Right. Okay, so let's. All right, let's go back. What are we trying to achieve with the calculus of missing? We would like to. Once we're in comput. Okay, there's on the way in, you know, the, the blank column in the spreadsheet, the blank cell in the spreadsheet or whatever. There's, you know, there's the -999 in the spreadsheet, know what I'm saying? Where we have. Okay.

Where proxy element is used.

Yes.

Yeah.

Okay. So this is so on import. It's like a number was given as. Blank was given as - 999, etc. What do we want it to do? Or, or it could be. Or, or could come. From curated data where we know, eg, you know, no atomic mass for an radioactive element. So in other words, if we say. Right, because there isn't, you know, there's a. No natural abundance. So it is, you know, people usually quote, they put it. Usually put in parentheses, you know, the, the most common isotopes mass or something. That's an example of a, of a very controlled missing, if you get what I'm saying. We kind of really know what's going on at some level as opposed to just it was blank in the spreadsheet, in which case we have no idea what was going on without giving additional information. So what are some other, what are other cases on import where we want to get a missing. Well, okay, another, another Issue is the specified data type can't.

Represent.

Yep. The specified data. Another case is something like, you know, look up, you know, something like. We attempted a lookup for an element that isn't there. Right. So those are, I mean, honestly, in that case we just want that error to propagate. We don't want something where, you know, if I say look up in, you know, A goes to X and I say look up B. I don't really ever want that. I mean, I never want that. I really want that to infect any computation. It seems extremely unlikely that I would want to just set that to zero in computing a mean. Does that make sense to.

People? I think it will depend on the computation. If we are looking at, I don't know, some data about millions of people, we will not care about.

One. Yeah.

Okay. This is something that you typically decide at the beginning. I mean, some algorithm can are built to deal with missing data inside them, so you want to leave them some.

Others. The question is why is the data missing? Is this something which is where you have a structural error, in which case you better be told your code is wrong, or is it something where there's messiness in the.

Data? I think that's minor. I mean, there is some stuff that is. If we are very aggressive in creating the missing, it might be that you have missings because of some error. But typically missing is because the data is simply not there with a questionnaire and people didn't answer the.

Question. Yeah, okay, fair enough, fair.

Enough. So by far I think the major case of missing and a lot of misting we are going to generate by being smart and maybe suggesting. We need to discuss, to do a nice discussion about schema. But some stuff that we can suggest is if you have a string column like none known empty string could be suggested as possible missing element. I mean, typically there's consistency in how you define it, but it really depends on the file. I mean a single file typically use the same convention to define the missing, but they are different conventions. Not too many, but there.

Are. Okay, but okay, so one question is on import. You know what we probably think we're going to get is a missing with possibly some defined missing behavior that came in from the import. Okay, but I mean, so missing generally has a reason which is mostly human readable. And I mean, let's imagine that missing essentially had a missing behavior option and you could if you wanted to set it when you create the missing, does that make sense? And then that missing behavior option would determine how various kinds of Functions handle this missing. Now in a perfect world, You know, we just went through up at the top here, these cases here. So in other words, one way to define the missing behavior instead of giving a missing behavior option. Okay, so one possibility is, you know, missing behavior option. Okay, there's missing behavior option. Handling is basically fully global with either dollar missing behavior or with missing behavior and so on from the outside fully global. Okay. Now another possibility would be when the missing is first introduced. So then it would be a missing blah blah blah blah blah with missing behavior arrow whatever you want it to.

Do. So like practical example here is like import creates missing because the data is not.

Available.

Yes. So like translate and that by default goes to a specific missing.

Behavior. Exactly. Okay, so now, okay, then there's in a particular operation. In other words, we could imagine something like mean blah blah blah blah blah missing behavior arrow blah blah blah blah blah. We could imagine fine grained control in specific functions or more interestingly and more relevantly, something like, you know, find fit missing behavior or whatever. Comments on this? I think this is fairly sensible. I mean it may not be precisely right, but I think it's fairly.

Sensible. So maybe one advantage, I'm not sure. This is great, but one advantage of this is that it means it can be a lot more efficient than say it means you only have to go through the list once, for example, because you don't have to first replace all the missings with whatever and then run.

Meme. Definitely. I mean, which, which thing are you saying to this? I mean, we've got cases one, two.

Three. Basically, I, I mean.

Three. Three, okay. Yeah. I mean three would be a thing that gradually seeps into the system with different functions, you know, handling it.

Differently. And sometimes using tabular, we can experiment early. Yeah, because you don't need to modify mean. You just modified what mean does when it's compiling to.

Tabular. Sure, but I mean I can imagine missing behavior for some image processing operation, for example, some very elaborate things, machine learning things. What does it do when there's missing stuff? What does classify do when there's missing.

Stuff? Depends on what you ask as an option, right? Yeah. I mean some methods can deal missing data, but most of the methods cannot. In that case, if you have imputation on which is the default, it imputes them. Otherwise it fails to the message that you have missing.

Images. But we can't even represent missing in images.

Right? Yeah, I was about to say that we don't have a way to plant missing.

Images. No, I mean, okay, you can.

Fake it with alpha channel use up by the.

Way. But yeah, what to do in visualization, etc. So one possibility is drop the data point, which is what we often do. Two is drop with an.

Indicator. Sorry, by visualization you mean type, setting of tabular or.

Like. No, no, no, I mean list, plot, eg, we know the X value but not the Y.

Value. You put a bunch of straight.

Vertical lines like indicate with a tick like thing. You know what I'm saying? Drop with an indicator. Okay, so another thing would be eg, array.

Plot. Also in visualization. We didn't have missing when we developed visualization and users feed us so many things that are not missing but are not real. So the correct characterization for a function, like for visualization functions is non.

Real. Yeah, okay, well, complex or whatever. But I mean, so this is the case. Okay, look, look, there are, there are three cases. Depends on whether any coordinate is defined. And that could be very subtle because it could be in a stream plot or something. Okay, so case one, no coordinate is defined, probably have to drop except if there's a legend. And then some coordinates are defined, some placement coordinates are defined, so you know where the data is. And then there's all placement coordinates are defined. I mean there's things. What other things are there? Visualization. There's modeling as well, which we already talked about. You know, with modeling again, you could have a situation where the X coordinate is defined but the Y coordinate isn't. And can you do anything better with that than just saying both are.

Undefined. You mean like fitting or what kind of.

Modeling? Yeah, something like that. I mean like for example, here's an example. You have 10 dimensional data and some of your data points are missing two of their dimensions. What happens? Yeah, eg, You know, what about dimension reduction? In that case you should be able to. In other words, if you have 10 dimensional data and you have 8 of the coordinates, that's enough to be able to do a lot of things, even though you couldn't place it.

Precisely. Well, you could do with dimensionality reduction, if you have a function trained to do dimensionality reduction already, you can also impute the missing.

Data. Yeah, okay, right.

Fine. Yeah, but yeah, to do. In general, you can do dimensional reduction on, on missing.

Data. Right, but so look, a fundamental thing is, okay, four is just impute the missing data. That's a different case, probably. Well, no, actually that's just a thing to.

Do. No, but fill the missing data is reducing it to cases that are.

Understood.

Right? So that makes it normal data which is missing.

Free. Yes, I agree, but that, that's another one of these. Look, I mean, in a mean, for example, imputing the missing data is very different from ignoring the missing data. Well, actually for mean specifically, it might not be so.

Different. I don't know. I mean, depends on how you impute or what you.

Impute. Okay, but let me take a more extreme example. Let's take max as an example. See what I'm.

Saying?

Yeah. You know, handling for mean versus max. So in.

Max, you know, we need a max version that's more like descriptive statistics than a mathematical.

Function. What do you.

Mean? Like a list max that behaves on lists? Kind of like how a descriptive statistics.

Does. In what.

Way? So that you can get column wise maxes, for instance, you know, it doesn't reduce it to.

Us. Yeah, you know, yes, maybe we do need that. A list.

Max. That's. That. Remember that's the element wise we talked about a few weeks ago. So element wise of Max does the list.

Max. I.

See.

Okay. Well, I think it's probably pretty common to use.

Max. Well, I mean, we, we cooked that goose. So I mean, it.

Is. No, no, but I, I mean list max, you know, it's worth having its own.

Possibly. Yes. What the hell does that mean? Are you using a. Oh, it's drilled all the way.

Down. Oh, no. Yeah, it has. I mean, you. I mean, to make it do anything, you need to have two arguments. So you could put a comma, five for the second argument.

Or. Why.

That. No, no, no. Element with maxax is a.

Function. Yeah, max is a function. So. So add another argument to what.

After the lists to element wise max. Element was. Max has a single argument now. So it just returns.

It. Right, so just. Yeah, where you. There you go. That would do.

It. Exactly. It's boring because the first one is all.

Bigger. Not very.

Interesting. Yeah, okay, I get.

It. But we even need one that works just on one big array type.

Thing.

Like. Like a.

Mean. Like a.

Data. Sorry, he does that. Max does that on a.

Bigger. No, no, the problem is what Roger is saying is that we want this to basically not drill. So I'm saying in this case to.

To. To behave like mean. Like if you give that, that array that Stephen is typing in now.

To.

Mean, you will get column wise means. Okay, and you want column wise.

Maxes. Yeah, I mean, it's possibly a mistake that max flattens, but it's a very old.

Mistake. You can apply element wise max and get.

That. Yes, yes you.

Can. Very complicated. But yeah. So it is probably the case that we need a list max to deal with this situation. Because. Okay, we should probably wrap in.

A moment where I see people having a ton of problem right now. So what I would put front and center is kind of descriptive statistics at least like list processing functions. This is where it's like a real pain today with missing stuff and then progressively seep in deeper and deeper into solvers and other.

Things. Yeah, well, I mean visualization is working right.

Now. I think that we don't have a lot of problems in visualization because we tackled it, but we can make it richer like you.

Suggest.

Yeah. The other thing is I think we need a list of tools in the tabular thing that gives you, you know, to fill missing, to remove missing, you know, and that will mitigate everything from down, you know, the whole workflow. And then progressively the whole system gets kind of more.

Adaptive. Well, okay, but then the question is, what do we do? I mean, so one way to mitigate missing is replace them all by zeros or some.

Bad. Yeah, I mean, or nothing. I mean but we have like at Julia, we curated like a whole table of sort of common ways to do, you know, fill.

Missing. Yeah, yeah, yeah. We've got. And it depends on the type, like strings. You don't want to replace with a zero. You want to replace probably with an empty.

String. Pretty much, yeah. This is a column based operation.

Typically. Where is your list of these things.

Open? Replace missing. The ref page. I haven't renamed that.

Yet. Replace.

Missing. Yeah. Oh, I did rename that, sorry, Transport missing. I forgot to delete.

That.

Yeah. Okay, so if you look at the table, there's a bunch of things.

There. We have got to reduce the size of.

These. Yeah, yeah, you send an email. Brian.

Answered. Okay.

Okay. This on the, on the published.

Pages. So that the first one that says constant, that's your zero case.

Stephen.

Yes.

Yeah, yeah, these are all interesting. What about the previous non missing element as opposed to the next non missing element? Oh, you have previous there.

Sorry. Yeah, I need to sort it non alphabetically to make it more useful to read, but. Or maybe split it in different.

Tables that using the previous element. I mean from the nature of data collection, that's not an uncommon.

Case. No, it's not uncommon.

Case. I think the people that do census, they don't fill in the addresses. If it's like four people in the same house. Yeah, you know, they use the same. From the.

Above. Well, I can. No, these are all very reasonable.

And these are, these are basically with the Exception of interpolate, that I'm going to in a minute. These are univariate. Let's say they work on a single column interpolate. You have two ways. You can assume the data is equally spaced and work on the unordered and still work on the single column, same as the others. Or if you signal that one column is the independent variable, like time typically means almost 99% of the time is time. So if we had that time series in tabular, for example, this would be very interesting to be able to use.

That. And then we have time series itself. Do we have any concept of missing data in time.

Series?

Yeah. And how is it.

Handled? Typically, interpolation, you can give any function you want to handle it, but that's not necessarily a good thing. I mean, it's. That means that you also have full.

Responsibility. Yeah.

Yeah. But this is what we're trying to do here to take care of the most common cases. So you don't have to write functions to do.

That.

Exactly. Then there are techniques that are fancier. Like for example, this nearest I have written down in the table is basically the closest between next and previews. But you could. It's unfortunate, but you have nearest neighbors and I need to do some work on the naming here. But nearest neighbor is looking at the whole row. So it's defining a distance function between the. Is the similar concept of the minia is looking at missing and not missing and waiting the distance. Yeah. And you replace with the closest.

Row. Nearest row, Closest row or something. Use the word row because it's the whole.

Thing. Exactly. Is like replace, replace, like find the closest row and then take that element for that specific column and replace it. And then you have even fancier techniques which are still simpler than what we do with the synthesized missing value, which is very complicated. Where you, for example, fit a linear model like column B, could be a linear combination of the other columns. And you can do this recursively and try to impute missing data in all the columns. There are a lot of techniques to do this.

Yeah. From nearest row as a name. So the from gives you the indication that there's more to it than just dealing with this element on its.

Own. Yeah, okay. I mean, we can. We can probably go through these in a separate.

Meeting. Yeah.

Okay.

Yeah. Okay. Zooming out. And then we have to wrap up here. So with respect to the original question, you know, what I think we're concluding is that there should be variants of missing where either we're specifying the outer sort of operation Wrapper missing behavior always specifying the missing behavior burnt into the missing itself that will then sort of be made use of in operations that. In the case of, I mean all this stuff that you've defined here. But transform missing would also apply to import. In other words, this is an attempt to just get rid of the missings. Yes. As opposed to carrying the missings and tagging a missing.

Behavior. Yeah, yeah. I would say that those two are token.

Approaches. Right. You know, you can imagine exotic things like a missing behavior in a missing. In a list which says go, always pick up the value. If you need to get a value here, get it from the next element of the list. So I mean, I think one point is there are situations in which you can carry a symbolic missing and there's situations in which you just have to commit to something. Do you know what I'm.

Saying?

Yes. Okay, so we've got two different things we're doing here. So carrying the symbolic missing, we've got the missing object and missing behavior replacing, we've got transform missing and we've also got options to import and so on. And in a sense missing behavior is like a lazy transform. Correct. Missing behavior says transform this only at the moment when you absolutely need to use it in a. You know, in other words, okay, so look, there are, there are different, you know, you know, eventually missing must be transformed for many kinds of operations. Okay, so one possibility is case one, keep missing symbolic recording a missing action somewhere, a missing behavior. Somewhere. In case two, transform the missing explicitly into something that isn't missing. So that's the eager and that's the lazy case. Do we agree?

Yeah.

Yep.

Yep.

Okay. So the lazy is passing the buck basically.

Right. But keeping the information that this is missing, whereas the transform is just like blow it.

Up. We also have a third case for tabular which is that when we do transformations there we can, since we explicitly record and efficiently record what's missing, we can make it work on only the non missing things very.

Easily. Yeah, okay. I mean, I, I don't understand. I mean, but, but fundamentally to the user it's like do you keep the, the trace of the missing or do you replace it by zero or something? And sometimes you only replace it by zero inside an.

Operation. But I think for instance, like Rob has now implemented mean of, let's say, okay, you know, of, of a column, right. And if they're missing, it's just ignores.

Missing. That's only one possibility. As you, as you've said in the, in the. Yes, there are many Many different possibilities.

Yes. Right, right. And it's, you know, the, so the, the, the way, the way the missing works is, you know, as it is now when I just have this really generic missing is very efficient because it's just using, it's just using bitmaps to propagate the missing. So we can, you know, we'll be able to. So, so if we can have a default case that. It does a bit of that. But that would be nice. It would keep things very efficient for big tables. But, but we can, we can have these different flavors of missing. But it's going to add, you know, quite a bit of memory and processing time to propagation of.

Missing.

Right. But given that, you know, we can do. I mean, I guess, you know, there, there will be questions like if I add two missings with different missing behaviors. You know.

What? Yeah, I understand, I understand. But that's like two plots with different, you know, you show two plots for different axes. Well, we have to make a.

Decision.

Yeah, but this seems to me fairly flexible and usable and a really common thing. And something we should be able to do is like on something like importing a spreadsheet with missing behavior or automatic. See, I don't know, maybe the minus 999, you just never detect it. I mean, there's no way to automatically flag.

That. Well, we have a missing value pattern option that, that would allow you to say that that's to be.

Detected.

Yeah. What's it do with it when it detects.

It? Records.

It. Replace it with.

Missing. Oh, it.

Does. It's missing.

Yeah. Okay. Yeah.

Let'S. Again, that's a value column thing. Typically like this column minus one is.

Missing. Okay, well, so again let's. Okay, how about we do this? Let's try and make a guide page of all things missing, so to speak, which includes things like missing value pattern, all these kinds of things to try and pull together what our different design approaches are to this. Recognizing that you're either getting rid of the missings or you're keeping the missings and saying what to do with them. Fair. I mean, it's sometimes to say what to do with them means if there's ever a missing blow up the computation, you know, Makes.

Sense.

Yep.

Yeah.

Yeah. All right. Anything else for this meeting? No, no. Great. Let's wrap it up. See you guys.